Skip to main content
Send bot and AI agent traffic data from any platform to Scrunch AI. This endpoint accepts traffic events in JSON or NDJSON format and automatically classifies bots based on user agent strings.

​
Endpoint

POST https://webhooks.scrunchai.com/v1/sites/{site_id}/platforms/custom/web-traffic

​
Authentication

Authenticate using the X-Api-Key header with a JWT token provided in the Scrunch AI dashboard when creating a site with the “API” platform. To generate the X-Api-Key, add a new website on the Agent Traffic page and select API as your platform. 
X-Api-Key: <JWT_TOKEN>

​
Path parameters

ParameterTypeRequiredDescription
site_idstringYesThe Site ID (ULID format) assigned in the dashboard.
The platform_id is always custom for this endpoint.

​
Request headers

HeaderValueRequiredDescription
X-Api-Key<JWT_TOKEN>YesAuthentication token from the dashboard.
Content-Typeapplication/json or application/x-ndjsonYesUse application/json for a single event, or application/x-ndjson for multiple events (one JSON object per line).

​
Request body

The body can be either a single JSON object or NDJSON (newline-delimited JSON, one object per line).

​
Fields

FieldTypeRequiredDescription
domainstringYesThe domain of the site (e.g. example.com).
user_agentstringYesThe full User-Agent string of the request.
urlstringYesThe full URL that was requested (e.g. https://example.com/page).
pathstringYesThe URL path (e.g. /page).
methodstringYesThe HTTP method (e.g. GET, POST).
status_codeintegerYesThe HTTP response status code (e.g. 200, 404).
timestampinteger / floatYesUnix epoch timestamp in seconds (e.g. 1700000000).
response_timeintegerNoResponse time in milliseconds.
ipstringNoThe IP address of the client making the request.

​
Example: single event (JSON)

curl -X POST "https://webhooks.scrunchai.com/v1/sites/{site_id}/platforms/custom/web-traffic" \
  -H "Content-Type: application/json" \
  -H "X-Api-Key: YOUR_API_KEY" \
  -d '{
    "domain": "example.com",
    "user_agent": "Mozilla/5.0 (compatible; GPTBot/1.0; +https://openai.com/gptbot)",
    "url": "https://example.com/blog/post",
    "path": "/blog/post",
    "method": "GET",
    "status_code": 200,
    "timestamp": 1700000000,
    "response_time": 120,
    "ip": "203.0.113.1"
  }'

​
Example: batch events (NDJSON)

Send multiple events in a single request using newline-delimited JSON. Each line is a complete JSON object. 
curl -X POST "https://webhooks.scrunchai.com/v1/sites/{site_id}/platforms/custom/web-traffic" \
  -H "Content-Type: application/x-ndjson" \
  -H "X-Api-Key: YOUR_API_KEY" \
  -d '{"domain":"example.com","user_agent":"Mozilla/5.0 (compatible; GPTBot/1.0)","url":"https://example.com/page-1","path":"/page-1","method":"GET","status_code":200,"timestamp":1700000000}
{"domain":"example.com","user_agent":"Mozilla/5.0 (compatible; ClaudeBot/1.0)","url":"https://example.com/page-2","path":"/page-2","method":"GET","status_code":200,"timestamp":1700000060,"response_time":95,"ip":"198.51.100.42"}'

​
Responses

Status CodeDescription
200Events accepted and queued for processing.
401Unauthorized — invalid or missing API key.
422Validation error — check the request body against the schema above.
429Rate limited — wait and retry. Respect the Retry-After header if present.
500Internal server error.

​
Success response (200)

{
  "status": "ok"
}

​
Notes

  • Batch size: For NDJSON batches, keep each request under ~1 MB uncompressed.
  • Timestamp: Must be a Unix epoch in seconds. Both integers and floats are accepted. Invalid timestamps will fall back to the current server time.
  • Bot detection: The user_agent field is used to automatically identify and classify the bot/AI agent. Pass the original User-Agent string from the incoming request.
  • Activation: After creating a site with the “API” platform in the dashboard, it starts in “pending” status. Once the first valid request is received, the site automatically transitions to “active” within 5 minutes.

​
Best practices

  • Use NDJSON for batch uploads to reduce request overhead
  • Keep batch sizes under 1 MB for optimal performance
  • Always pass the original User-Agent string for accurate bot classification
  • Monitor response status codes and implement retry logic for 429 and 500 errors
  • Use the response_time field to track performance metrics alongside bot traffic