Send Custom Agent Traffic Events

Send bot and AI agent traffic data from any platform to Scrunch AI. This endpoint accepts traffic events in JSON or NDJSON format and automatically classifies bots based on user agent strings.

Endpoint

POST https://webhooks.scrunchai.com/v1/sites/{site_id}/platforms/custom/web-traffic

Authentication

Authenticate using the X-Api-Key header with a JWT token provided in the Scrunch AI dashboard when creating a site with the “API” platform. To generate the X-Api-Key, add a new website on the Agent Traffic page and select API as your platform.

X-Api-Key: <JWT_TOKEN>

Path parameters

Parameter	Type	Required	Description
`site_id`	string	Yes	The Site ID (ULID format) assigned in the dashboard.

The platform_id is always custom for this endpoint.

Request headers

Header	Value	Required	Description
`X-Api-Key`	`<JWT_TOKEN>`	Yes	Authentication token from the dashboard.
`Content-Type`	`application/json` or `application/x-ndjson`	Yes	Use `application/json` for a single event, or `application/x-ndjson` for multiple events (one JSON object per line).

Request body

The body can be either a single JSON object or NDJSON (newline-delimited JSON, one object per line).

Fields

Field	Type	Required	Description
`domain`	string	Yes	The domain of the site (e.g. `example.com`).
`user_agent`	string	Yes	The full User-Agent string of the request.
`url`	string	Yes	The full URL that was requested (e.g. `https://example.com/page`).
`path`	string	Yes	The URL path (e.g. `/page`).
`method`	string	Yes	The HTTP method (e.g. `GET`, `POST`).
`status_code`	integer	Yes	The HTTP response status code (e.g. `200`, `404`).
`timestamp`	integer / float	Yes	Unix epoch timestamp in seconds (e.g. `1700000000`).
`response_time`	integer	No	Response time in milliseconds.
`ip`	string	No	The IP address of the client making the request.

Example: single event (JSON)

curl -X POST "https://webhooks.scrunchai.com/v1/sites/{site_id}/platforms/custom/web-traffic" \
  -H "Content-Type: application/json" \
  -H "X-Api-Key: YOUR_API_KEY" \
  -d '{
    "domain": "example.com",
    "user_agent": "Mozilla/5.0 (compatible; GPTBot/1.0; +https://openai.com/gptbot)",
    "url": "https://example.com/blog/post",
    "path": "/blog/post",
    "method": "GET",
    "status_code": 200,
    "timestamp": 1700000000,
    "response_time": 120,
    "ip": "203.0.113.1"
  }'

Example: batch events (NDJSON)

Send multiple events in a single request using newline-delimited JSON. Each line is a complete JSON object.

curl -X POST "https://webhooks.scrunchai.com/v1/sites/{site_id}/platforms/custom/web-traffic" \
  -H "Content-Type: application/x-ndjson" \
  -H "X-Api-Key: YOUR_API_KEY" \
  -d '{"domain":"example.com","user_agent":"Mozilla/5.0 (compatible; GPTBot/1.0)","url":"https://example.com/page-1","path":"/page-1","method":"GET","status_code":200,"timestamp":1700000000}
{"domain":"example.com","user_agent":"Mozilla/5.0 (compatible; ClaudeBot/1.0)","url":"https://example.com/page-2","path":"/page-2","method":"GET","status_code":200,"timestamp":1700000060,"response_time":95,"ip":"198.51.100.42"}'

Responses

Status Code	Description
200	Events accepted and queued for processing.
401	Unauthorized — invalid or missing API key.
422	Validation error — check the request body against the schema above.
429	Rate limited — wait and retry. Respect the `Retry-After` header if present.
500	Internal server error.

Success response (200)

{
  "status": "ok"
}

Notes

Batch size: For NDJSON batches, keep each request under ~1 MB uncompressed.
Timestamp: Must be a Unix epoch in seconds. Both integers and floats are accepted. Invalid timestamps will fall back to the current server time.
Bot detection: The user_agent field is used to automatically identify and classify the bot/AI agent. Pass the original User-Agent string from the incoming request.
Activation: After creating a site with the “API” platform in the dashboard, it starts in “pending” status. Once the first valid request is received, the site automatically transitions to “active” within 5 minutes.

Best practices

Use NDJSON for batch uploads to reduce request overhead
Keep batch sizes under 1 MB for optimal performance
Always pass the original User-Agent string for accurate bot classification
Monitor response status codes and implement retry logic for 429 and 500 errors
Use the response_time field to track performance metrics alongside bot traffic

​Endpoint

​Authentication

​Path parameters

​Request headers

​Request body

​Fields

​Example: single event (JSON)

​Example: batch events (NDJSON)

​Responses

​Success response (200)

​Notes

​Best practices