Documentation Index
Fetch the complete documentation index at: https://developers.scrunch.com/llms.txt
Use this file to discover all available pages before exploring further.
Send bot and AI agent traffic data from any platform to Scrunch AI. This endpoint accepts traffic events in JSON or NDJSON format and automatically classifies bots based on user agent strings.
Endpoint
POST https://webhooks.scrunchai.com/v1/sites/{site_id}/platforms/custom/web-traffic
Authentication
Authenticate using the X-Api-Key header with a JWT token provided in the Scrunch AI dashboard when creating a site with the “API” platform.
To generate the X-Api-Key, add a new website on the Agent Traffic page and select API as your platform.
Path parameters
| Parameter | Type | Required | Description |
|---|
site_id | string | Yes | The Site ID (ULID format) assigned in the dashboard. |
platform_id is always custom for this endpoint.
| Header | Value | Required | Description |
|---|
X-Api-Key | <JWT_TOKEN> | Yes | Authentication token from the dashboard. |
Content-Type | application/json or application/x-ndjson | Yes | Use application/json for a single event, or application/x-ndjson for multiple events (one JSON object per line). |
Request body
The body can be either a single JSON object or NDJSON (newline-delimited JSON, one object per line).
Fields
| Field | Type | Required | Description |
|---|
domain | string | Yes | The domain of the site (e.g. example.com). |
user_agent | string | Yes | The full User-Agent string of the request. |
url | string | Yes | The full URL that was requested (e.g. https://example.com/page). |
path | string | Yes | The URL path (e.g. /page). |
method | string | Yes | The HTTP method (e.g. GET, POST). |
status_code | integer | Yes | The HTTP response status code (e.g. 200, 404). |
timestamp | integer / float | Yes | Unix epoch timestamp in seconds (e.g. 1700000000). |
response_time | integer | No | Response time in milliseconds. |
ip | string | No | The IP address of the client making the request. |
Example: single event (JSON)
curl -X POST "https://webhooks.scrunchai.com/v1/sites/{site_id}/platforms/custom/web-traffic" \
-H "Content-Type: application/json" \
-H "X-Api-Key: YOUR_API_KEY" \
-d '{
"domain": "example.com",
"user_agent": "Mozilla/5.0 (compatible; GPTBot/1.0; +https://openai.com/gptbot)",
"url": "https://example.com/blog/post",
"path": "/blog/post",
"method": "GET",
"status_code": 200,
"timestamp": 1700000000,
"response_time": 120,
"ip": "203.0.113.1"
}'
Example: batch events (NDJSON)
Send multiple events in a single request using newline-delimited JSON. Each line is a complete JSON object.
curl -X POST "https://webhooks.scrunchai.com/v1/sites/{site_id}/platforms/custom/web-traffic" \
-H "Content-Type: application/x-ndjson" \
-H "X-Api-Key: YOUR_API_KEY" \
-d '{"domain":"example.com","user_agent":"Mozilla/5.0 (compatible; GPTBot/1.0)","url":"https://example.com/page-1","path":"/page-1","method":"GET","status_code":200,"timestamp":1700000000}
{"domain":"example.com","user_agent":"Mozilla/5.0 (compatible; ClaudeBot/1.0)","url":"https://example.com/page-2","path":"/page-2","method":"GET","status_code":200,"timestamp":1700000060,"response_time":95,"ip":"198.51.100.42"}'
Responses
| Status Code | Description |
|---|
| 200 | Events accepted and queued for processing. |
| 401 | Unauthorized — invalid or missing API key. |
| 422 | Validation error — check the request body against the schema above. |
| 429 | Rate limited — wait and retry. Respect the Retry-After header if present. |
| 500 | Internal server error. |
Success response (200)
Notes
- Batch size: For NDJSON batches, keep each request under ~1 MB uncompressed.
- Timestamp: Must be a Unix epoch in seconds. Both integers and floats are accepted. Invalid timestamps will fall back to the current server time.
- Bot detection: The
user_agent field is used to automatically identify and classify the bot/AI agent. Pass the original User-Agent string from the incoming request.
- Activation: After creating a site with the “API” platform in the dashboard, it starts in “pending” status. Once the first valid request is received, the site automatically transitions to “active” within 5 minutes.
Best practices
- Use NDJSON for batch uploads to reduce request overhead
- Keep batch sizes under 1 MB for optimal performance
- Always pass the original User-Agent string for accurate bot classification
- Monitor response status codes and implement retry logic for 429 and 500 errors
- Use the
response_time field to track performance metrics alongside bot traffic
