Crawl Readiness API: Check AI Crawler Access to a Site

Overview

The Crawl Readiness API reports whether AI crawlers can reach one of your brand’s domains, based on the brand’s most recent finished crawl of that domain. It returns the same signals the AI Access card shows in the Scrunch dashboard. Use it to check from your own backend whether a customer’s site is reachable by AI crawlers before investing in AI visibility work, or to surface readiness inside a partner integration. The endpoint is read-only. Nothing about how readiness is computed changes when you call it.

What the Crawl Readiness API includes

Whether a robots.txt file was found for the domain
Whether the site turns crawlers away at the server level (based on HTTP 403 rates across the crawl)
Whether the site renders mostly client-side, which limits what non-JavaScript crawlers can read
A sample of pages that robots.txt blocks, with the AI assistant families (OpenAI, Meta, Perplexity, Gemini, Claude) blocked from each

When to use the Crawl Readiness API

Use the Crawl Readiness API when you need to:

Check whether a brand’s domain is reachable by AI crawlers from a partner backend or CMS integration
Flag robots.txt rules that block AI assistants from specific pages
Detect sites that render client-side and may appear empty to non-JavaScript crawlers

For a detailed per-page audit score with content-quality checks, use the Site Audit API instead. To see which AI bots actually visit your pages, use the Agent Traffic API.

Endpoint

Method	Path	Purpose
GET	`/{brand_id}/crawl-readiness`	Crawl readiness signals for one of the brand’s domains.

The endpoint requires a bearer token with the query scope. See Authentication. Pass the optional domain query parameter to choose which of the brand’s domains to describe. Omit it for the brand’s primary website.

Example: check the primary website

curl -X GET \
  "https://api.scrunchai.com/v1/1234/crawl-readiness" \
  -H "Authorization: Bearer $SCRUNCH_API_TOKEN"

Response:

{
  "domain": "example.com",
  "robots_txt_present": true,
  "is_spa": false,
  "is_blocking_bots": false,
  "blocked_pages": [
    {
      "path": "/internal/pricing-drafts",
      "url": "https://example.com/internal/pricing-drafts",
      "blocked_by": ["OpenAI", "Perplexity"]
    }
  ],
  "pages_truncated": false
}

To describe a different domain the brand owns:

curl -X GET \
  "https://api.scrunchai.com/v1/1234/crawl-readiness?domain=shop.example.com" \
  -H "Authorization: Bearer $SCRUNCH_API_TOKEN"

Reading the response

Two fields are easy to misread:

robots_txt_present: false means permissive, not blocked. With no robots.txt, crawlers are allowed by default.
blocked_pages is a sample, not a complete list. Only the 50 shallowest pages of the crawl are checked against robots.txt. When pages_truncated is true, the crawl held more pages than that, so an empty or short list does not prove nothing else on the site is blocked.

The remaining fields:

Field	Type	Description
`domain`	String	The domain the signals describe.
`is_blocking_bots`	Boolean	The site turns crawlers away, derived from the share of pages that answered HTTP 403 across the whole crawl. This is a server-level verdict and is unrelated to `blocked_pages`, which is about robots.txt.
`is_spa`	Boolean	Most pages render client-side, so a crawler that does not run JavaScript reads little or nothing.
`blocked_pages[].blocked_by`	Array of strings	The AI assistant families robots.txt blocks from that page. Always non-empty for a listed page.

Limits and behavior

The response is null when the brand owns the requested domain but no crawl of it has finished yet.
Requesting a domain the brand does not own returns 404. An unknown brand_id also returns 404.
Signals come from the brand’s most recent finished crawl of the domain, not a live probe.
Pass domain as a bare hostname such as shop.example.com. A value with a path or port may not match any of the brand’s domains and returns 404.

Query API

Responses API

Agent Traffic API

AI Referrals API

Site Audit API