> ## Documentation Index
> Fetch the complete documentation index at: https://developers.scrunch.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Crawl Readiness API: Check AI Crawler Access to a Site

> Read whether AI crawlers can reach your brand's domains: robots.txt status, bot blocking, client-side rendering, and which pages robots.txt blocks.

## Overview

The Crawl Readiness API reports whether AI crawlers can reach one of your brand's domains, based on the brand's most recent finished crawl of that domain. It returns the same signals the AI Access card shows in the Scrunch dashboard.

Use it to check from your own backend whether a customer's site is reachable by AI crawlers before investing in AI visibility work, or to surface readiness inside a partner integration.

The endpoint is read-only. Nothing about how readiness is computed changes when you call it.

***

## What the Crawl Readiness API includes

* Whether a `robots.txt` file was found for the domain
* Whether the site turns crawlers away at the server level (based on HTTP 403 rates across the crawl)
* Whether the site renders mostly client-side, which limits what non-JavaScript crawlers can read
* A sample of pages that `robots.txt` blocks, with the AI assistant families (OpenAI, Meta, Perplexity, Gemini, Claude) blocked from each

***

## When to use the Crawl Readiness API

Use the Crawl Readiness API when you need to:

* Check whether a brand's domain is reachable by AI crawlers from a partner backend or CMS integration
* Flag robots.txt rules that block AI assistants from specific pages
* Detect sites that render client-side and may appear empty to non-JavaScript crawlers

For a detailed per-page audit score with content-quality checks, use the [Site Audit API](/api-reference/site-audit/overview) instead. To see which AI bots actually visit your pages, use the [Agent Traffic API](/api-reference/agent-traffic/overview).

***

## Endpoint

| Method | Path                          | Purpose                                                 |
| ------ | ----------------------------- | ------------------------------------------------------- |
| GET    | `/{brand_id}/crawl-readiness` | Crawl readiness signals for one of the brand's domains. |

The endpoint requires a bearer token with the `query` scope. See [Authentication](/getting-started/authentication).

Pass the optional `domain` query parameter to choose which of the brand's domains to describe. Omit it for the brand's primary website.

***

## Example: check the primary website

```bash theme={null}
curl -X GET \
  "https://api.scrunchai.com/v1/1234/crawl-readiness" \
  -H "Authorization: Bearer $SCRUNCH_API_TOKEN"
```

**Response:**

```json theme={null}
{
  "domain": "example.com",
  "robots_txt_present": true,
  "is_spa": false,
  "is_blocking_bots": false,
  "blocked_pages": [
    {
      "path": "/internal/pricing-drafts",
      "url": "https://example.com/internal/pricing-drafts",
      "blocked_by": ["OpenAI", "Perplexity"]
    }
  ],
  "pages_truncated": false
}
```

To describe a different domain the brand owns:

```bash theme={null}
curl -X GET \
  "https://api.scrunchai.com/v1/1234/crawl-readiness?domain=shop.example.com" \
  -H "Authorization: Bearer $SCRUNCH_API_TOKEN"
```

***

## Reading the response

Two fields are easy to misread:

* `robots_txt_present: false` means **permissive**, not blocked. With no robots.txt, crawlers are allowed by default.
* `blocked_pages` is a **sample, not a complete list**. Only the 50 shallowest pages of the crawl are checked against robots.txt. When `pages_truncated` is `true`, the crawl held more pages than that, so an empty or short list does not prove nothing else on the site is blocked.

The remaining fields:

| Field                        | Type             | Description                                                                                                                                                                                                 |
| ---------------------------- | ---------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `domain`                     | String           | The domain the signals describe.                                                                                                                                                                            |
| `is_blocking_bots`           | Boolean          | The site turns crawlers away, derived from the share of pages that answered HTTP 403 across the whole crawl. This is a server-level verdict and is unrelated to `blocked_pages`, which is about robots.txt. |
| `is_spa`                     | Boolean          | Most pages render client-side, so a crawler that does not run JavaScript reads little or nothing.                                                                                                           |
| `blocked_pages[].blocked_by` | Array of strings | The AI assistant families robots.txt blocks from that page. Always non-empty for a listed page.                                                                                                             |

***

## Limits and behavior

* The response is `null` when the brand owns the requested domain but no crawl of it has finished yet.
* Requesting a domain the brand does not own returns `404`. An unknown `brand_id` also returns `404`.
* Signals come from the brand's most recent finished crawl of the domain, not a live probe.
* Pass `domain` as a bare hostname such as `shop.example.com`. A value with a path or port may not match any of the brand's domains and returns `404`.


## Related topics

- [Get AI Crawl Readiness](/api-reference/crawl-readiness/get-ai-crawl-readiness.md)
- [Site Audit API: Score pages for AI search readiness](/api-reference/site-audit/overview.md)
- [Agent Traffic API: Monitor AI Bot Crawls](/api-reference/agent-traffic/overview.md)