> ## Documentation Index > Fetch the complete documentation index at: https://developers.scrunch.com/llms.txt > Use this file to discover all available pages before exploring further. # Get AI Crawl Readiness > Whether AI crawlers can reach one of the brand's domains, based on the brand's most recent finished crawl of it. These are the same signals the AI Access card shows in the Scrunch dashboard. Pass `domain` to choose which of the brand's domains to describe; omit it for the primary website. The response is `null` when the brand owns the domain but no crawl of it has finished yet; requesting a domain the brand does not own returns `404`. Reading the fields: - `robots_txt_present`: whether a robots.txt was found. **False means permissive**, not blocked: with no robots.txt, crawlers are allowed by default. - `is_blocking_bots`: the site turns crawlers away, derived from the share of pages that answered HTTP 403 across the whole crawl. This is a server-level verdict and is unrelated to `blocked_pages`, which is about robots.txt. - `is_spa`: most pages render client-side, so a crawler that does not run JavaScript reads little or nothing. - `blocked_pages`: pages robots.txt blocks, with the AI assistant families blocked from each. **This is a sample, not a complete list**: only the 50 shallowest pages of the crawl are checked. `pages_truncated` is true when the crawl held more than that, so an empty or short list does not mean nothing else on the site is blocked. ## OpenAPI ````yaml /api-reference/openapi.json get /{brand_id}/crawl-readiness openapi: 3.1.0 info: title: Scrunch Data API version: 0.1.0 servers: - url: https://api.scrunchai.com/v1 security: [] paths: /{brand_id}/crawl-readiness: get: tags: - crawl-readiness summary: Get AI Crawl Readiness description: >- Whether AI crawlers can reach one of the brand's domains, based on the brand's most recent finished crawl of it. These are the same signals the AI Access card shows in the Scrunch dashboard. Pass `domain` to choose which of the brand's domains to describe; omit it for the primary website. The response is `null` when the brand owns the domain but no crawl of it has finished yet; requesting a domain the brand does not own returns `404`. Reading the fields: - `robots_txt_present`: whether a robots.txt was found. **False means permissive**, not blocked: with no robots.txt, crawlers are allowed by default. - `is_blocking_bots`: the site turns crawlers away, derived from the share of pages that answered HTTP 403 across the whole crawl. This is a server-level verdict and is unrelated to `blocked_pages`, which is about robots.txt. - `is_spa`: most pages render client-side, so a crawler that does not run JavaScript reads little or nothing. - `blocked_pages`: pages robots.txt blocks, with the AI assistant families blocked from each. **This is a sample, not a complete list**: only the 50 shallowest pages of the crawl are checked. `pages_truncated` is true when the crawl held more than that, so an empty or short list does not mean nothing else on the site is blocked. operationId: getCrawlReadiness parameters: - name: brand_id in: path required: true schema: type: integer title: Brand Id description: The unique identifier for the brand. description: The unique identifier for the brand. - name: domain in: query required: false schema: anyOf: - type: string minLength: 1 - type: 'null' title: Domain description: >- Which of the brand's domains to describe. Omit for the primary website. description: >- Which of the brand's domains to describe. Omit for the primary website. responses: '200': description: >- Successful Response. `null` when the brand owns the domain but no crawl of it has finished yet. content: application/json: schema: anyOf: - $ref: '#/components/schemas/CrawlReadinessModel' - type: 'null' title: Response Getcrawlreadiness '404': description: Brand not found, or the brand does not own the requested domain. '422': description: Validation Error content: application/json: schema: $ref: '#/components/schemas/HTTPValidationError' security: - HTTPBearer: - query components: schemas: CrawlReadinessModel: properties: domain: type: string title: Domain description: The domain the readiness signals describe. robots_txt_present: type: boolean title: Robots Txt Present description: >- Whether a robots.txt was found for the domain. False means permissive, not blocked: with no robots.txt, crawlers are allowed by default. is_spa: type: boolean title: Is Spa description: >- Most pages render client-side, so a crawler that does not run JavaScript reads little or nothing. is_blocking_bots: type: boolean title: Is Blocking Bots description: >- The site turns crawlers away at the server level, derived from the share of pages that answered HTTP 403 across the whole crawl. blocked_pages: items: $ref: '#/components/schemas/BlockedPageModel' type: array title: Blocked Pages description: >- Pages robots.txt blocks from at least one AI assistant family. A sample from the 50 shallowest pages of the crawl, not a complete list. pages_truncated: type: boolean title: Pages Truncated description: >- True when the crawl held more pages than the 50-page scan cap, so blocked_pages is a sample. type: object required: - domain - robots_txt_present - is_spa - is_blocking_bots - blocked_pages - pages_truncated title: CrawlReadinessModel description: >- AI crawl readiness signals for one of the brand's domains, from its most recent finished crawl. HTTPValidationError: properties: detail: items: $ref: '#/components/schemas/ValidationError' type: array title: Detail type: object title: HTTPValidationError BlockedPageModel: properties: path: type: string title: Path description: The page's path on the site, for example `/pricing`. url: type: string title: Url description: The page's full URL. blocked_by: items: type: string type: array title: Blocked By description: >- The AI assistant families (OpenAI, Meta, Perplexity, Gemini, Claude) that robots.txt blocks from this page. Always non-empty for a page in this list. type: object required: - path - url - blocked_by title: BlockedPageModel description: A page robots.txt blocks from one or more AI assistant families. ValidationError: properties: loc: items: anyOf: - type: string - type: integer type: array title: Location msg: type: string title: Message type: type: string title: Error Type input: title: Input ctx: type: object title: Context type: object required: - loc - msg - type title: ValidationError securitySchemes: HTTPBearer: type: http scheme: bearer ```` ## Related topics - [Crawl Readiness API: Check AI Crawler Access to a Site](/api-reference/crawl-readiness/overview.md) - [Site Audit API: Score pages for AI search readiness](/api-reference/site-audit/overview.md) - [Sitemap API: Pages, Audit Scores, AI Metrics](/api-reference/sitemap/overview.md)