Skip to main content
GET
Get AI Crawl Readiness

Authorizations

Authorization
string
header
required

Bearer authentication header of the form Bearer <token>, where <token> is your auth token.

Path Parameters

brand_id
integer
required

The unique identifier for the brand.

Query Parameters

domain
string | null

Which of the brand's domains to describe. Omit for the primary website.

Minimum string length: 1

Response

CrawlReadinessModel · object | null

Successful Response. null when the brand owns the domain but no crawl of it has finished yet.

AI crawl readiness signals for one of the brand's domains, from its most recent finished crawl.

domain
string
required

The domain the readiness signals describe.

robots_txt_present
boolean
required

Whether a robots.txt was found for the domain. False means permissive, not blocked: with no robots.txt, crawlers are allowed by default.

is_spa
boolean
required

Most pages render client-side, so a crawler that does not run JavaScript reads little or nothing.

is_blocking_bots
boolean
required

The site turns crawlers away at the server level, derived from the share of pages that answered HTTP 403 across the whole crawl.

blocked_pages
BlockedPageModel · object[]
required

Pages robots.txt blocks from at least one AI assistant family. A sample from the 50 shallowest pages of the crawl, not a complete list.

pages_truncated
boolean
required

True when the crawl held more pages than the 50-page scan cap, so blocked_pages is a sample.