Agent Pages Publish: Add Pages on Demand

Overview

Publish makes the pages you name Agent Pages now. Each path is fetched from your live site, optimized and served to AI agents, usually within minutes. It works for pages your sitemap doesn’t list too. Use it from your CMS when a page goes live, or to bring back a page you took down with Agent Pages Unpublish. Publish acts on the Agent Page only. If the path also has curated (Adaptive) content, agents keep receiving that version, and the response doesn’t report it. See Which API do I need?.

What happens to a path

On a page-by-page setup, the page is marked Serve on Agent Pages, exactly as if you selected it in the Site Optimization Pages table, and it counts against your Agent Pages budget. On a whole-site (legacy) setup, nothing is selected, the page isn’t added to any list, and no budget applies: the site keeps serving every page, and the path joins the scheduled checks.
If Agent Pages Unpublish put the path on the Ignore List, that entry is removed. An Ignore List entry or path rule you added in the dashboard is never overridden: the path returns excluded.
A background job fetches the page from your site, optimizes it and publishes it to the edge. The page is served once that job finishes, usually within minutes.

On a page-by-page setup, the site serves only its Agent Pages set: the pages marked in the dashboard or published through this API. With Site Optimization, a site with an empty set serves no Agent Pages. Without it, a site serves every page until its first page is marked or published, and from then on only the set: the first publish stops serving every page outside the set, and Unpublish refuses to remove the last one (conflict).

`queued` is not yet served

queued means the job was sent, not that the page is live. If the send itself fails, the page is still picked up by the hourly scheduled check, usually within an hour. If your site doesn’t serve the page (for example it answers 404), the job fails. The page stays selected and counts against your Agent Pages budget, but it isn’t served until a scheduled check finds it live (about every 30 days) or you send it again. The response can’t tell you, because it returns before the fetch, and there is no status endpoint. Publish a page only once it is live on your site. If your site redirects the path to another path, the page isn’t published either, and the redirect is recorded as a scheduled check records it; a redirect that only changes the host within the same domain (adding or dropping www., or another subdomain) and keeps the path still publishes. To retry a failed publish, send the path again after its 5-minute cooldown. A queued path can also carry warnings about its history:

Warning	Meaning
`Retired (<reason>), last checked <date>: it serves only once the origin answers 200.`	The page was retired earlier. It serves again only once your site answers `200` for it. `<reason>` is `http_gone` (your site answered `404` or `410`), `sitemap_absent` (your sitemap dropped it), `non_html` (it isn’t an HTML page) or `admin_retired` (retired by Scrunch). `<date>` is the page’s last scheduled check, not the day it was retired. Either part is left out when unknown, down to `Retired: it serves only once the origin answers 200.`
`Recent checks saw this path redirect to <target>: it serves once a check sees the page answer for itself.`	Recent checks saw the path redirect. It serves once a check sees the page answer without redirecting. `<target>` reads `another URL` when no target was recorded.

Pages your sitemap doesn’t list

Each published path reports in_sitemap. A queued page your sitemap doesn’t list is published as a manual Agent Page and carries the warning Not in the site's sitemap…. An already_published page never carries it: there, in_sitemap: false is informational. The sitemap check never removes a manual Agent Page. It stays served until your site answers 404 or redirects it at a scheduled check, or until you call Agent Pages Unpublish. A queued page your sitemap lists is managed by the sitemap, as before: if the sitemap later drops it, the sitemap check can retire it. Publish doesn’t change how an already_published page is managed.

Paths your CDN doesn’t route to the edge

Agents only receive Agent Pages for paths your CDN sends to the Scrunch edge worker. If your CDN rules cover part of the site only (for example /us-en/*), a path outside them is published and stored, but never served. The API can’t detect this, so check your CDN routing before publishing outside it.

Endpoint

Method	Path	Purpose
POST	`/v1/{brand_id}/sites/{site_id}/agent-pages/publish`	Publish a list of paths on one site.

The endpoint requires a bearer token with the optimize scope. See Authentication. Use the site ID that List Sites reports with agent_pages_active: true; see Find your site ID.

Request

Field	Type	Required	Description
`paths`	`string[]`	Yes	Site-relative paths to publish, for example `/pricing`. 1 to 100 paths per call.

Paths follow the same rules as Force Refresh: send the public URL path, site-relative, up to 2,048 characters. Paths are lowercased, the trailing slash and query string are dropped, and duplicates are merged. / is the homepage. See Path rules. One difference: .model.json and other non-HTML files return invalid_path here, not not_found. The page is fetched with the spelling you sent first, so a case-sensitive site gets the URL it expects. The query string is dropped from the fetched URL too, so a tracking or session parameter is never fetched.

Response

The endpoint returns 200 OK with one result per requested path, in request order, and the brand’s remaining publish quota.

Field	Type	Description
`results[].path`	`string`	The normalized path this outcome applies to.
`results[].outcome`	`string`	What happened to this path. See the table below.
`results[].retry_after`	`string \| null`	For `cooldown`, when this path can be published again (ISO 8601, UTC).
`results[].in_sitemap`	`boolean \| null`	For `queued` and `already_published` paths, whether your sitemap lists the page. `null` for other outcomes.
`results[].warnings`	`string[]`	Why the page may not be served soon, or may be removed later. Empty when there is nothing to report. See `queued` is not yet served and Pages your sitemap doesn’t list.
`quota.hourly` / `quota.daily`	`object`	The brand’s two publish quota windows, each with `limit`, `remaining`, and `resets_at`.

Outcome values

Outcome	Meaning	What to do
`queued`	The page is an Agent Page now and its fetch was sent. It is served once the fetch and optimization finish. If the send fails, the hourly scheduled check picks the page up, usually within an hour.	Nothing. If it isn’t served within the hour, check that the page is live and send it again.
`already_published`	The page is already being served. Nothing was fetched. On a page-by-page setup, a served page that wasn’t marked yet is added to the set, if your budget allows, and counts against your Agent Pages budget.	To pick up a change on your site, use Force Refresh.
`excluded`	An Ignore List entry or path rule from the dashboard blocks this path. On a whole-site (legacy) setup with Priority List entries, a path outside them is also `excluded`. The API never overrides the dashboard.	Remove the entry under Configure → Universal Optimizations, then retry.
`over_budget`	Your organization’s Agent Pages budget is full (page-by-page setups only). In a batch, the first paths that fit are queued. Nothing was written for this path.	Stop serving another page, or ask your account team for more, then retry.
`invalid_path`	Not a web page (a file such as `.pdf`, `.json` or `.model.json`), or a path the page list can’t store.	Send the public URL path of an HTML page.
`cooldown`	This path was published in the last 5 minutes.	Retry after `retry_after`.
`rate_limited`	The brand’s hourly or daily quota ran out partway through this batch.	Retry after `quota.*.resets_at`.

Only queued paths count toward the quota.

Limits

Limit	Value	When exceeded
Paths per request	100	`422`; nothing is published.
Per path	1 publish every 5 minutes	That path returns `cooldown`.
Per brand, hourly	500 paths per rolling hour	Extra paths return `rate_limited`; `429` if nothing could be published.
Per brand, daily	5,000 paths per rolling 24 hours	Extra paths return `rate_limited`; `429` if nothing could be published.
Agent Pages budget	Your organization’s plan (page-by-page setups only)	Extra paths return `over_budget`.

The quotas are shared by all of a brand’s sites and are separate from the Force Refresh and Unpublish quotas.

Example: publish new pages

curl -X POST \
  "https://api.scrunchai.com/v1/1234/sites/01JEXAMPLE00000000000000/agent-pages/publish" \
  -H "Authorization: Bearer $SCRUNCH_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "paths": [
      "/pricing",
      "/launch/fall-event",
      "/about",
      "/brochure.pdf"
    ]
  }'

Response:

{
  "results": [
    { "path": "/pricing", "outcome": "queued", "retry_after": null, "in_sitemap": true, "warnings": [] },
    {
      "path": "/launch/fall-event",
      "outcome": "queued",
      "retry_after": null,
      "in_sitemap": false,
      "warnings": [
        "Retired (http_gone), last checked 2026-10-20: it serves only once the origin answers 200.",
        "Not in the site's sitemap. Published as a manual Agent Page: it is not removed by the sitemap check, only by an origin 404 or redirect, or the unpublish API."
      ]
    },
    { "path": "/about", "outcome": "already_published", "retry_after": null, "in_sitemap": true, "warnings": [] },
    { "path": "/brochure.pdf", "outcome": "invalid_path", "retry_after": null, "in_sitemap": null, "warnings": [] }
  ],
  "quota": {
    "hourly": { "limit": 500, "remaining": 498, "resets_at": "2026-10-28T15:02:41Z" },
    "daily": { "limit": 5000, "remaining": 4987, "resets_at": "2026-10-29T09:15:03Z" }
  }
}

Errors

Status	When
`401`	The API key is missing or invalid.
`403`	The API key lacks the `optimize` scope or access to the brand.
`404`	No site with the supplied ULID exists for the brand, or Agent Pages aren’t enabled for the brand.
`409`	Agent Pages aren’t active on this site. Check that `site_id` is the site List Sites reports with `agent_pages_active: true`. Nothing is published and no quota is used.
`422`	The request body failed validation (empty `paths`, more than 100 entries, an empty path, a path over 2,048 characters, or a full URL). A malformed or duplicated `Content-Type` header also returns `422`; send exactly `Content-Type: application/json`.
`429`	The brand’s hourly or daily publish quota is used up and nothing could be published, or another publish request for the same brand is still being processed. The `Retry-After` header gives the seconds to wait (`1` for a request in progress).

A path that is excluded, over budget, invalid or in cooldown doesn’t cause an error: the request still returns 200 with those outcomes.

Best practices

Publish once the page is live. The fetch runs right after the response; a page your site doesn’t serve yet fails silently. Read warnings on queued paths: a retired or redirecting page waits for your site to answer for it.
Retry safely. Re-sending a served path returns already_published and fetches nothing. On a page-by-page setup it is also added to the set, if it wasn’t already and your budget allows. Re-sending a queued path within 5 minutes returns cooldown, or already_published once it is served; after that it fetches the page again, which is how you retry a failed publish.
Send one request at a time per brand. A request sent while another publish for the brand is in progress gets 429 with Retry-After: 1. Batch paths into one request (up to 100).
Publish after Unpublish. A path taken down with Agent Pages Unpublish can be published again with this API. A path you put on the Ignore List in the dashboard can’t: remove it there first.
Check curated content. If the dashboard shows an edited or deployed version at the path, agents keep receiving that version instead of the published Agent Page.

Measure

Optimize

Deliver

Configure

Deprecated

Agent Pages Publish: Add Pages on Demand

Overview

What happens to a path

`queued` is not yet served

Pages your sitemap doesn’t list

Paths your CDN doesn’t route to the edge

Endpoint

Request

Response

Outcome values

Limits

Example: publish new pages

Errors

Best practices

​Overview

​What happens to a path

​queued is not yet served

​Pages your sitemap doesn’t list

​Paths your CDN doesn’t route to the edge

​Endpoint

​Request

​Response

​Outcome values

​Limits

​Example: publish new pages

​Errors

​Best practices

Overview

What happens to a path

`queued` is not yet served

Pages your sitemap doesn’t list

Paths your CDN doesn’t route to the edge

Endpoint

Request

Response

Outcome values

Limits

Example: publish new pages

Errors

Best practices