Agent Pages Unpublish: Stop Serving Pages on Demand

Overview

Unpublish stops serving the Agent Pages you name. Each path is added to the site’s Ignore List, so it stays unpublished: later crawls, sitemap checks and page selections don’t bring it back. Its copy is then removed from the edge, within minutes or, with immediate: true, before the response. Use it from your CMS when a page is taken down, or when an Agent Page must stop being served right away. Unpublish acts on the Agent Page only. If the path also has curated (Adaptive) content, agents keep receiving that version after the unpublish, and the response doesn’t report it. Remove it with the AXP Unpublish API. See Which API do I need?.

What happens to a path

The path is added to the site’s Ignore List under Configure → Universal Optimizations, as a single page (its subpages are not affected).
The Agent Page is archived. On a page-by-page setup, the page is also no longer marked Serve on Agent Pages, so it stops counting against your Agent Pages budget.
Its copy is deleted from the edge. Without immediate, a background job does this within minutes of the response.

The API never changes a rule you set in the dashboard. If the path is itself an entry you added to the Priority List in the dashboard, either a page entry or a prefix entry, it returns conflict and nothing is written. Only that exact path conflicts: a page under a prefix entry (/blog/post under /blog) can still be unpublished, because the Ignore List wins over the Priority List. Unpublishing doesn’t change the site’s Agent Pages settings published to the edge, and doesn’t touch other pages. Curated (Adaptive) content at the same path keeps serving, and the response doesn’t report it.

How long a copy can still be served

Once its copy is deleted, the edge stops serving the Agent Page within about 60 seconds. A cache in front of the edge can still hold a copy it already served to an AI agent, and serves it until it expires:

Platform	Edge copy gone	Cached copies in front of the edge
Cloudflare	About 60 seconds after the delete	None by default. A cache you add in front of the worker can keep a copy for up to 1 hour.
Akamai	About 60 seconds after the delete	Your Akamai property can keep serving a copy it cached, for as long as its caching rules allow. Copies held for 2 to 4 hours have been observed.

Without immediate, add the few minutes the background job takes to delete the copy. Scrunch doesn’t purge your CDN cache; purge the URL yourself if it can’t wait.

Publishing the page again

An unpublished path stays off until you remove it from the Ignore List under Configure → Universal Optimizations. On a page-by-page setup, then select Serve on Agent Pages for it again in the Site Optimization Pages table. Selecting it while the path is still on the Ignore List doesn’t serve it.

Endpoint

Method	Path	Purpose
POST	`/v1/{brand_id}/sites/{site_id}/agent-pages/unpublish`	Unpublish a list of paths on one site.

The endpoint requires a bearer token with the optimize scope. See Authentication. Use the site ID that List Sites reports with agent_pages_active: true; see Find your site ID.

Request

Field	Type	Required	Description
`paths`	`string[]`	Yes	Site-relative paths to unpublish, for example `/pricing`. 1 to 100 paths per call.
`immediate`	`boolean`	No	Delete the edge copies before responding instead of within minutes. Default `false`.

Paths follow the same rules as Force Refresh: send the public URL path, site-relative, up to 2,048 characters. Paths are lowercased, the trailing slash and query string are dropped, and duplicates are merged. / is the homepage. See Path rules.

Response

The endpoint returns 200 OK with one result per requested path, in request order, and the brand’s remaining unpublish quota.

Field	Type	Description
`results[].path`	`string`	The normalized path this outcome applies to.
`results[].outcome`	`string`	What happened to this path. See the table below.
`results[].retry_after`	`string \| null`	For `cooldown`, when this path can be unpublished again (ISO 8601, UTC).
`results[].edge`	`string \| null`	With `immediate: true`, for every requested path whose Agent Page is no longer served but whose copy was still at the edge, whatever its outcome: the result of the edge delete. See Edge results.
`quota.hourly` / `quota.daily`	`object`	The brand’s two unpublish quota windows, each with `limit`, `remaining`, and `resets_at`.

Outcome values

Outcome	Meaning	What to do
`unpublished`	The Agent Page was being served at the edge. It is on the Ignore List now and its edge copy is removed.	Nothing.
`already_unpublished`	No Agent Page was being served at this path: the Ignore List already covers it, the Agent Page was already retired, or it was never published to the edge. The path is added to the Ignore List if it wasn’t on it, so it can’t be published later.	Nothing. A retry of an earlier unpublish returns this.
`not_found`	The site has no Agent Page at this path, and none is scheduled. Nothing is written.	Check that it is the public URL path, not a CMS or export URL. If the path serves curated content, use the AXP Unpublish API.
`conflict`	The path is itself an entry you added to the Priority List in the dashboard, either a page entry or a prefix entry. Pages under a prefix entry don’t conflict. This includes an entry added while the request was running. On a brand without Site Optimization, when the request would remove every remaining Priority List entry on the site, those paths also return `conflict`, because a site with no entries serves every page. Nothing is written and no quota is used.	Change the Priority List under Configure → Universal Optimizations, then retry.
`cooldown`	This path was taken off the Ignore List and unpublished again within 5 minutes of its last unpublish. A plain resend returns `already_unpublished` instead.	Retry after `retry_after`.
`rate_limited`	The brand’s hourly or daily quota ran out partway through this batch.	Retry after `quota.*.resets_at`.

Only unpublished and already_unpublished paths count toward the quota, and only when the path was not already on the Ignore List. conflict paths never do.

Edge results

Reported only with immediate: true, for every requested path whose Agent Page is no longer served but whose copy was still at the edge, whatever its outcome. That includes an already_unpublished path whose copy an earlier call left behind, so an immediate retry cleans it up. null means there was no copy to delete. A page this call left serving (for example conflict, cooldown or rate_limited on a live page) is never deleted and reports null. For an unpublished path, the page stays unpublished whatever edge says: it is on the Ignore List and won’t be republished.

`edge`	Meaning
`deleted`	The edge copy is gone.
`failed`	The edge delete failed for this path. The unpublish still holds, and the background job retries the delete within minutes. If the delete fails as a whole, every path with an edge copy reports `failed`.
`deferred`	The site has no CDN platform configured, so no edge delete was attempted.

Limits

Limit	Value	When exceeded
Paths per request	100	`422`; nothing is unpublished.
Per path	1 unpublish every 5 minutes	A path taken off the Ignore List and unpublished again within 5 minutes returns `cooldown`. A plain resend returns `already_unpublished`.
Per brand, hourly	100 paths per rolling hour	Extra paths return `rate_limited`; `429` if nothing could be unpublished.
Per brand, daily	1,000 paths per rolling 24 hours	Extra paths return `rate_limited`; `429` if nothing could be unpublished.

The quotas are shared by all of a brand’s sites and are separate from the Force Refresh quotas. For a bulk cleanup, use the Ignore List in the dashboard, which also accepts a CSV upload.

Example: take a page down right away

curl -X POST \
  "https://api.scrunchai.com/v1/1234/sites/01JEXAMPLE00000000000000/agent-pages/unpublish" \
  -H "Authorization: Bearer $SCRUNCH_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "paths": [
      "/pricing",
      "/old-campaign",
      "/careers"
    ],
    "immediate": true
  }'

Response:

{
  "results": [
    { "path": "/pricing", "outcome": "unpublished", "retry_after": null, "edge": "deleted" },
    { "path": "/old-campaign", "outcome": "already_unpublished", "retry_after": null, "edge": null },
    { "path": "/careers", "outcome": "not_found", "retry_after": null, "edge": null }
  ],
  "quota": {
    "hourly": { "limit": 100, "remaining": 99, "resets_at": "2026-10-28T15:02:41Z" },
    "daily": { "limit": 1000, "remaining": 987, "resets_at": "2026-10-29T09:15:03Z" }
  }
}

Errors

Status	When
`401`	The API key is missing or invalid.
`403`	The API key lacks the `optimize` scope or access to the brand.
`404`	No site with the supplied ULID exists for the brand, or Agent Pages aren’t enabled for the brand.
`409`	Agent Pages aren’t active on this site. Check that `site_id` is the site List Sites reports with `agent_pages_active: true`. Nothing is unpublished and no quota is used.
`422`	The request body failed validation (empty `paths`, more than 100 entries, an empty path, a path over 2,048 characters, or a full URL). A malformed or duplicated `Content-Type` header also returns `422`; send exactly `Content-Type: application/json`.
`429`	The brand’s hourly or daily unpublish quota is used up and nothing could be unpublished, or another unpublish request for the same brand is still being processed. The `Retry-After` header gives the seconds to wait (`1` for a request in progress).
`5xx`	An unexpected server error. The paths may already be unpublished; retrying is safe.

A path that is in cooldown, not found, or in conflict doesn’t cause an error: the request still returns 200 with those outcomes. A failed edge delete doesn’t either: those paths report edge: "failed".

Best practices

Retry safely. Re-sending a path returns already_unpublished and changes nothing on the Ignore List. With immediate: true, a retry also deletes any edge copy still left from the earlier call and reports it in edge.
Use immediate when it matters. It makes the request slower, so leave it off for routine takedowns; the background job removes the copy within minutes.
Send one request at a time per brand. A request sent while another unpublish for the brand is in progress gets 429 with Retry-After: 1. Batch paths into one request (up to 100).
Remove curated content too. If the dashboard shows an edited or deployed version at the path, unpublish it with the AXP Unpublish API as well, or agents keep receiving it after the Agent Pages unpublish.

Measure

Optimize

Deliver

Configure

Deprecated

Agent Pages Unpublish: Stop Serving Pages on Demand

Overview

What happens to a path

How long a copy can still be served

Publishing the page again

Endpoint

Request

Response

Outcome values

Edge results

Limits

Example: take a page down right away

Errors

Best practices

​Overview

​What happens to a path

​How long a copy can still be served

​Publishing the page again

​Endpoint

​Request

​Response

​Outcome values

​Edge results

​Limits

​Example: take a page down right away

​Errors

​Best practices

Overview

What happens to a path

How long a copy can still be served

Publishing the page again

Endpoint

Request

Response

Outcome values

Edge results

Limits

Example: take a page down right away

Errors

Best practices