Skip to main content

Overview

Agent Pages are kept fresh automatically: Scrunch checks your pages on a schedule and re-optimizes the ones that changed. Force Refresh lets you skip the wait. It refetches the Agent Pages you name from your live site and re-optimizes them now. If the result differs from what agents are served, it is published within minutes; if not, agents keep seeing the same page. Force Refresh only works on paths that already have an Agent Page; any other path returns not_found. To add a page, including one that isn’t in your sitemap, use Agent Pages Publish. Use it right after you publish a change on your site, so AI agents see the new version as soon as possible.

When to use Force Refresh

Use this API when you need to:
  • Push a content update, correction, or pricing change to agents right after it goes live on your site.
  • Refresh a set of pages after a deploy, from your CI or CMS publish hook.
  • Re-run optimization on a page you believe is out of date.
Force Refresh only refreshes pages that already have an active Agent Page. It doesn’t add or remove pages: to add one use Publish, and to remove one use Unpublish; see Which API do I need?.

Which API do I need?

A path on your site can carry two layers of content for AI agents, and the edge serves the first one it finds:
  1. Curated (Adaptive) content: pages you edit or deploy from the Scrunch dashboard. This includes a reviewed page deployed from Site Optimization and any Agent Page you edit in the Agent Pages content editor, which saves your edit as a curated page at that path.
  2. The Agent Page: the version Scrunch generates automatically from your live site, kept fresh by the scheduled checks.
Force Refresh and the Agent Pages Unpublish API act on the Agent Page layer only. The AXP Unpublish API acts on the curated layer only. A page that has both layers needs both unpublishes to be removed completely; after an AXP unpublish alone, agents receive the Agent Page at that path. How you add and remove Agent Pages depends on your setup. If your Site Optimization Pages table offers Serve on Agent Pages, you select pages one by one; an Agent Pages brand without Site Optimization also works page by page, through the Agent Pages APIs. Otherwise Agent Pages covers your whole site and is configured under Configure → Universal Optimizations. On a page-by-page setup, a site serves only its Agent Pages set. With Site Optimization, a site with an empty set serves no Agent Pages. Without it, a site serves every page until its first page is marked or published, and from then on only the set. On a brand without Site Optimization, Agent Pages Unpublish refuses to remove the last page of the set (conflict), because the site would go back to serving every page, including pages nobody chose. Agent Pages Publish has no such refusal: its first page narrows serving to the set you chose. The AXP Unpublish API never removes an Agent Page. A path with no curated content returns not_found, and on sites below AXP engine 2.1.0 the whole request is refused with 409.

Removing an Agent Page

For an immediate removal, call Agent Pages Unpublish from your CMS or CI. It works on either setup and keeps the page off through later crawls. In the dashboard, Stop serving on Agent Pages in the Site Optimization Pages table (page-by-page setup) or the Ignore List under Configure → Universal Optimizations (whole-site setup) does the same. If the page also has curated content, remove that with the AXP Unpublish API or archive it from the dashboard. Removing the page from your site also works, but it is slow. Return 404 or 410, or redirect it, and the scheduled freshness check retires the Agent Page when it next looks at the page. With the default schedule a frequently visited page is rechecked within about 7 days and a rarely visited page within about 30 days; a redirect within your domain needs two consecutive checks, and a redirect to another domain is acted on at the first. For anything time-sensitive, use Unpublish or the dashboard actions above and treat the scheduled check as cleanup.
Don’t force-refresh a page you have removed or redirected. The API returns queued for a path that now returns 404, then the refresh fails in the background and the old copy keeps serving until the scheduled check. Unpublish it instead. For a path that now redirects to another path, the page isn’t republished: the redirect is recorded as a scheduled check records it, the old copy keeps serving, and the page retires at a later scheduled check, usually within a day. A site with many redirects at once can take several days, because removals per site per day are capped. A redirect that only changes the host within the same domain (adding or dropping www., or another subdomain) and keeps the path still republishes.

Endpoint

The endpoint requires a bearer token with the optimize scope. See Authentication.

Find your site ID

A domain can have more than one site in Scrunch, and only the one serving Agent Pages accepts a refresh. The Site ID in your Agent Traffic setup instructions may be a different one: it identifies the site that sends Agent Traffic logs. To get the right ID, call List Sites and use the id of the site for your domain with agent_pages_active: true. If no site has it, Agent Pages aren’t serving for the brand and the refresh isn’t available. Any API key with access to the brand can call List Sites; no scope is required.
Response (abridged):
You can also copy it from the Site ID line on the Agent Pages page in the Scrunch app.

Request

Path rules

  • Each path is trimmed and can be up to 2,048 characters.
  • Send the public URL path, the one agents request. If your CMS path differs from the public URL (for example AEM /content/... versus /en/about.html), send the public URL. .model.json and other non-HTML exports never have an Agent Page and return not_found.
  • Paths must be site-relative. Full URLs (https://example.com/pricing) and protocol-relative URLs (//example.com/pricing) are rejected with a 422.
  • Paths are normalized the same way Agent Pages are stored: lowercased, with the trailing slash and query string dropped. Duplicates after normalization are merged, keeping the first.
  • / is the homepage.

Response

The endpoint returns 202 Accepted: refreshes run in the background after the response. The body holds one result per requested path, in request order, and the brand’s remaining quota.

Outcome values

Only queued paths count toward any limit.

Follow-up checks

Your origin or CDN can still serve the old version of a page for a short while after you publish a change. A refresh that runs in that window still fetches the old page, so agents keep seeing it until a follow-up check finds the new version. To catch the update once it goes live, Scrunch may check a queued page again a few times over the next hours, and republishes it only when what agents are served would change. The full schedule is four checks: the first a few minutes after your call, the last between four and seven hours after it, so that it lands after a typical CDN cache of up to four hours has expired. Each result’s recheck_checks tells you what Scrunch scheduled for that path: Follow-up checks use no refresh quota and don’t start a cooldown: only your own refreshes count toward the limits below. A newer refresh of the same path replaces the follow-up checks of an earlier one. To refresh a page once only, send "recheck": false:

Limits

The hourly and daily quotas are shared by all of a brand’s sites. Queued refreshes run a few at a time per site, so a batch of 100 completes over several minutes rather than instantly.

Example: refresh pages after a deploy

Response:

Errors

A path that is in cooldown, not found, or excluded doesn’t cause an error: the request still returns 202 with those outcomes.

Best practices

  • Refresh only what changed. Send the paths your deploy touched rather than your whole site; the scheduled checks already cover everything else.
  • Retry safely. Re-sending a path within 5 minutes returns cooldown and uses no quota.
  • Respect the quota. On rate_limited or 429, wait for quota.*.resets_at or the Retry-After header before sending more.
  • Send one request at a time per brand. Requests for the same brand are processed one at a time; a request sent while another is in progress gets 429 with Retry-After: 1. Batch paths into one request (up to 100) rather than sending many single-path requests in parallel.
  • Don’t refresh what you removed or redirected. The path is queued, then the refresh fails in the background: a 404 keeps the old copy serving, and a redirect to another path is recorded and retires the page at a later scheduled check, usually within a day, or over several days when a site has many redirects at once. See Removing an Agent Page.
  • Don’t poll for completion yet. queued means the refresh was accepted, not that it finished. Allow a few minutes for a batch to republish.