How to Build an Agent-Ready Website

Goal and scope

This guide walks through the concrete work of making a website agent-ready, level by level. It is the implementation companion to The Three Levels of an Agent-Ready Website, which defines the framework. The short version: agent-ready is not the same as AI SEO. A site that agents can find is not necessarily a site agents can understand, and a site agents can understand is not necessarily a site they can safely act on.

The three parts below follow that order:

Discoverable — crawlers and agents can find and fetch your pages.
Understandable — an agent can accurately interpret what your pages say.
Actionable — an agent can safely perform actions, not just read.

Work through the parts in order. Each one ends with a checklist you can run against your own site, and the Agent-Ready Website Checklist tool scores the result.

Prerequisites: you can edit your site’s HTML templates or framework config, and you can add files to the site root. No AI API, crawler infrastructure, or backend rewrite is needed for Parts 1 and 2.

Part 1 — Discoverable

An agent that cannot fetch your page cannot do anything else. Level 1 is mostly standard technical SEO, and that is fine: agents ride on the same rails as search crawlers.

Serve crawlable HTML

The single most common discoverability failure is content that only exists after client-side JavaScript runs. Google can render JavaScript, but rendering is deferred and not guaranteed, and many AI agents fetch raw HTML without executing scripts at all — see Google’s JavaScript SEO basics for the crawler side of this.

The fix is server-side rendering (SSR), static site generation, or at minimum pre-rendering for important routes. The test is simple: run curl -s https://yoursite.com/page | grep "your key sentence". If the sentence is not in the response, an agent fetching raw HTML will never see it. Navigation should use ordinary <a href> links, not click handlers, so agents can traverse the site.

Publish robots.txt

robots.txt tells crawlers what they may fetch. For a public content site, a permissive file is usually correct:

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

Two cautions. First, robots.txt is a convention crawlers choose to honor, not access control — Google documents this explicitly in its robots.txt introduction. Never use it to protect private data; use authentication. Second, if you block AI crawlers here, that is a legitimate policy choice, but know that it is the choice you are making.

Publish sitemap.xml

A sitemap lists your important URLs with modification dates so crawlers can discover changes efficiently. Minimal example:

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://example.com/articles/agent-ready-website/</loc>
    <lastmod>2026-07-26</lastmod>
  </url>
</urlset>

Most static frameworks generate this at build time. Submit it in Google Search Console and reference it from robots.txt, per Google’s sitemap documentation. A sitemap does not guarantee indexing; it removes discovery friction.

Fix canonical URLs and status codes

Every page should return a single canonical URL. If the same content answers at /page, /page/, and /page?utm=..., pick one and declare it with <link rel="canonical">, and redirect the others. Also audit status codes: pages should return 200, moved pages 301, gone pages 404 or 410. A common failure scenario is a site that returns 200 with a “not found” template — agents and crawlers index the error page as content, and your real 404 reporting becomes meaningless.

Build deliberate internal links

Internal links are how crawlers and agents map your topics. Link with descriptive anchor text from every page to its related pages, and make sure no important page is orphaned. On this site, for example, a mention of “agent-ready website” links to the framework definition rather than re-explaining it. Do the same on your site: define each concept once, link everywhere else.

Level 1 checklist

Key content present in raw HTML (curl test passes without JavaScript)
Navigation uses real <a href> links
robots.txt published, references the sitemap, blocks nothing unintentionally
sitemap.xml generated at build and submitted to Search Console
One canonical URL per page; duplicates redirect
Correct status codes (200 / 301 / 404 / 410)
No orphaned important pages

Part 2 — Understandable

A fetched page is not an understood page. Level 2 is about removing ambiguity so an agent can extract the right facts, entities, conditions, and limits.

Structure pages for extraction

One H1 per page, descriptive H2s, and a short summary near the top that answers the page’s question directly. An agent asked “what is this page about” should be able to answer from the first two paragraphs. Define specialized terms explicitly where they first appear, in plain sentences — “An idempotency key is a unique value that makes a retried request safe to repeat” — instead of assuming context. See What Is the Agentic Web? for why this interpretation layer matters to agents generally.

Add JSON-LD structured data

Structured data tells machines what kind of thing a page describes. Use JSON-LD, and follow the structured data guidelines: mark up only content actually visible on the page. A realistic Article example:

<script type="application/ld+json">
{
  "@context": "https://schema.org",
  "@type": "Article",
  "headline": "How to Build an Agent-Ready Website",
  "datePublished": "2026-05-02",
  "dateModified": "2026-07-26",
  "author": {
    "@type": "Organization",
    "name": "Learn Web4"
  }
}
</script>

Useful types by page: WebSite and Organization on the homepage, Article on articles, FAQPage on real FAQ sections, BreadcrumbList on navigation, DefinedTermSet on glossaries. Vocabulary reference: Schema.org.

Show dates, authorship, and sources

Agents need to judge freshness and provenance. Put a visible publication date, an updated date, and an author or organization on every content page, and mirror them in the JSON-LD. Cite primary sources inline next to factual claims. Pages that answer “when was this true, and who says so” get extracted with more confidence than anonymous, undated ones — that last claim is editorial interpretation, not an official ranking rule.

Use tables and lists for comparisons

When a page compares options, a table is the most reliably extractable format. Keep one fact per cell and plain-language column headers. If the same comparison lives only in prose or in an image, expect agents to paraphrase it badly or miss rows.

Consider llms.txt — proposed convention, varying support

llms.txt is a proposed convention: a markdown file at /llms.txt that summarizes your site and links to key pages for language-model consumers. Current status: tool support varies and broad adoption should not be assumed, so treat it as a cheap, low-risk extra signal, never a replacement for the work above. What Is llms.txt? covers the format and its limits in detail.

Level 2 checklist

One H1, descriptive H2s, summary in the first paragraphs
Specialized terms defined in plain sentences on first use
Valid JSON-LD on key page types; validates with no errors
Visible publish date, updated date, and authorship
Comparisons presented as tables or lists, not images
Conditions, prices, limits, and warnings stated as text
Optional: /llms.txt published as a bonus summary

Part 3 — Actionable

Level 3 applies when you want an agent to do something: create an order, book a slot, update a record. The bar here is much higher, because a misunderstood page wastes tokens while a misunderstood action wastes money or trust. This is a Learn Web4 framework position: reading and acting need different infrastructure.

Stable, documented APIs

Give agents a real API instead of forcing them to scrape forms and click buttons. “Stable” means versioned endpoints, documented request and response schemas, and a deprecation policy — breaking an undocumented endpoint silently is how automated integrations turn into incidents.

Authentication and permission scopes

Issue credentials per agent or integration, and scope them narrowly: a research agent gets read-only keys, a purchasing agent gets only the endpoints it needs. Scoped access is a core part of OAuth 2.0 (official standard), and it is the difference between “an agent leaked a key” and “an agent leaked a read-only key.”

Confirmation for high-risk actions

Classify your actions by reversibility. Reading data is low-risk; sending messages or creating draft orders is medium; charging money or deleting data is high-risk. High-risk actions should require explicit confirmation — a human approval step or a two-phase commit (create a pending action, confirm it separately). Risk note: an agent acting on stale or misread context will eventually attempt a high-risk action you did not intend; confirmation is the layer that catches it.

Idempotency keys

Agents retry. Networks fail, timeouts fire, and the agent cannot know whether the first attempt succeeded. Accept an idempotency key on mutating endpoints so a retried request is a no-op instead of a duplicate charge. Stripe’s API is the reference implementation of this pattern — see Stripe’s idempotent requests documentation:

curl https://api.example.com/v1/orders \
  -H "Authorization: Bearer sk_agent_..." \
  -H "Idempotency-Key: 9f2c1b7a-order-2026-07-26" \
  -d '{"item": "pro-plan", "quantity": 1}'

A retry with the same key returns the original result instead of creating a second order.

Structured errors, audit logs, and human handoff

Return machine-readable errors — a stable code, a human message, and a retryable flag — instead of HTML error pages, so agents can react correctly instead of guessing. Log every agent action with actor, timestamp, request, and outcome; when something goes wrong, the audit log is how you reconstruct it. Finally, give agents an explicit way out: a documented escalation path (support endpoint, webhook, or contact route) for cases outside their competence, so “I don’t know” becomes a handoff instead of a hallucinated action.

Level 3 checklist

Versioned API with documented schemas
Per-agent credentials with minimal scopes
Risk-tiered actions; high-risk actions require confirmation
Idempotency keys on all mutating endpoints
Structured error responses with stable codes
Audit log of every agent action
Documented human-handoff path

How to test each level with a real agent

Do not assume the implementation works — test it the way an agent experiences it.

Level 1 — fetch test. curl -s https://yoursite.com/key-page/ and confirm the headings, body text, and links are in the raw HTML. Then check https://yoursite.com/robots.txt and /sitemap.xml return 200.
Level 2 — extraction test. Give a real agent the page URL (or the raw HTML) and ask pointed questions: “What is the price, and what are its conditions?” “When was this page last updated?” “Define the main term on this page.” Score each answer against the page. Any wrong or missing answer is a Level 2 defect, not an agent defect.
Level 3 — sandbox action test. Against a staging environment or test-mode API, run one full action end-to-end: authenticate with a scoped key, perform a low-risk action, retry it with the same idempotency key (expect exactly one result), trigger a deliberate error (expect a structured error, not an HTML page), and verify the audit log entry exists.

We ran a version of the Level 1 and 2 tests against this site; the method and results are in How We Tested Whether AI Agents Can Read Learn Web4.

Monitoring after launch

Agent-readiness decays. Watch four signals:

Crawl and server logs. Which user agents fetch your pages, how often, and what status codes they get. Spikes of 404s or 5xxs for agent user agents mean something broke.
Agent traffic. Referrers from AI interfaces and known agent user agents, tracked separately from human traffic.
API error rates by error code. A rising rate of one structured error code usually means one integration or agent model is misusing an endpoint — the code tells you which.
Extraction drift. Re-run the extraction test quarterly and after any redesign or CMS change. Templates change silently; agents notice before you do.

When this guide does not apply

Pure content sites. If agents only need to read your site — a blog, docs, a portfolio — Parts 1 and 2 are the whole job. Skip Part 3; there is nothing to action.
Private or gated tools. If your product is behind login and you do not want outside agents acting in it, Level 3 applies only to agents you operate yourself.
Sites that deliberately block AI crawlers. That is a valid policy, but it makes most of Part 1 moot for those crawlers by definition.
Native apps with no web surface. The framework assumes web pages and web APIs; a closed app needs a different distribution story first.

Wrapping up

Build in order: make pages fetchable, then interpretable, then — only if you need it — actionable. Test each level the way an agent experiences it, and monitor the signals that tell you when something regresses. The Agent-Ready Website Checklist turns Parts 1–3 into a scored audit, and Web4 for Beginners gives the broader context if someone on your team is new to the topic.

FAQ

Do I need to rebuild my site to make it agent-ready?

Usually no. Levels 1 and 2 are mostly content, rendering, and metadata changes. Only Level 3, letting agents take actions, typically requires real backend work.

Does a pure content site need APIs to be agent-ready?

No. If agents only need to read your site, Levels 1 and 2 are enough. Level 3 applies when you want an agent to buy, book, submit, or modify something.

Is llms.txt worth adding?

It is cheap to add and harmless, but it is a proposed convention with varying tool support. Treat it as a bonus summary file, not a substitute for crawlable HTML and structured data.

How do I know whether an agent can actually read my site?

Fetch the page as plain HTML without JavaScript, then ask a real agent pointed extraction questions about facts, prices, dates, and definitions. Compare its answers against the page.

Changelog

Jul 26, 2026: Rewrote as the agent-ready implementation pillar, organized by the three levels, with code samples, per-level checklists, and testing and monitoring sections.
May 2, 2026: Initial publication.