How We Tested Whether AI Agents Could Read LearnWeb4

Proposed test design. No experimental result is claimed.

This article describes an experiment Learn Web4 will run on its own pages, published in full before the first test question is ever asked. Nothing below reports a finding. Where the experiment eventually produces numbers, this page will be updated with a dated results section and links to raw transcripts; until then, every claim about outcomes is a hypothesis, not a measurement.

Why publish the design first? Because the site itself argues that agent-readable content should be verifiable. If we tell site owners that structure helps AI agents, we should be able to show the test that would prove or disprove it — and the test should be fixed before we see any results, not tuned afterward.

Research question

Does page structure affect how accurately AI agents extract facts from learnweb4.com pages?

More precisely: given the same underlying information presented three different ways, do agents make fewer extraction errors on structured versions of a page than on prose-only versions?

This sits squarely at Level 2 of the three levels of an agent-ready website — Understandable. Level 1 (Discoverable) is about being found and fetched; Level 3 (Actionable) is about safe actions. This experiment only probes the middle layer: accurate interpretation.

The framing matters. Agent-ready is not the same as AI SEO. A site that agents can find is not necessarily a site agents can understand, and a site agents can understand is not necessarily a site they can safely act on. This test targets understanding only — not rankings, not citations, not actions. For the broader distinction, see Agent-Ready Is Not AI SEO.

Hypotheses

Three explicit hypotheses, registered before any run:

H1 (Hypothesis): Agents extract discrete facts (dates, definitions, list items) more accurately from a page with headings, a summary, and a table than from the same content as prose-only.
H2 (Hypothesis): Adding JSON-LD structured data (a Schema.org vocabulary, consumed by agents that parse it) produces a measurable additional gain over headings and tables alone — or it does not. Either outcome is worth publishing; H2 is deliberately stated as an open comparison, not an assumed win.
H3 (Hypothesis): Errors on the prose-only version cluster in a predictable place: relational facts (which item belongs to which category, which date applies to which version) rather than simple lookups.

Null results are publishable. If structure makes no measurable difference, that is a finding about our rubric, our samples, or the agents — and it changes what we recommend.

Test environment

The run must pin down everything that could drift between today and the day we execute. Placeholders are marked TODO and will be filled at run time, then frozen:

# Test environment record (to be completed at run time)
run_date:            TODO   # single calendar day for the full run
agents:              TODO   # e.g. specific model + tool versions, recorded verbatim
retrieval_method:    TODO   # e.g. direct URL fetch vs. pasted HTML, one method only
context_settings:    TODO   # temperature / reasoning settings if exposed, else "default"
site_snapshot:       TODO   # git commit hash of the learnweb4 repo at run time
question_order:      randomized per session, seed recorded

(Limitation) Model drift is real: an agent version tested in one month may behave differently the next. That is why the environment record includes exact version strings and the site’s git commit hash — the goal is a rerun-able configuration, not a timeless one.

Page samples: three versions of the same content

One topic from this site (a glossary-style explainer with dates, definitions, and categorized items) will be rendered three ways on staging URLs, identical in information content:

Version	Name	What it contains
A	Prose-only	Plain paragraphs. No headings, no summary, no lists, no table, no structured data.
B	Structured	Same text reorganized: clear H2/H3 headings, a short page summary, a comparison table, explicit definition lists.
C	Structured + JSON-LD	Version B plus JSON-LD markup describing the article and its key entities.

Version B mirrors what Google documents as structured, well-organized page content — a human-facing structure first, machine annotations second. Version C adds the machine-readable layer.

The test flow, per agent session:

            ┌──────────────────────┐
            │  staging URL (A|B|C) │
            └──────────┬───────────┘
                       │ single retrieval
                       ▼
            ┌──────────────────────┐
            │  agent session       │  fresh session, no memory of prior runs
            └──────────┬───────────┘
                       │ 10 fixed test questions, one at a time
                       ▼
            ┌──────────────────────┐
            │  full transcript     │  saved verbatim, unedited
            └──────────┬───────────┘
                       │ field-level scoring against answer key
                       ▼
            ┌──────────────────────┐
            │  accuracy per field  │  published with rubric + transcripts
            └──────────────────────┘

Each agent sees each version in a fresh session. Nine sessions per agent (3 versions × 3 repetitions) to catch variance, assuming run cost stays reasonable; the final repetition count will be recorded in the environment section.

Test questions

Ten questions per page version, split into three difficulty tiers. Examples (the final list will be frozen at run time):

Lookup questions (single fact):

“What date does the page give as the initial publication date?”
“What is the one-sentence definition of the main term on this page?”
“How many items are in the page’s category list?”

Relational questions (fact tied to context): 4. “Which items does the page place in the second category, and which in the third?” 5. “Which version of the tool does the page say the instructions were last tested with?” 6. “According to the page, what is explicitly not covered by the main recommendation?”

Boundary questions (limits, conditions, negatives): 7. “What limitation does the page state about its own claim?” 8. “Does the page say the convention is officially required? Quote the relevant wording.” 9. “Under what condition does the page say the advice does not apply?” 10. “What does the page say will happen if the recommended step is skipped?”

Boundary questions matter most. H3 predicts these are where prose-only pages fail: a paragraph that buries a “does not apply when…” clause is easy for a fast-reading agent to drop.

Success criteria and scoring rubric

Scoring is field-level, not vibe-level. Every question has a pre-written answer key extracted from the source content before the run. Each answer is decomposed into fields, and each field is scored independently.

Score	Code	Definition
2	Correct	Field content matches the answer key exactly in substance; wording may differ.
1	Partial	Field is present but imprecise: wrong qualifier, swapped category, vague paraphrase that changes meaning.
0	Wrong	Field contradicts the key, or is a plausible-sounding invention not present on the page.
N/A	Missing	Field is absent from the answer (scored separately from wrong, so silence is visible).

Two derived metrics per page version:

Field accuracy: (sum of field scores) / (2 × total fields). Reported per question tier.
Fabrication rate: share of answers containing at least one score-0 invented field. An agent that says “I don’t know” beats one that confidently invents, and the rubric is built to reward that.

(Editorial interpretation) We treat a fabricated fact as the worst outcome — worse than silence — because a reader relying on the agent’s summary acts on it. That weighting is our editorial choice, stated here so others can rescore the transcripts differently.

How results will be published

Every transcript saved verbatim, unedited, linked from this article.
The frozen answer key and question list published alongside.
Scores computed with the rubric above, per agent, per version, per tier.
Any deviation from this pre-registered design disclosed in the results section with a reason.
The changelog on this page records the date results were added.

The current run of this site also follows a Proposed convention worth noting: llms.txt is a proposed convention for AI-readable site summaries, and support varies by tool — broad adoption should not be assumed. The experiment does not test llms.txt; it tests page structure.

What we changed on this site

The experiment exists because of a real content upgrade happening on learnweb4.com right now. This section documents the change itself — explicitly without claiming any measured outcome from it.

Before the upgrade, articles shared a uniform skeleton, repeated the same beginner explanations across posts, and had no visible record of when content changed. After:

Three-level framework rollout. Articles now map their claims onto a Learn Web4 framework — Discoverable, Understandable, Actionable — instead of scattering loosely related advice. The pillar statements live in The Three Levels of an Agent-Ready Website and How to Build an Agent-Ready Website; individual articles link to them rather than restating basics.
Per-type CTAs. Tutorials, comparisons, concepts, and experiments each carry a call-to-action matched to reader intent, instead of one generic footer.
Changelog and last-tested fields. Articles now carry a changelog in front matter (visible history), and tutorials carry lastTestedAt plus a recorded test environment — so a reader can tell a tested procedure from an untested one.
Removed duplicated boilerplate. Repeated “what is Web4 / why this matters” paragraphs were cut from individual articles and consolidated into pillars.

(Limitation) This upgrade is a content decision, not a validated intervention. The experiment above is precisely how we intend to check whether structured pages actually help agents — rather than asserting it.

Known limitations

Single site. Samples come from learnweb4.com only. Results would not generalize to forums, documentation, or e-commerce pages.
Small sample. One topic rendered three ways is a narrow probe. Ten questions per version is enough for signal, not for statistical confidence.
Model drift. Agents update frequently; a result is a snapshot of specific versions on a specific date, which is why the environment record exists.
Author bias. We wrote the content, the questions, and the answer key. Publishing raw transcripts and the frozen rubric is the mitigation, not a cure.
Retrieval method sensitivity. Results may differ between direct URL fetching and pasted content. The method is pinned for the run but limits scope either way.

How to reproduce this test on your own site

The Agent-Ready Website Checklist covers the static checks (crawlability, structure, metadata) that a page should pass before you bother running agent sessions against it — a cheap pre-screen for this kind of test.

FAQ

Has this experiment been run yet?

No. This article is a published test design. It documents the hypothesis, method, and scoring rubric in advance so the results can be checked against a fixed plan. Results will be appended after the run.

Why test your own site instead of a neutral one?

We control every variable on our own pages, which makes before/after comparisons possible. The trade-off is bias: we wrote both the content and the test questions, so the rubric and raw transcripts are published for outside scrutiny.

Can I reuse this test design on my own site?

Yes. The reproduction checklist at the end is written to be site-agnostic. Swap in your own page versions and test questions, keep the scoring rubric, and publish your raw transcripts.

Changelog

Jul 26, 2026: Initial publication: full test design published before any run; results will be appended after the experiment is executed.