Kaliber Group — First-Party Research

The Retrieval Economy Arrives: Zero of 11 Industries Are AI-Ready

Cloudflare data shows bots now generate 57.5% of HTML web traffic — the first machine majority in internet history.¹ Kaliber's audit of 399 websites across 11 industries shows zero industries average above the 'Needs work' band on AI Readability. The retrieval economy is here. Almost no brand is ready for it.

The retrieval economy: why AI readability suddenly matters

On 3 June 2026, Cloudflare co-founder and CEO Matthew Prince posted a milestone that changes how brands need to think about their website: bots had, for the first time in the history of the internet, generated more HTML traffic than humans. Cloudflare Radar — which tracks approximately a fifth of all websites globally — put the split at 57.5% automated requests versus 42.5% from humans.¹

Prince himself had projected this tipping point for late 2027. It arrived a year and a half early. The reason, he noted, was "agentic traffic growing so fast" — a category that barely existed 24 months prior.¹

The trajectory is what should focus every brand's attention. Cloudflare's 2025 Year in Review reported bot traffic at approximately 30% of global web requests.⁴ Twelve months later, that figure has roughly doubled. AI crawlers — GPTBot, ClaudeBot, PerplexityBot, Google-Extended, Applebot-Extended, Bytespider, Meta-ExternalAgent — now account for approximately 26.7% of verified bot activity, per Cloudflare's May 2026 data.³

For a brand website, the composition of that visiting traffic has practical consequences. A meaningful and rising share of the machines visiting the site are not indexing it for a search results page. They are retrieving content to synthesise an answer inside an AI chat interface — an answer the human user reads without ever visiting the source. The site is being read; it is not being visited. Whether that reading produces citation, brand mention, or a link back depends entirely on how legible the content is to the machine reading it.

"The site is being read; it is not being visited."

The crawl-to-referral asymmetry: extract vs return

Cloudflare's most instructive metric for understanding the retrieval economy is the crawl-to-referral ratio — how many pages an AI platform fetches for every one visitor it sends back to the source. Introduced on Cloudflare Radar in July 2025, this ratio makes visible the value exchange that used to be implicit in the SEO era: content is indexed, users find it via search, users visit the source.²

The 2026 data shows the exchange is now catastrophically one-sided for AI platforms:

Read that table twice. Traditional Google search sends one human visitor for every five pages it crawls. ClaudeBot, at Q1 2026 rates, crawls nearly 24,000 pages for every one referral. Perplexity — often held up as the AI platform with the most publisher-friendly attribution model — still crawls 111 times more than it refers.²

The pattern is not accidental. Cloudflare's crawler analysis attributes the majority of AI crawler activity to model training, not search. Per Cloudflare's May 2026 data, 51.8% of AI crawler requests are for training purposes and only 9.3% are for search. The rest — 35.7% — is mixed-purpose retrieval, and the search-only share is rising month-on-month (up from 7.5% in April 2026).³ Full-year 2025 data shows an even more training-heavy picture: nearly 80% of AI bot activity was training-related.³

The implication for brand websites: most of the AI activity on a site is content extraction with zero referral compensation. The brands that show up in AI-synthesised answers — cited, mentioned by name, linked back — are those the models were trained on well and the retrieval systems can parse cleanly. Both depend on the same thing: the site being legible to machine readers.

Our benchmark: 399 sites, 11 industries, one uncomfortable pattern

Between 22 May 2026 (tool launch) and 8 July 2026, Kaliber's free AI Readability Checker ran 425 real audits submitted by real users. Of those, 399 fall into 11 industries with sample sizes large enough for a research-grade benchmark (n ≥ 15 per industry). The remaining 26 audits sit in three thin-sample industries (Manufacturing n=11, Media n=12, Nonprofit n=3) that are noted in the methodology but excluded from the primary table.

The finding is consistent across every industry with enough data to matter:

"No industry averaged above the 'Needs work' band. Not fintech. Not SaaS. Not e-commerce. The industry closest to passing is still below the pass line."

What we measured

The Kaliber AI Readability Checker scores a website from 0 to 100 across 19 structural checks in four categories:

Discovery surface — whether AI engines can find and interpret the site at the meta level. Checks include llms.txt, AGENTS.md, sitemap.md, robots directives for AI crawlers, and canonical entity markup.

Page-level structure — whether the HTML delivered to a crawler is semantic and self-contained. Checks include heading hierarchy, canonical URLs, JavaScript-rendered content availability, Markdown mirror files, and meta-description-to-first-paragraph consistency.

Schema depth — whether structured data tells AI engines what the site is. Checks include Organization, Product, Service, FAQ, Article, and BreadcrumbList JSON-LD blocks.

Citation quality — whether the content itself is quotable. This is the only category not scored deterministically. It is scored by Anthropic Claude on the public content of the audited domain, evaluating whether the site's answer to buyer questions is unambiguous, self-contained, and would be retrieved as a citation.

The bands are: 85+ Excellent, 70–84 Good, 60–69 Fair, below 60 Needs work. The 60-point threshold matters more than any other line. Above 60, AI engines can reliably surface the brand when a buyer asks a category question. Below 60, the engine tends to skip the brand and cite competitors — or worse, cite generic aggregator sites that outrank the brand's own domain in the retrieval index.

The industry breakdown

Every industry in the primary benchmark sits below the 60-point Fair threshold. Sorted by mean score, highest to lowest:

Primary benchmark: 11 industries, 399 audits, sample sizes 16–75 per industry. Industries with fewer than 15 samples (Manufacturing n=11, Media n=12, Nonprofit n=3 — 26 audits total) are excluded from the table for research rigour and noted in the methodology footnote. Excluding them shifts the overall weighted mean by only 0.2 points (41.1 → 41.3).

Why every industry is failing the same test

The four categories that make up the AI Readability score correspond to four things AI engines need in order to cite a brand cleanly. In our audit data, three of the four are failing across the board, and the fourth — citation quality — is the primary drag on almost every score.

Discovery surface is failing because it is new. llms.txt and AGENTS.md as conventions were proposed in 2024 and gained meaningful adoption only in late 2025. Most sites in the sample were built before these conventions existed and have not yet added them. This is the easiest category to fix — the files are small, the schema is documented, and a single afternoon's work can lift a site by 8 to 15 points on discovery alone. Notably, Cloudflare's robots.txt analysis for Q2 2026 (snapshot 29 June 2026) found the most-named AI user-agents were GPTBot (690 mentions), ClaudeBot (594), Google-Extended (558), CCBot (554) and Bytespider (467) — meaning site operators are starting to declare AI-crawler policy in the discovery layer, but adoption is still small in absolute terms.⁵

Page-level structure fails when JavaScript hides content. Sites built on Webflow, Squarespace, Wix, and modern SPA frameworks routinely serve empty HTML shells to crawlers that do not execute JavaScript. Most AI engines' retrieval crawlers do not execute JavaScript by default. If the primary content of the page appears only after client-side rendering, the AI engine sees nothing to cite. This is the category that most cleanly separates Education-tier sites (which tend to use plain HTML) from consumer brand sites (which lean on JavaScript for animation and interactivity).

Schema depth fails through absence, not incorrectness. Fewer than one in four sites in the sample publish JSON-LD Organization schema. Fewer than one in seven publish Product or Service schema. FAQ schema is rare enough that its presence alone is a competitive edge in AI retrieval. The gap here is not sites getting schema wrong — it is sites having no schema at all. This is the second-easiest category to fix: pick a schema type, drop a JSON-LD block in the page head, done.

Citation quality is the hardest and the largest drag. A page can pass every structural check and still score below 60 on Citation Quality if the content itself is not quotable. Marketing pages that lead with visual hooks and reveal the answer three paragraphs down produce weak citation candidates. Pages that answer buyer questions in the first paragraph with self-contained 40 to 120-word blocks produce strong ones. Most brand websites are optimised for human click-through and conversion flows, not for machine retrieval. The optimisation that produces the highest conversion rate for humans often produces the lowest citation rate for AI engines. This is the tension every brand now has to resolve.

Why Education is a 10-point outlier

Education sites lead every other industry in the sample by a wide margin (57.1 vs 46.1 for the next-highest, Agency). The pattern in the underlying audit data is consistent enough to point at a structural reason rather than a random cluster.

Educational websites tend to carry a native content structure that aligns with what AI engines are trained to retrieve. Course catalogue pages produce clean canonical URLs. Programme descriptions produce self-contained answer paragraphs. Glossary and FAQ pages produce the exact 40 to 120-word answer blocks that AI engines cite. Faculty pages produce Person schema. Departments and courses produce Organization and Course schema naturally.

None of this is intentional AI optimisation. Educational sites were built to help students find programme information — which happens to look almost exactly like what AI engines look for when answering buyer-decision questions. The alignment is accidental and instructive. The brands that will do best in AI search over the next 24 months are the ones that structure their content the way an educational institution structures its programme catalogue: canonical, taxonomic, self-contained, answer-first.

What AI readability actually is

AI readability is not SEO. It is not accessibility. It is not content quality in the abstract. It is a specific and narrow measure: how legible a website is to generative AI engines when those engines retrieve, quote, and cite content in response to buyer questions.

A site can rank first on Google organic search and score below 40 on AI readability. Traditional SEO optimises for the ranking systems Google's Search team built for the ten blue links era. AI readability optimises for the retrieval systems Anthropic, OpenAI, Google's AI team, and Microsoft built for the citation-and-answer era. The two systems overlap on some signals — semantic HTML, canonical URLs, clean headings — but diverge sharply on others.

AI retrieval systems reward: self-contained answer blocks (40–120 words), unambiguous entity naming, structured data schemas, plain-text and Markdown alternatives to JavaScript-rendered content, and explicit AI-directed files like llms.txt and AGENTS.md.

Traditional search ranking systems reward: backlink authority, keyword-density-adjusted content depth, on-page engagement signals, mobile page speed, and structured data (the one signal both systems reward). A site can be excellent at one and weak at the other. The 399 audits in this benchmark suggest the average brand website is currently optimised for traditional search and structurally invisible to AI retrieval.

What to do about it

The four category failures above each have concrete fixes. The order of impact-per-effort, based on our audit data, is:

1. Publish llms.txt and AGENTS.md at the site root. These are small text files (under 1 KB each) that tell AI engines how to represent the brand: what to recommend, what not to misrepresent, how to characterise the engagement model. Time: one afternoon. Typical score impact: +8 to +15 points on discovery category.

2. Add JSON-LD schema for Organization, Service or Product, and FAQ. The schema types are documented at schema.org. The JSON-LD blocks go in the page head. Time: a few hours per template. Typical score impact: +5 to +12 points on schema category.

3. Rewrite the primary page so the answer to the buyer's question sits in the first paragraph. Not the hero image. Not the marketing headline. The literal answer, in 40 to 120 words, self-contained enough that it could be quoted as a citation without the surrounding context. Time: one to two writing sessions per priority page. Typical score impact: +10 to +20 points on citation quality.

4. Add a Markdown mirror of primary pages. Publish /about.md, /services.md, and equivalents for the top-priority pages. Add <link rel="alternate" type="text/markdown"> in the HTML. Optionally support Accept: text/markdown at the edge. Time: half a day. Typical score impact: +3 to +8 points on page-level structure.

Combined, these four changes take a typical week of engineering and content work. In our refresh data, sites that ship all four move from a "Needs work" score in the 30s to a "Fair" or "Good" score in the 60s to 70s inside seven days. The single largest verified uplift in our refresh log is a jump from 10 to 69 in 73 minutes — an AI tooling product that ran the audit, took the Markdown report to Claude, and shipped the recommended fixes in one afternoon.

What this signals for the next 12 months

Three shifts are already visible in the underlying data. Each has practical implications for how brands should plan the next four quarters.

Search-purpose AI crawling is growing faster than training-purpose crawling. Cloudflare's data shows the search-only share of AI crawler purposes rose from 7.5% in April 2026 to 9.3% in May 2026 — a 24% relative increase in a single month.³ The mixed-purpose category (training plus retrieval) accounts for another 35.7%. The centre of gravity is shifting from pure model training toward real-time retrieval that produces cited answers. Brands that improve their readability now capture the compounding upside as retrieval share continues to grow.

Cloudflare and other infrastructure providers are moving toward crawl compensation. Cloudflare launched Pay Per Crawl in July 2025 and expanded it into "Pay Per Use" in 2026, and now blocks mixed-use AI crawlers by default for new domains.⁶ The economic terms of AI crawling are being renegotiated at the infrastructure layer. Brands that treat their content as a licensable asset — with clear machine-readable declarations of what AI engines may and may not do with it — will be positioned to participate in whatever compensation model wins. Brands that leave their sites open and unreadable get the worst of both outcomes: extracted freely, cited rarely.

PerplexityBot is the outlier that proves the direction. Cloudflare's robots.txt analysis notes that PerplexityBot and ChatGPT-User are among the very few AI user-agents that appear more often in ALLOW rules than DISALLOW rules — because they actually return traffic.⁵ The lesson: AI platforms that build reciprocity with publishers get access; extractive-only platforms get blocked. The same logic applies at the brand level. Sites that make themselves cleanly readable receive more citations. Citations produce brand mention. Brand mention produces the search-and-visit behaviour that even the retrieval economy still ultimately depends on.

What this benchmark does not answer

Which industries improve fastest. The refresh data (users who re-audit after making changes) is skewed toward the industries that ran the audit earliest. We have re-audit data for SaaS, Agency, and B2B Services in useful volume; the picture on Education, Healthcare, and Hospitality is thin. A follow-up study in Q4 2026 will address which industries convert audit findings into score improvements most reliably.

Whether AI Readability score correlates with AI-search visibility. An adjacent Kaliber tool, the AI Visibility Audit, measures whether a brand actually appears in AI-engine responses to buyer questions. In a sample of 60 brands audited on both tools, the correlation between Readability score and Visibility score is positive but noisy (r ≈ 0.58). The Readability score is a necessary but not sufficient condition. A site scoring 80 on Readability but 0 on Visibility is possible if the brand simply is not part of the retrieval index yet. Both scores need to move together, and Readability is the upstream of the two.

How much of this shifts with the next platform update. Google's May 2026 AI Search documentation formalised many of the checks in the audit; the June 2026 spam update tightened enforcement on cloaked JavaScript rendering. The specific weights of the 19 checks will shift with each major platform update. The category structure — discovery, page, schema, citation — is stable. The exact points-per-check will not be.

Frequently asked questions