AI systems now answer questions, cite sources, and complete tasks on the web. They can only do that on sites they can read. We wanted a real number for how much of the popular web is open to them. So we sampled 500 of the web’s most visited sites and scanned every one. Each one got an agent readiness (ARO) score from 0 to 100. It is the same score every PageClimber scan produces.
The average score was 63.9. More than a third of the sites (37 percent) scored below 60. More than one in five scored below 40. In large part, the web that humans use every day is closed to the machines that people now ask about it.
The headline numbers
Across all 500 sites:
- 22 percent block fetch-based AI agents at the network edge. Agents of this kind power most AI search and assistant products.
- 23 percent serve a near-empty page to agents, with fewer than 150 words of readable text.
- 18 percent block at least one named AI crawler in robots.txt. 14 percent block four or more.
- 14 percent put a CAPTCHA on their homepage.
- 14 percent adopted llms.txt, the emerging guide file for AI systems.
The most blocked crawlers were CCBot (76 sites), GPTBot (68), and ClaudeBot (68). Google-Extended followed at 61, then PerplexityBot at 53.
The scores people ask about
| Site | ARO | Notes |
|---|---|---|
| github.com | 100 | llms.txt |
| stripe.com | 100 | llms.txt |
| shopify.com | 100 | llms.txt |
| wordpress.org | 100 | llms.txt |
| apple.com | 95 | clean |
| microsoft.com | 95 | clean |
| wikipedia.org | 95 | clean |
| walmart.com | 90 | clean |
| anthropic.com | 86 | clean |
| linkedin.com | 84 | clean |
| netflix.com | 75 | CAPTCHA |
| bbc.com | 70 | robots.txt blocks 8 AI bots |
| openai.com | 70 | blocks fetch agents |
| youtube.com | 65 | clean |
| cnn.com | 65 | robots.txt blocks 8 AI bots |
| twitter.com | 50 | clean |
| google.com | 45 | clean |
| amazon.com | 45 | blocks fetch agents, robots.txt blocks 7 AI bots |
| facebook.com | 42 | login wall |
| nytimes.com | 33 | CAPTCHA, robots.txt blocks 8 AI bots |
| bloomberg.com | 33 | CAPTCHA, robots.txt blocks 8 AI bots |
| tiktok.com | 28 | robots.txt blocks 8 AI bots |
| reddit.com | 20 | blocks fetch agents |
| pinterest.com | 16 | CAPTCHA, login wall |
| instagram.com | 9 | login wall, robots.txt blocks 4 AI bots |
| stackoverflow.com | 0 | blocks fetch agents, blocks browsers |
| espn.com | 0 | blocks fetch agents, robots.txt blocks 5 AI bots |
| yelp.com | 0 | blocks fetch agents, robots.txt blocks 8 AI bots |
| medium.com | 0 | blocks fetch agents, CAPTCHA, robots.txt blocks 2 AI bots |
| quora.com | 0 | blocks fetch agents, login wall, robots.txt blocks 5 AI bots |
| Site | ARO | Notes |
|---|
Who aces it
The best scores went to companies that sell to developers. GitHub, Stripe, Shopify, WordPress.org, HubSpot, Zendesk, and Cisco all scored a perfect 100. GitHub, Stripe, and Shopify also publish llms.txt files. The pattern is hard to miss. The companies closest to the AI ecosystem made their sites fully readable to it.
Other strong scores: Apple (95), Microsoft (95), Wikipedia (95), Walmart (90), and Anthropic (86).
Who fails it
Ten well-known sites scored zero: StackOverflow, Yelp, ESPN, Roblox, Epic Games, Medium, Quora, Fandom, Pixabay, and McAfee. All ten turn agents away completely. Our scanner could not read them with a plain fetch. It could not read them with a full headless browser either.
The social platforms fare little better. Instagram scored 9 behind a login wall. Pinterest scored 16. Reddit scored 20. TikTok scored 28. These platforms hold much of the content people ask AI about. The AI cannot see it.
Two surprises sit near the bottom. Google scored 45 on its own homepage. Amazon also scored 45, blocks fetch agents, and disallows seven AI crawlers in robots.txt.
The news industry blocks in lockstep
The clearest category pattern came from news. The New York Times (33), Bloomberg (33), CNN (65), and BBC (70) each disallow eight AI crawlers in robots.txt. The Times and Bloomberg add a CAPTCHA on top. The industry made a coordinated choice: AI systems do not get the news. Does that choice protect the business, or does it remove the news from the answer layer of the web? That is the open question of the decade.
The irony award
openai.com blocks fetch-based agents. Its crawlers knock on every door on the web. Its own door stays shut to the same kind of visitor. It scored 70. Anthropic scored 86 with no blocks. Developer-tool companies that build on these models outscored both.
What this means for your site
The web now has two kinds of readers. Humans see your rendered page. AI systems see whatever your server, your CDN, and your JavaScript leave for them. On more than a third of these sites, that second reader gets a degraded page or none at all.
For a large brand, that may be a deliberate trade. For everyone else, it is usually an accident. A default firewall rule, a JavaScript-only template, or a copied robots.txt can hide a site from AI search. Nobody notices. The traffic does not disappear. It goes to a competitor the AI can read.
You can test your own site in seconds. Our free agent readiness checker runs the structural checks with no account. A full PageClimber scan scores your site on all four channels: SEO, AEO, GEO, and ARO. It lists the exact fixes.
Methodology
We sampled 500 of the most visited sites on the web from a public ranking of popular domains, keeping sites that serve a normal HTML homepage. This is a sample of popular sites, not a definitive top 500.
Each site got the same ARO audit our customers get, with PageClimber scanner v0.4.0 on August 10 and 11, 2026. The Index score and the in-product ARO score are one number, from one engine. The audit makes about four polite requests per site: one plain fetch, one headless-browser render, plus robots.txt and llms.txt. The score starts at 100. It loses points for blocked fetches, near-empty rendered pages, CAPTCHAs, login walls, unusable controls, missing llms.txt, and robots.txt rules against AI crawlers.
The scan ran from a datacenter IP address, because that is where AI agents live. A site that blocks datacenter traffic blocks the agents too. We measure that condition on purpose. The full check list is described on our agent readiness page.
We will run the Index again from time to time and track how these numbers move. If the past year is any guide, they will move fast.
Frequently asked
What is the AI Readability Index?
A recurring report from PageClimber that scores a sample of 500 of the most visited websites for agent readiness (ARO): how well AI systems and agents can read and use each site. The score runs from 0 to 100.
Where does the list of sites come from?
We sampled 500 of the most visited sites on the web from a public ranking of popular domains. It is a sample of popular sites, not a definitive top 500.
Is the Index score the same as the PageClimber ARO score?
Yes. The Index score and the in-product ARO score are one number, from one engine. Every site in the Index got the same audit a PageClimber scan runs.
How can I check my own site?
The free agent readiness checker runs the structural checks with no account. A full PageClimber scan scores your site on SEO, AEO, GEO, and ARO, and lists the exact fixes.