How AI Agents See Your Website (We Checked Ours. It Was Invisible.)

The most instructive way to learn how AI agents see your website is to catch one seeing yours. We did. In July 2026 we requested our own article pages the way OpenAI's crawler does, and what came back was 9,481 bytes of nothing: an empty application shell, the homepage title on the wrong page, and 0 words of article content. Humans saw a full page. The machines that decide recommendations saw a blank. This guide is what we found, what we fixed, the receipts for both, and the 20-minute test that shows you what agents see on your site.

The visitor you can't see is now the majority

Automated traffic passed human traffic for the first time: 57.4% of web requests versus 42.6%¹, per Cloudflare's measurement in June 2026, arriving well ahead of Cloudflare's own forecast. Imperva's report had already measured the 2024 crossover at 51%. And the machine audience does not behave like the human one: TollBit data shows Anthropic's crawlers fetch thousands of pages for every single human referral they send back. The machine reads everything and clicks nothing.

An agent is not a rare visitor anymore; it is the majority of your traffic, because the crossover already happened. Your analytics cannot see this audience, because agents do not execute the tracking scripts that dashboards depend on. And the research on agentic shopping adds the commercial edge: agent demand concentrates on few options, with strong position biases. A machine that reads 5,000 pages does not widen the shortlist; it narrows it. In an agent-mediated market, the brand the machine can tell apart takes the share, and the generic brand rounds to zero.

The web is meanwhile splitting into two camps about this audience. Per Cloudflare Radar, the share of crawler requests met with a block jumped from 10.2% to 35.8% in a single year, while robots files like Hostino's in our own market explicitly welcome AI search bots in. Blocking is a legitimate licensing stance for publishers who sell content. For a business that wants to be found, chosen and transacted with, walling out the majority visitor is self-harm; the working posture is to let the legitimate agents in and make what they read worth carrying.

What an agent saw on our website: the receipts

We publish measurement for a living, so we pointed the instrument at ourselves. The request was simple: fetch our newest article the way GPTBot fetches it, no JavaScript execution, and compare it with what a human browser renders.

The cause was ordinary: like most modern sites, ours is a JavaScript application. Humans get the content because their browser runs the code. The big AI crawlers do not: OpenAI's own publisher guidance² confirms its crawlers rely on the served HTML, and Vercel's measurement of 569 million monthly GPTBot requests³ (winter 2024–25) found ChatGPT's and Claude's crawlers download JavaScript files without ever executing them. The precise picture: GPTBot, ClaudeBot and PerplexityBot read initial HTML only; Google's Gemini renders JavaScript through Googlebot's infrastructure, and Applebot and Bingbot render too. Betting your visibility on which crawler fetches you is a bad bet; content in the initial HTML works for all of them. The fix was build-time prerendering: every page now ships its full content in the initial HTML, and the human-facing application takes over on top of it. Nothing about the design changed. What changed is that the majority visitor can now read the site.

Agents read your site like a screen reader

The technical detail that explains almost everything: agents perceive pages through the accessibility tree. OpenAI states that its Atlas browser uses ARIA tags, the same labels and roles that support screen readers⁴, to interpret page structure. Microsoft's agent tooling serves accessibility snapshots instead of screenshots. Vision is the expensive fallback, not the primary channel. Google's own guidance⁵ says it plainly: sites designed to be beautiful for humans can be functionally broken for agents.

The performance data makes the stakes concrete. A CHI 2026 study measured a leading model completing 78% of web tasks under normal conditions, collapsing to 28% as interface legibility degraded. The interface IS the capability. What breaks agents is specific and boring: clickable divs instead of buttons, unlabeled form fields, content that appears only on hover or inside collapsed tabs, layouts that shift while the agent reasons, headings that skip levels. Every one of these is an accessibility defect that quietly became a revenue defect.

The failure nobody warns about: machine-facing drift

Beyond the empty shell, we checked llms.txt, the markdown file that tells language models what your site is⁶. Ours existed. It was well-written. And it described a product positioning we retired months ago: the single file machines are told to read first was confidently feeding them our old story.

A few sharp practitioners have flagged pieces of this problem: documentation teams have shown llms.txt files silently drifting out of sync with the docs they describe⁷, and schema specialists warn that stale structured data turns markup into a liability. What we have not seen named is the whole failure class, so we are naming it: machine-facing drift — every surface only machines read (llms.txt, structured data, meta descriptions, prerendered content) falling out of sync with what the brand has become. Not coverage drift; story drift. No human notices, because no human reads these files. The machine notices, and the machine is building your shortlist entry from them. If your company has repositioned, renamed an offer, or changed pricing in the last year, assume drift until you have checked.

Two honest caveats so you weight this correctly. Measurement firms report that only a tiny share of AI-crawler requests fetch llms.txt today, and studies so far find no direct citation lift from having one; the file is a hygiene layer, not a rankings lever. And that cuts both ways: a machine-facing file that IS read while telling the wrong story is worse than no file at all. Google clearly considers the layer real; llms.txt compliance is 1 of the 4 audits in Lighthouse's new Agentic Browsing category (shipped May 2026, still marked experimental), and our own site failed exactly that audit until the rewrite took the category to 100.

The 20-minute self-test

No specialist tools required. Run these 4 checks on your most important page.

1. The view-source test (5 minutes). Open your money page, view the page source (not inspect), and search for a sentence from the middle of your content. Not there? Most AI crawlers cannot see your page. This is the test our site failed.

2. The accessibility-tree test (5 minutes). Run your page through a screen reader (VoiceOver on Mac, NVDA on Windows) or Lighthouse's accessibility audit. If a screen reader can complete your key flow, an agent likely can, because both read the same tree.

3. The machine-files test (5 minutes). Read your own llms.txt, robots.txt, and structured data as if you were new to the company. Is the story current? Are AI crawlers allowed in or accidentally blocked? Does the schema still match your offer?

4. The completion test (5 minutes). Trace your conversion flow asking one question at each step: could software complete this? Unlabeled fields, CAPTCHAs before value, phone-only steps, and multi-page forms all end an agent's journey. Our audit form is 2 labeled fields; an agent can complete it, and that is deliberate.

The practices that matter, ranked by what agents read

1. Content in the initial HTML. Server-side render or prerender everything that matters. The single highest-impact fix, and the most commonly failed.

2. Native elements everywhere. Real buttons, real links, real selects, labels tied to inputs with autocomplete attributes. The accessibility tree is built from these.

3. Critical facts never hidden. Prices, specs, and FAQs behind tabs and accordions can be skipped by AI systems unless they are present in the served HTML.

4. One clean heading hierarchy. A single h1, no skipped levels, descriptive section headings. This is the document outline agents navigate by.

5. A current llms.txt with links. One markdown file: who you are, what you sell, where the key pages live. Then keep it in sync with your positioning, which is the part that gets missed.

6. Stable layout and visible state. Agents that cross-check visually get confused by shifting layouts; actions must visibly change the interface.

Why this decides revenue, not scores

Being readable is the entry fee, and the money on the table is measured: Adobe's Q1 2026 traffic report found AI-referred retail visits up 393% year over year and converting 42% better than non-AI traffic. The buyer who arrives from an AI answer arrives pre-convinced. The chain that matters runs further though: the machine has to retrieve you, cite you, name you, and recommend you before any of this pays, and a recommendation an agent cannot complete is not a recommendation; it is a lost transaction. That chain has a generic-brand problem long before it has a technical problem, because AI answers default to the same safe names unless a brand gives the machine something distinct to hold onto. Legibility without distinctiveness makes you perfectly readable and never chosen. Distinctiveness without legibility makes you the best option the machine cannot parse. The brands that win the agent-mediated market will hold both, and the ladder now has a name at each rung: SEO made you rank, AEO made you cited, GEO put you in the answer, and AAIO lets agents complete the task.