How Does AI Search Actually Find and Cite Website Content?

There's no separate "AI crawler" running a fundamentally different process than the one search engines have used for years. A small number of named, documented bots (OpenAI's, Anthropic's, Perplexity's, Google's) fetch your pages, most of them respect the same robots.txt rules crawlers always have, and whether you get cited comes down to two things: can they actually reach your content, and does that content give a clear, specific, directly usable answer once they do. Structured data and special AI markup, despite a lot of advice claiming otherwise, aren't the lever most of that advice implies.

This is the third piece in our AI search optimization series. The first covered the AEO/GEO hype generally; the second went deep on whether an llms.txt file is worth building. This one steps back to the actual mechanics: what's crawling your site, and what makes it choose your page over someone else's.

The crawlers behind AI search, by name

Each major AI system runs its own set of bots, documented in their own official pages, not some hidden process:

• OpenAI: GPTBot collects content to improve its models (training). OAI-SearchBot is the one that actually feeds ChatGPT's search features, closer to a traditional search crawler. ChatGPT-User fires only when someone's live ChatGPT session asks it to fetch a specific page. GPTBot and OAI-SearchBot respect robots.txt; ChatGPT-User may not, since it's a direct, user-triggered request rather than automated crawling.
• Anthropic: the same three-bot pattern. ClaudeBot for training, Claude-SearchBot to improve search result quality, Claude-User for live user-triggered fetches. All three respect robots.txt, and they're controlled independently, so blocking one doesn't block the others.
• Perplexity: PerplexityBot handles general crawling and indexing (not training), and Perplexity-User fires on a live user's question. One useful nuance straight from Perplexity's own documentation: if you block PerplexityBot, it may still index your domain, headline, and a brief factual summary. Blocking isn't all-or-nothing.
• Google: there's no separate crawler for AI Overviews specifically. Google-Extended controls whether your content can be used to train Gemini and other AI features; the citations that show up in AI Overviews come from the same index the standard Googlebot already built. Google's own guidance is direct about this: its generative search features are built on the same core systems as regular Search, using the same crawlable, indexed content, no separate crawl or markup required.

The one genuinely new wrinkle compared to traditional SEO is the training/search split. A site can allow the bot that feeds citations (OAI-SearchBot, Claude-SearchBot) while blocking the one that feeds model training (GPTBot, ClaudeBot), and plenty of sites now do exactly that.

One more thing worth knowing if your site runs on Cloudflare (as ours, and every site we build, does): starting September 15, 2026, new domains onboarded to Cloudflare will block AI training and agent bots by default on ad-monetized pages, while continuing to allow search bots by default. It's an opt-out change, not opt-in, so it's worth checking your security settings if a client's AI visibility suddenly seems off after that date.

Getting crawled isn't the same as getting cited

Even fully open to every bot above, showing up in Google's results and getting cited by AI search are more loosely connected than most advice assumes. Ahrefs tested 15,000 long-tail queries across ChatGPT, Gemini, Copilot, and Perplexity and found that only 12% of AI-cited URLs also ranked in Google's top 10 for that same query. Roughly 80% of AI citations didn't appear anywhere in Google's results for the query at all. Perplexity was the outlier, with about 29% of its citations overlapping Google's top 10, noticeably higher than the roughly 8% average across the other assistants, likely because it leans more heavily on live retrieval rather than a pre-built index. Ahrefs' own caveat is worth repeating here: the data was collected in mid-2025 and the authors note results may have shifted since.

The practical takeaway is that ranking #1 on Google doesn't guarantee an AI citation, and a page nowhere near page one can still get cited if it directly and specifically answers the question being asked. AI assistants tend to break one prompt into several smaller searches behind the scenes, rather than reading down a single ranked list the way a person scanning Google results would.

Does schema markup actually help?

This is where most AI search advice gets ahead of the evidence, and it's worth being honest about, including with ourselves. When we added structured data (JSON-LD) to our own site in July 2026, an earlier, correlational stat suggested pages with structured data were more likely to get cited by ChatGPT. That correlation was real. A newer, controlled study published by Ahrefs in May 2026 tested whether it was causal, and it wasn't.

Ahrefs tracked 1,885 pages that newly added schema markup against roughly 4,000 similar pages that didn't, all starting from a comparable citation level, over an eight-month window. The results: Google AI Overviews citations actually dropped 4.6% on the pages that added schema, a small but statistically real decline. Google AI Mode and ChatGPT both showed small increases (2.4% and 2.2%), but the authors describe both as statistically indistinguishable from noise, meaning not a real effect either way. A separate real-time test in the same research went further: when five major AI systems fetched pages live, none of them actually parsed the JSON-LD, hidden Microdata, or hidden RDFa. Every one of them read only the visible HTML content a person would see on the page.

The likely explanation is straightforward: sites that already invest in strong, citable content also tend to be the kind of sites that add schema markup, not the other way around. Structured data still earns real value on Google, mainly rich results and clearer entity information, and Google's own guidance still says no special AI markup is required for AI Overviews or AI Mode either way. It's worth having for what it's actually proven to do. It just isn't the AI-citation shortcut a lot of advice makes it out to be, the same honest conclusion our llms.txt piece landed on for that specific tactic.

What actually determines whether you get cited

Strip out the tactics that don't hold up, and what's left lines up with ordinary SEO fundamentals, not a separate discipline:

• Being reachable in the first place. If a bot is blocked, intentionally or by a hosting default you didn't check, it can't cite what it can't read.
• A clear, specific, directly quotable answer. AI systems are built to answer questions. A page that states a real answer plainly, in the first paragraph or two, gets used more than one that buries it under brand language.
• Genuine expertise and specificity. A focused page written from real, first-hand knowledge of the subject tends to beat a vague, general one covering the same ground.
• Freshness, especially for assistants that lean on live retrieval. Perplexity's higher overlap with real-time results suggests recently updated, currently accurate content has an edge with tools that actively search rather than only replaying an older index.

None of this is a separate "AI SEO" checklist. It's the same fundamentals that have mattered for search all along: a crawlable site, clear and genuinely useful content, and a solid technical foundation underneath it.

---

Not sure your site actually has those fundamentals covered? That's exactly what our free Website Grader checks, no AI markup or special file required first.