Real ChatGPT answer · index seedRxLlMoQxeOccompleteMarketing and advertising · AI search and generative engine optimizationcommercial
My senior capstone team is trying to understand technical SEO for LLMs, specifically how robots txt and structured data affect crawling by ChatGPT and Claude, so what are the best recent guides?
Download run
Fan-out diagram
The searches the AI actually ran for this prompt.
SEED QUERY
My senior capstone team is trying to understand technical SEO for LLMs, specifically how robots txt and structured data affect crawling by ChatGPT and Claude, so what are the best recent guides?
3 real searches · 1 real-search pass
OpenAI official documentation OAI-SearchBot GPTBot ChatGPT-User robots.txt crawling websites 2026Google ↗
OBSERVEDOther / navigational
Anthropic official documentation ClaudeBot Claude-SearchBot robots.txt web crawler 2026Google ↗
OBSERVEDOther / navigational
technical SEO guide structured data schema.org robots.txt LLM crawlers ChatGPT Claude 2025 2026Google ↗
OBSERVEDInstruction
Badge and edge weight = share of passes that produced the query. Hover a query for its intent and routing format.
Query classes
Google's semantic query class for every sub-query: what kind of answer each search is after, simulated vs real.
ChatGPT real searches (3)
Classes come from the query-classifier (a local Laya model). 3 of 3 calls are low-confidence (under 50%) — hover a query in the table to see its confidence.
Searches and sources
ChatGPT3 real searches · gpt-6
OpenAI official documentation OAI-SearchBot GPTBot ChatGPT-User robots.txt crawling websites 2026 Google ↗
Anthropic official documentation ClaudeBot Claude-SearchBot robots.txt web crawler 2026 Google ↗
technical SEO guide structured data schema.org robots.txt LLM crawlers ChatGPT Claude 2025 2026 Google ↗
- Schema.org · schema.org
- Google Introduction to Structured Data · developers.google.com
- Article Structured Data · developers.google.com
- General Structured Data Guidelines | Google Search Central | Documentation | Google for Developers · developers.google.com
- Google Search's Guidance on Generative AI Content on Your Website | Google Search Central | Documentation | Google for Developers · developers.google.com
- Overview of OpenAI Crawlers · developers.openai.com
- Does Anthropic crawl data from the web, and how can site owners block the crawler? · support.anthropic.com
Queries
3 queries · click a column to sort
| Query / type | Search | Intent | Query class | Routing format | Cluster | Freq ↓ | Weight |
|---|---|---|---|---|---|---|---|
OpenAI official documentation OAI-SearchBot GPTBot ChatGPT-User robots.txt crawling websites 2026 OBSERVED | Google ↗ AI Mode ↗ | — | Other / navigational | — | — | 1x of 1 | |
Anthropic official documentation ClaudeBot Claude-SearchBot robots.txt web crawler 2026 OBSERVED | Google ↗ AI Mode ↗ | — | Other / navigational | — | — | 1x of 1 | |
technical SEO guide structured data schema.org robots.txt LLM crawlers ChatGPT Claude 2025 2026 OBSERVED | Google ↗ AI Mode ↗ | — | Instruction | — | — | 1x of 1 |