Real ChatGPT answer · index seedOT9xFj8z5gIcompleteMarketing and advertising · AI search and generative engine optimizationbrand
What are the exact robots.txt configurations required to block aggressive LLM agents while still allowing our content to appear in Google SGE?
Download run
Fan-out diagram
The searches the AI actually ran for this prompt.
SEED QUERY
What are the exact robots.txt configurations required to block aggressive LLM agents while still allowing our content to appear in Google SGE?
4 real searches · 1 real-search pass
Google Search Central robots.txt AI Overviews AI Mode Googlebot max-snippet nosnippet documentation 2026Google ↗
OBSERVEDOther / navigational
OpenAI GPTBot robots.txt user agent official documentationGoogle ↗
OBSERVEDOther / navigational
Anthropic web crawler robots.txt ClaudeBot Claude-User official documentationGoogle ↗
OBSERVEDInstruction
Google-Extended robots.txt does not affect Google Search AI Overviews officialGoogle ↗
OBSERVEDYes / no
Badge and edge weight = share of passes that produced the query. Hover a query for its intent and routing format.
Query classes
Google's semantic query class for every sub-query: what kind of answer each search is after, simulated vs real.
ChatGPT real searches (4)
Classes come from the query-classifier (a local Laya model). 4 of 4 calls are low-confidence (under 50%) — hover a query in the table to see its confidence.
Searches and sources
ChatGPT4 real searches · gpt-6
Google Search Central robots.txt AI Overviews AI Mode Googlebot max-snippet nosnippet documentation 2026 Google ↗
OpenAI GPTBot robots.txt user agent official documentation Google ↗
Anthropic web crawler robots.txt ClaudeBot Claude-User official documentation Google ↗
Google-Extended robots.txt does not affect Google Search AI Overviews official Google ↗
- AI Features and Your Website | Google Search Central | Documentation | Google for Developers · developers.google.com
- Google's common crawlers | Google Crawling Infrastructure | Crawling infrastructure | Google for Developers · developers.google.cn
- Robots Meta Tags Specifications | Google Search Central | Documentation | Google for Developers · developers.google.com
- Does Anthropic crawl data from the web, and how can site owners block the crawler? · support.anthropic.com
- Overview of OpenAI Crawlers · developers.openai.com
- Anthropic clarifies how Claude bots crawl sites and how to block them · searchengineland.com
- Google: AI features and your website · developers.google.com
- Google: robots.txt documentation · developers.google.com
- OpenAI: crawler documentation · developers.openai.com
Queries
4 queries · click a column to sort
| Query / type | Search | Intent | Query class | Routing format | Cluster | Freq ↓ | Weight |
|---|---|---|---|---|---|---|---|
Google Search Central robots.txt AI Overviews AI Mode Googlebot max-snippet nosnippet documentation 2026 OBSERVED | Google ↗ AI Mode ↗ | — | Other / navigational | — | — | 1x of 1 | |
OpenAI GPTBot robots.txt user agent official documentation OBSERVED | Google ↗ AI Mode ↗ | — | Other / navigational | — | — | 1x of 1 | |
Anthropic web crawler robots.txt ClaudeBot Claude-User official documentation OBSERVED | Google ↗ AI Mode ↗ | — | Instruction | — | — | 1x of 1 | |
Google-Extended robots.txt does not affect Google Search AI Overviews official OBSERVED | Google ↗ AI Mode ↗ | — | Yes / no | — | — | 1x of 1 |