Real ChatGPT answer · index seedrN1d3mqhkKgcompleteMarketing and advertising · AI search and generative engine optimizationhowto
How do I configure our technical website settings to make sure Google SGE and other AI bots are allowed to crawl and use our content?
Download run
Fan-out diagram
The searches the AI actually ran for this prompt.
SEED QUERY
How do I configure our technical website settings to make sure Google SGE and other AI bots are allowed to crawl and use our content?
4 real searches · 1 real-search pass
Google Search Central AI features robots.txt Googlebot AI Overviews crawling indexing documentationGoogle ↗
OBSERVEDOther / navigational
OpenAI GPTBot OAI-SearchBot robots.txt official documentationGoogle ↗
OBSERVEDOther / navigational
Perplexity crawler user agent robots.txt official documentationGoogle ↗
OBSERVEDInstruction
Anthropic ClaudeBot robots.txt crawler official documentationGoogle ↗
OBSERVEDOther / navigational
Badge and edge weight = share of passes that produced the query. Hover a query for its intent and routing format.
Query classes
Google's semantic query class for every sub-query: what kind of answer each search is after, simulated vs real.
ChatGPT real searches (4)
Classes come from the query-classifier (a local Laya model). 4 of 4 calls are low-confidence (under 50%) — hover a query in the table to see its confidence.
Searches and sources
ChatGPT4 real searches · gpt-6
Google Search Central AI features robots.txt Googlebot AI Overviews crawling indexing documentation Google ↗
OpenAI GPTBot OAI-SearchBot robots.txt official documentation Google ↗
Perplexity crawler user agent robots.txt official documentation Google ↗
Anthropic ClaudeBot robots.txt crawler official documentation Google ↗
- Overview of OpenAI Crawlers · developers.openai.com
- Does Anthropic crawl data from the web, and how can site owners block the crawler? · support.anthropic.com
- AI Features and Your Website | Google Search Central | Documentation | Google for Developers · developers.google.com
- What Is Googlebot | Google Search Central | Documentation | Google for Developers · developers.google.com
- SearchBot IP documentation · openai.com
- Search generative AI control · support.google.com
- Google's common crawlers | Google Crawling Infrastructure | Crawling infrastructure | Google for Developers · developers.google.com
- Google's official Search generative AI control instructions · support.google.com
- Open Search Console · search.google.com
- Open Bing Webmaster Tools · bing.com
- Test website performance · pagespeed.web.dev
- Validate schema · validator.schema.org
Queries
4 queries · click a column to sort
| Query / type | Search | Intent | Query class | Routing format | Cluster | Freq ↓ | Weight |
|---|---|---|---|---|---|---|---|
Google Search Central AI features robots.txt Googlebot AI Overviews crawling indexing documentation OBSERVED | Google ↗ AI Mode ↗ | — | Other / navigational | — | — | 1x of 1 | |
OpenAI GPTBot OAI-SearchBot robots.txt official documentation OBSERVED | Google ↗ AI Mode ↗ | — | Other / navigational | — | — | 1x of 1 | |
Perplexity crawler user agent robots.txt official documentation OBSERVED | Google ↗ AI Mode ↗ | — | Instruction | — | — | 1x of 1 | |
Anthropic ClaudeBot robots.txt crawler official documentation OBSERVED | Google ↗ AI Mode ↗ | — | Other / navigational | — | — | 1x of 1 |