Real ChatGPT answer · index seedpyJ1dEOhuY8completeMarketing and advertising · AI search and generative engine optimizationcomparison
What are the best technical SEO practices for making sure LLM web crawlers actually index my product pages correctly?
Download run
Fan-out diagram
The searches the AI actually ran for this prompt.
SEED QUERY
What are the best technical SEO practices for making sure LLM web crawlers actually index my product pages correctly?
8 real searches · 1 real-search pass
official documentation OpenAI OAI-SearchBot robots.txt crawler website ownersGoogle ↗
OBSERVEDOther / navigational
official Anthropic ClaudeBot web crawler robots.txt documentationGoogle ↗
OBSERVEDInstruction
PerplexityBot crawler robots.txt official documentationGoogle ↗
OBSERVEDOther / navigational
Google Search Central product structured data JavaScript SEO sitemap canonical documentationGoogle ↗
OBSERVEDOther / navigational
official documentation OpenAI OAI-SearchBot GPTBot robots.txt crawler ChatGPT searchGoogle ↗
OBSERVEDOther / navigational
Google Search Central product structured data ecommerce product snippets merchant listings crawling JavaScript SEO sitemap canonicalGoogle ↗
OBSERVEDOther / navigational
Anthropic ClaudeBot Claude-SearchBot web crawler robots.txt officialGoogle ↗
OBSERVEDOther / navigational
Perplexity crawler PerplexityBot robots.txt official documentationGoogle ↗
OBSERVEDOther / navigational
Badge and edge weight = share of passes that produced the query. Hover a query for its intent and routing format.
Query classes
Google's semantic query class for every sub-query: what kind of answer each search is after, simulated vs real.
ChatGPT real searches (8)
Classes come from the query-classifier (a local Laya model). 7 of 8 calls are low-confidence (under 50%) — hover a query in the table to see its confidence.
Searches and sources
ChatGPT8 real searches · gpt-6
official documentation OpenAI OAI-SearchBot robots.txt crawler website owners Google ↗
official Anthropic ClaudeBot web crawler robots.txt documentation Google ↗
PerplexityBot crawler robots.txt official documentation Google ↗
Google Search Central product structured data JavaScript SEO sitemap canonical documentation Google ↗
official documentation OpenAI OAI-SearchBot GPTBot robots.txt crawler ChatGPT search Google ↗
Google Search Central product structured data ecommerce product snippets merchant listings crawling JavaScript SEO sitemap canonical Google ↗
Anthropic ClaudeBot Claude-SearchBot web crawler robots.txt official Google ↗
Perplexity crawler PerplexityBot robots.txt official documentation Google ↗
- Understand JavaScript SEO Basics | Google Search Central | Documentation | Google for Developers · developers.google.com
- Generate Structured Data with JavaScript | Google Search Central | Documentation | Google for Developers · developers.google.com
- Technical SEO Techniques and Strategies | Google Search Central | Documentation | Google for Developers · developers.google.com
- How to Specify a Canonical with rel="canonical" and Other Methods | Google Search Central | Documentation | Google for Developers · developers.google.com
- Advertiser Guidance for Allowing OpenAI Web Crawlers · help.openai.com
- Overview of OpenAI Crawlers · developers.openai.com
- Publishers and Developers - FAQ · help.openai.com
Semantic map
Every query placed by meaning (EmbeddingGemma 2, 256-d). Shaded regions are topics, nearby points are near-paraphrases, and a circle's size is how often the query came back.
Colour
SimulatedReal search
Hover a query to see its counterpart · zoom with Ctrl/⌘ + scroll, a pinch, double-click or +
Queries
8 queries · click a column to sort
| Query / type | Search | Intent | Query class | Routing format | Cluster | Freq ↓ | Weight |
|---|---|---|---|---|---|---|---|
official documentation OpenAI OAI-SearchBot robots.txt crawler website owners OBSERVED | Google ↗ AI Mode ↗ | — | Other / navigational | — | — | 1x of 1 | |
official Anthropic ClaudeBot web crawler robots.txt documentation OBSERVED | Google ↗ AI Mode ↗ | — | Instruction | — | — | 1x of 1 | |
PerplexityBot crawler robots.txt official documentation OBSERVED | Google ↗ AI Mode ↗ | — | Other / navigational | — | — | 1x of 1 | |
Google Search Central product structured data JavaScript SEO sitemap canonical documentation OBSERVED | Google ↗ AI Mode ↗ | — | Other / navigational | — | — | 1x of 1 | |
official documentation OpenAI OAI-SearchBot GPTBot robots.txt crawler ChatGPT search OBSERVED | Google ↗ AI Mode ↗ | — | Other / navigational | — | — | 1x of 1 | |
Google Search Central product structured data ecommerce product snippets merchant listings crawling JavaScript SEO sitemap canonical OBSERVED | Google ↗ AI Mode ↗ | — | Other / navigational | — | — | 1x of 1 | |
Anthropic ClaudeBot Claude-SearchBot web crawler robots.txt official OBSERVED | Google ↗ AI Mode ↗ | — | Other / navigational | — | — | 1x of 1 | |
Perplexity crawler PerplexityBot robots.txt official documentation OBSERVED | Google ↗ AI Mode ↗ | — | Other / navigational | — | — | 1x of 1 |