Real ChatGPT answer · index seedOXydAhdZVGUcompleteMarketing and advertising · AI search and generative engine optimizationhowto
What are the most common technical mistakes that prevent smaller websites from getting crawled and cited by Perplexity and other AI tools?
Download run
Fan-out diagram
The searches the AI actually ran for this prompt.
SEED QUERY
What are the most common technical mistakes that prevent smaller websites from getting crawled and cited by Perplexity and other AI tools?
4 real searches · 1 real-search pass
Perplexity crawler documentation PerplexityBot robots.txt user agent officialGoogle ↗
OBSERVEDOther / navigational
OpenAI search crawler OAI-SearchBot robots.txt official documentationGoogle ↗
OBSERVEDOther / navigational
Google Search Central JavaScript SEO crawling rendering robots sitemap canonical structured data documentationGoogle ↗
OBSERVEDOther / navigational
Microsoft Bing webmaster guidelines IndexNow crawling robots.txt officialGoogle ↗
OBSERVEDOther / navigational
Badge and edge weight = share of passes that produced the query. Hover a query for its intent and routing format.
Query classes
Google's semantic query class for every sub-query: what kind of answer each search is after, simulated vs real.
ChatGPT real searches (4)
Classes come from the query-classifier (a local Laya model). 4 of 4 calls are low-confidence (under 50%) — hover a query in the table to see its confidence.
Searches and sources
ChatGPT4 real searches · gpt-6
Perplexity crawler documentation PerplexityBot robots.txt user agent official Google ↗
OpenAI search crawler OAI-SearchBot robots.txt official documentation Google ↗
Google Search Central JavaScript SEO crawling rendering robots sitemap canonical structured data documentation Google ↗
Microsoft Bing webmaster guidelines IndexNow crawling robots.txt official Google ↗
- Overview of OpenAI Crawlers · developers.openai.com
- Publishers and Developers - FAQ · help.openai.com
- Understand JavaScript SEO Basics | Google Search Central | Documentation | Google for Developers · developers.google.com
- Dynamic Rendering as a workaround | Google Search Central | Documentation | Google for Developers · ccn.web.tr
- Technical SEO Techniques and Strategies | Google Search Central | Documentation | Google for Developers · developers.google.com
- Google Crawling and Indexing | Google Search Central | Documentation | Google for Developers · developers.google.cn
- Why is My Site Not in the Index? · bing.com
- Webmaster Support · bing.com
- Webmaster Guidelines · bing.com
- Understand JavaScript SEO Basics | Google Search Central | Documentation | Google for Developers · developers.google.com
- crawler documentation · docs.perplexity.ai
- Updated Guide – Perplexity Crawlers & AWS WAF Setup - Announcements · community.perplexity.ai
- Open Search Console · search.google.com
- Open Bing Webmaster Tools · bing.com
- Explore Schema.org · schema.org
- Open Rich Results Test · search.google.com
Queries
4 queries · click a column to sort
| Query / type | Search | Intent | Query class | Routing format | Cluster | Freq ↓ | Weight |
|---|---|---|---|---|---|---|---|
Perplexity crawler documentation PerplexityBot robots.txt user agent official OBSERVED | Google ↗ AI Mode ↗ | — | Other / navigational | — | — | 1x of 1 | |
OpenAI search crawler OAI-SearchBot robots.txt official documentation OBSERVED | Google ↗ AI Mode ↗ | — | Other / navigational | — | — | 1x of 1 | |
Google Search Central JavaScript SEO crawling rendering robots sitemap canonical structured data documentation OBSERVED | Google ↗ AI Mode ↗ | — | Other / navigational | — | — | 1x of 1 | |
Microsoft Bing webmaster guidelines IndexNow crawling robots.txt official OBSERVED | Google ↗ AI Mode ↗ | — | Other / navigational | — | — | 1x of 1 |