Real ChatGPT answer · index seedyzXZyggNQO0completeMarketing and advertising · AI search and generative engine optimizationinformational
Why do some generative search engines ignore robots meta tags and how is the industry responding to this data scraping issue?
Download run
Fan-out diagram
The searches the AI actually ran for this prompt.
SEED QUERY
Why do some generative search engines ignore robots meta tags and how is the industry responding to this data scraping issue?
4 real searches · 1 real-search pass
generative AI search engines robots meta tags ignore robots.txt AI crawlers publisher controls 2025 2026Google ↗
OBSERVEDOther / navigational
Cloudflare AI crawler controls pay per crawl content signals robots.txt AI bots 2025Google ↗
OBSERVEDOther / navigational
Google-Extended robots.txt control Gemini training AI Overviews documentationGoogle ↗
OBSERVEDComparison
IETF robots exclusion protocol RFC 9309 voluntary crawler complianceGoogle ↗
OBSERVEDReason
Badge and edge weight = share of passes that produced the query. Hover a query for its intent and routing format.
Query classes
Google's semantic query class for every sub-query: what kind of answer each search is after, simulated vs real.
ChatGPT real searches (4)
Classes come from the query-classifier (a local Laya model). 4 of 4 calls are low-confidence (under 50%) — hover a query in the table to see its confidence.
Searches and sources
ChatGPT4 real searches · gpt-6
generative AI search engines robots meta tags ignore robots.txt AI crawlers publisher controls 2025 2026 Google ↗
Cloudflare AI crawler controls pay per crawl content signals robots.txt AI bots 2025 Google ↗
Google-Extended robots.txt control Gemini training AI Overviews documentation Google ↗
IETF robots exclusion protocol RFC 9309 voluntary crawler compliance Google ↗
- RFC 9309: Robots Exclusion Protocol · rfc-editor.org
- RFC 9309 · datatracker.ietf.org
- Robots Meta Tags Specifications | Google Search Central | Documentation | Google for Developers · developers.google.com
- AI Features and Your Website | Google Search Central | Documentation | Google for Developers · developers.google.com
- Control content use for AI training with Cloudflare’s managed robots.txt and blocking for monetized content · blog.cloudflare.com
- Google's common crawlers | Google Crawling Infrastructure | Crawling infrastructure | Google for Developers · developers.google.com
- Supporting a shared AI training control for websites | Google Search Central Blog | Google for Developers · developers.google.cn
- Introducing pay per crawl: Enabling content owners to charge AI crawlers for access · blog.cloudflare.com
- Pay Per Crawl FAQ · developers.cloudflare.com
- Major publishers sue Meta for copyright infringement over AI training · reuters.com
- Publishers sue Meta over training of AI platform Llama · washingtonpost.com
- Meta loses appeals against interim measures imposed by French regulators · reuters.com
Queries
4 queries · click a column to sort
| Query / type | Search | Intent | Query class | Routing format | Cluster | Freq ↓ | Weight |
|---|---|---|---|---|---|---|---|
generative AI search engines robots meta tags ignore robots.txt AI crawlers publisher controls 2025 2026 OBSERVED | Google ↗ AI Mode ↗ | — | Other / navigational | — | — | 1x of 1 | |
Cloudflare AI crawler controls pay per crawl content signals robots.txt AI bots 2025 OBSERVED | Google ↗ AI Mode ↗ | — | Other / navigational | — | — | 1x of 1 | |
Google-Extended robots.txt control Gemini training AI Overviews documentation OBSERVED | Google ↗ AI Mode ↗ | — | Comparison | — | — | 1x of 1 | |
IETF robots exclusion protocol RFC 9309 voluntary crawler compliance OBSERVED | Google ↗ AI Mode ↗ | — | Reason | — | — | 1x of 1 |