Why is Google Search Console ignoring our updated robots txt file and continuing to crawl blocked parameter URLs from our faceted navigation?
Download run
Fan-out diagram
The searches the AI actually ran for this prompt.
SEED QUERY
Why is Google Search Console ignoring our updated robots txt file and continuing to crawl blocked parameter URLs from our faceted navigation?
2 real searches · 1 real-search pass
Badge and edge weight = share of passes that produced the query. Hover a query for its intent and routing format.
Query classes
Google's semantic query class for every sub-query: what kind of answer each search is after, simulated vs real.
ChatGPT real searches (2)
Classes come from the query-classifier (a local Laya model). 2 of 2 calls are low-confidence (under 50%) — hover a query in the table to see its confidence.
Searches and sources
ChatGPT2 real searches · gpt-6
Google Search Central robots.txt caching 24 hours robots.txt report crawl blocked URLs faceted navigation parameter URLs Google ↗
Google Search Central robots.txt disallow URL removal indexing blocked pages crawl behavior Google ↗
- Managing crawling of faceted navigation URLs | Google Crawling Infrastructure | Crawling infrastructure | Google for Developers · developers.google.com
- How Google Interprets the robots.txt Specification | Google Crawling Infrastructure | Crawling infrastructure | Google for Developers · developers.google.com
- Crawl Stats report · support.google.com
- Page indexing report · support.google.com
- Robots.txt Introduction and Guide | Google Search Central | Documentation | Google for Developers · developers.google.com
Queries
2 queries · click a column to sort
| Query / type | Search | Intent | Query class | Routing format | Cluster | Freq ↓ | Weight |
|---|---|---|---|---|---|---|---|
Google Search Central robots.txt caching 24 hours robots.txt report crawl blocked URLs faceted navigation parameter URLs OBSERVED | Google ↗ AI Mode ↗ | — | Other / navigational | — | — | 1x of 1 | |
Google Search Central robots.txt disallow URL removal indexing blocked pages crawl behavior OBSERVED | Google ↗ AI Mode ↗ | — | Reason | — | — | 1x of 1 |