Google: 75% of Crawling Issues Caused by Two Common URL Errors

In a newly released post-mortem of the 2025 search landscape, Google’s technical lead, Gary Illyes, has identified a critical bottleneck for modern websites. According to Google’s year-end data, 75% of all crawling problems are now traced back to just two structural flaws: faceted navigation and action parameters.

These aren’t just minor technical debt; they are “crawling loops” that can swell a site from 1,000 pages to a million phantom URLs in a matter of hours, potentially melting server “pipes and whistles” and stalling your organic visibility.

The Anatomy of the 2026 Crawl Crisis

Based on the latest Search Off the Record findings, the waste is divided into a clear hierarchy of technical inefficiency:

• The 50% Heavyweight: Faceted Navigation. Common in e-commerce, this occurs when filters (size, color, brand, price) create near-infinite URL combinations. A single product line can generate thousands of URLs that lead to the same—or empty—content.
• The 25% Saboteur: Action Parameters. These are URLs that perform a function (like ?add-to-wishlist=true or ?sort=price) rather than displaying unique, indexable content.
• The 10% Noise: Irrelevant Tags. Session IDs and UTM tracking parameters that provide zero value to a searcher but occupy a bot’s attention.
• The 5% Plugin Bloat: Automated URLs generated by CMS widgets or plugins that confuse crawler logic.
• The 2% Edge Cases: Double-encoded URLs and “weird” server glitches that account for the remaining crawl failures.

Why “Bot Traps” are Harder to Fix Than You Think

The danger lies in Googlebot’s inherent design. As Gary Illyes explained, a crawler is “curious by nature.” It cannot know whether a specific section of your site is a waste of time until it has already crawled a large portion of it.

By the time Googlebot realizes it is stuck in a filtered navigation loop, your server might already be overloaded, and your budget for crawling high-value, new content will have been exhausted. Once a bot is trapped, recovery isn’t instant—it takes time for the “search brain” to unlearn those low-value paths.

The Blueprint for a Clean Crawl in 2026

Maintaining a lean URL structure is now the primary lever for server health. To avoid the traps identified in the 2025 report, experts recommend four foundational fixes:

1. Embrace the Fragment (#): Use hashtags for UI-only changes (like sorting or color filters). Since Google generally ignores everything after the #, your crawl space remains static while the user experience stays dynamic.
2. Hard-Stop Robots.txt Rules: Don’t leave it to chance. Explicitly Disallow common action parameters like *?*add-to-cart=* to prevent bots from ever entering the transaction funnel.
3. 404 for “Empty” Combinations: If a user filters for “Purple” shoes in “Size 14” and there is no stock, return a 404 Not Found status. Serving a “No Results Found” page with a 200 OK status encourages bots to keep digging.
4. Consistently Ordered Parameters: Ensure that /shoes?color=red&size=10 is the only version of that page. Allowing /shoes?size=10&color=red effectively doubles your site size in the eyes of a crawler.