Index bloat
What is index bloat?
Index bloat is when a search engine has indexed many more pages from a site than the site intends to rank, and most of the extra pages are low value: filter and sort URLs, internal search results, tag archives, parameter duplicates, empty categories, staging copies or old thin pages.
It usually builds up without anyone deciding it. A content management system creates tag and author archives automatically, faceted navigation generates filter URLs, tracking parameters get linked internally, or an old section is left live after a redesign. The site might have 2,000 real pages and 60,000 indexed URLs.
Why index bloat matters for SEO
Google has to crawl and process every URL, so bloat wastes crawl budget on large sites and can slow the discovery of new pages. Many near-duplicate pages also split signals, cause keyword cannibalisation, and make the site look thinner overall. Google’s quality systems look at a site as a whole as well as page by page, so large amounts of unhelpful content can hold back the good pages.
How to fix index bloat
- Compare what you expect to be indexed (your sitemap and the pages you want to rank) with what Google has indexed in the Search Console Pages report. The site: search operator gives only a rough estimate.
- Look for patterns among the extra URLs: parameters, archives, search pages, paginated duplicates, old sections.
- Remove the cause first: stop linking to the URLs and stop the CMS generating them.
- Noindex pages that people need but search does not, such as internal search results; canonicalise duplicates; 301 redirect old pages with a good replacement; return 404 or 410 for pages that should be gone.
- Use robots.txt for URLs that should not be crawled at all, but only after they have dropped out of the index, since a blocked page cannot show Google a noindex.
- Keep your XML sitemap limited to canonical, indexable URLs.
Index bloat FAQ
How do I know if my site has index bloat?
Compare the number of indexed pages in the Search Console Pages report with the number of pages you actually want in search. A large gap, made up of patterns such as parameter or archive URLs, suggests bloat.
Should I block bloated URLs in robots.txt?
Not first. If they are already indexed, Google needs to crawl them to see a noindex or redirect. Block them only once they have left the index, or for new URL patterns you never want crawled.
Related SEO terms
Crawl budget is the set of URLs Google can and wants to crawl on a site. It only becomes a constraint on very large or fast-changing sites.
Faceted navigation lets visitors filter a list by attributes such as size, colour or price, and can create huge numbers of crawlable URLs.
URL parameters are key-value pairs after a ? in a URL, used for tracking, sorting, filtering and pagination. Each variation can look like a new page.
Thin content is a page with little or no useful content of its own, such as near-empty pages, copied text or doorway pages.
Duplicate content is the same or very similar content available at more than one URL, on one site or across sites. Google picks one version to show.
Noindex is a robots directive, set in a meta tag or HTTP header, that tells search engines not to show a page in search results.
Content pruningWhite hat
Content pruning is removing, merging or redirecting low-value pages so a site’s remaining content is stronger and easier to crawl.