Handle keyword overlap and indexing on large sites

The most pages I've indexed on one site:

45,034,907

"but Ian, this site has absolutely tanked to just 1.34m pages", I hear you cry.

Yep - most people will never experience these indexing issues but I expect it at this scale.

It doesn't mean a site is dying, far from it.

This site is doing better than ever:

At this level, keyword cannibalisation is much more common and to protect resources, G might deindex your pages if they are ranking for very similar keywords.

The only real "fix" for this is to make sure you know which your most important pages are and focus on getting (and keeping) those pages indexed.

To do that:

- Consolidate cannibalising pages
- 404 or 410 pages you remove
- Fix soft 404s
- Block crawling of unimportant site sections

And finally - don't build sites with 3b+ fucking pages! 🤣

second place was a very respectable 3,060,009 pages!