A 40,000-page documentation site had every reason to rank well: original content, genuine topical authority, and years of accumulated backlinks. Instead, less than a third of its pages were showing up in Google's index at all. This is the audit that found out why, and the three fixes that brought it back.
The Symptom: Rankings That Wouldn't Move
The client's SEO team had already tried the usual playbook — more content, better on-page optimization, fresh internal links — and nothing moved the needle. New pages published in January still weren't indexed by April. Some pages that had ranked well a year earlier had quietly dropped out of the index entirely, with no manual action and no obvious quality issue.
That pattern — publishing effort going in, nothing coming back out — is the classic signature of a crawl budget problem, not a content problem. And it's one of the easiest issues to misdiagnose, because everything about the page itself can look perfectly fine.
What Crawl Budget Actually Means
Crawl budget is simply the number of pages Googlebot is willing and able to crawl on your site within a given period. It isn't infinite, and it isn't allocated evenly — Google adjusts it based on your site's perceived quality, its server response speed, and how efficiently your URLs are structured. Waste that budget on duplicate, thin, or low-value URLs, and the pages you actually need indexed simply never get a visit.
Reading the Server Logs
The first real signal came from server log files, not Search Console. Log file analysis shows exactly which URLs Googlebot is requesting, how often, and what it's finding when it gets there — and it's the only way to see crawl behavior with zero guesswork. In this case, over 60% of Googlebot's recorded hits were landing on faceted filter URLs and near-duplicate parameter variations that had no business being crawled at all.
Auditing the Sitemap
The XML sitemap told the rest of the story. It was auto-generated and included every URL the CMS could produce — including thousands of paginated, filtered, and orphaned variants of the same core documentation pages. Googlebot was treating the sitemap as a priority list, and that priority list was pointing it at the wrong 60,000 URLs instead of the right 40,000.
Crawl budget isn't a Googlebot problem. It's almost always a site architecture problem wearing a Googlebot costume. — Richi Meckvan
The Fix: Three Changes That Mattered
Once the log files and sitemap made the problem visible, the fix wasn't complicated — it just required being disciplined about what actually deserved to be crawled.
- Rebuilt the sitemap from scratch, limited strictly to canonical, indexable documentation pages — cutting it from roughly 60,000 URLs down to just over 40,000.
- Blocked low-value parameter URLs via robots.txt and canonical tags, redirecting Googlebot's attention away from filter and pagination variants entirely.
- Rebuilt internal linking so every canonical page was reachable within three clicks of the homepage, closing off the orphaned pages that had never been properly linked in the first place.
The Results
Within twelve weeks, Search Console showed all 40,000 canonical pages correctly indexed — up from roughly a third of that at the start of the audit. Organic sessions to the documentation section followed with a lag of a few weeks, as Google re-crawled and re-evaluated pages that had effectively been invisible for months.
| Metric | Before | After (Week 12) |
|---|---|---|
| Pages indexed | ~13,500 | 40,100 |
| Sitemap URL count | ~60,000 | 40,200 |
| Googlebot hits on low-value URLs | ~60% | <8% |
| Avg. time to index new pages | 8–12 weeks | 3–5 days |
Key Takeaways
- If new content isn't getting indexed and old content is quietly dropping out, check crawl behavior before touching the content itself.
- Server log files are the ground truth for crawl behavior — Search Console alone won't show you what's actually being wasted.
- A smaller, cleaner sitemap almost always outperforms a larger, comprehensive one.
If pages on your site are going stale in the index or new content is taking months to show up, that's usually a crawl issue, not a content one — reach out and I'll take a look at your logs.
