Search

Your search problem is probably an architecture problem

Four hundred good pages, ninety thousand crawlable URLs, and a crawler spending its budget on the wrong ninety thousand.

Projection Developers7 minute read

When organic traffic is flat, the usual conclusion is that the content is not good enough. Sometimes that is right. More often the content is fine and it is unreachable, duplicated, or competing with itself.

Look at what is being crawled, not what exists

Site structure as designed and site structure as crawled are different documents. Faceted navigation, session parameters, sort orders and print views can turn a few hundred real pages into tens of thousands of URLs, and a crawler will happily work through them.

The number to find is what share of crawl requests reach a page you would actually want someone to land on. Under ten percent is common, and it is not a content problem.

Consolidation beats production

Sixty pages that each half-answer a question will lose to eighteen that answer it. Merging them is unpopular because it looks like deleting work, and it is usually the single highest-return action available.

Deleting content is the least popular recommendation in search and frequently the most effective one.

Then structure, then coverage

  • Make what exists reachable and singular.
  • Organise it around what a person would search for, not around your org chart.
  • Only then decide what is genuinely missing.

Reversing that order is why so much search work reads as superstition. Publishing into a broken architecture produces more broken architecture, on schedule, at cost.