Skip to main content
PProfileMatch
← Path

SEO Fundamentals · SEO Fundamentals: Technical SEO, Sitemaps and Search Console

0 XP
  1. 1 current stop: learning
  2. 2 upcoming
  3. 3 upcoming
  4. 4 upcoming
  5. 5 upcoming

Finish after stop 5

Stop 1 of 5 · Road closed sign: still on the map

Trick to keep in mind+3 XP

robots.txt closes the road, not the catalogue: to delist, let the bot in and say noindex.

A disallow rule only stops well-behaved bots from fetching a URL. Other sites can still link to it, so it can sit in the index as a bare address with no snippet. To keep a page out of results, leave it fetchable and add a noindex directive. If you block the fetch, the bot never reads that directive.

Read it once, then a question tests whether the trick stuck.