- 1 current stop: learning
- 2 upcoming
- 3 upcoming
- 4 upcoming
- 5 upcoming
Finish after stop 5
Stop 1 of 5 · Road closed sign: still on the map
Trick to keep in mind+3 XP
robots.txt closes the road, not the catalogue: to delist, let the bot in and say noindex.
A disallow rule only stops well-behaved bots from fetching a URL. Other sites can still link to it, so it can sit in the index as a bare address with no snippet. To keep a page out of results, leave it fetchable and add a noindex directive. If you block the fetch, the bot never reads that directive.
Read it once, then a question tests whether the trick stuck.