Home › Guides › robots.txt Mistakes That Silently Deindex Pages

robots.txt Mistakes That Silently Deindex Pages

Five robots.txt errors that silently remove pages from Google — a trapped noindex, blocked CSS and JS, wildcards that over-match — plus the five-minute check.

1. Disallowing a page you also marked noindex

This is the most damaging one and it looks reasonable. You want a page out of the index, so you add noindex — and you also disallow it so crawlers stop wasting time. But a crawler that cannot fetch the page can never read the noindex. The URL stays in the index indefinitely, usually with a note saying it was not crawled.

The fix is counterintuitive: to remove a page using noindex, you must allow crawling. Let Google fetch it, see the directive, and drop it. Only then disallow, if you still want to.

2. Blocking CSS and JavaScript

An older habit, and still common in older configs. Google renders pages, and rendering requires your stylesheets and scripts. Block /assets/ or /static/ and Google sees a page with no layout, which it may judge as broken or as a poor mobile experience.

Check your file for broad disallows on asset directories. If they exist, remove them — there is no upside to blocking Google from seeing your own stylesheet.

3. Assuming order decides priority

It does not. Within a user-agent group, the longest matching path wins regardless of line order. So Disallow: /admin/ followed by Allow: /admin/public/ leaves /admin/public/ crawlable. Reordering those lines changes nothing, which is why people conclude the file is being ignored when it is behaving exactly as specified.

4. Wildcard patterns that over-match

Disallow: /*.pdf$ blocks PDFs — and if written slightly wrong, blocks far more. A misplaced * or a missing $ turns a narrow rule into a site-wide block. Because robots.txt produces no warnings, this can run for months before anyone notices the traffic drop.

Test every rule containing a wildcard against a few real URLs before deploying it. That is what the tester on this site is for.

5. Forgetting that staging is one typo away from public

A staging host protected by robots.txt alone is not protected. Any crawler that ignores the file — and there are many — will index it. Duplicate staging content competing with production is a slow, confusing problem to untangle later.

Put real authentication in front of staging. Keep the robots.txt disallow as well, but understand it is politeness, not a lock.

A five-minute check that catches all of these

Fetch your robots.txt and paste it into a tester. Then run five URLs through it: your homepage, a category page, a deep article, one asset from your stylesheet directory, and a URL you believe is blocked. If any of the five gives a verdict you did not expect, you have found the bug — and you have found it before Google did.

Do this again after any deploy that touches routing, and any time someone new edits the file. The failure mode here is silence: nothing breaks visibly, nothing appears in an error log, and traffic from one section of the site quietly stops arriving. A monthly five-minute check is the entire maintenance cost.

Worked example: the block that ate a product section

A small shop migrates its catalogue and, meaning to keep the old staging paths away from crawlers, adds Disallow: /old/. Harmless — until marketing launches a seasonal page at /old-favourites/ under the same prefix. Robots matching is by prefix, not by folder: /old/ as written matches only paths beginning with that exact string, but the rule the team actually typed during the migration was Disallow: /old, without the trailing slash. That one matches /old-favourites/ too. The page is indexed at launch, drops out over the following weeks, and nobody connects the two events because the robots.txt edit predates the launch by months.

The same prefix logic cuts the other way. A blog that disallows /tag/ to save crawl budget also blocks /tag-cloud/ if the rule was written Disallow: /tag. And an Allow written to rescue one page — Allow: /old-favourites/ — does rescue it, because the longest match wins: the allow rule is longer than the disallow, so it governs. Knowing that one fact turns a debugging session into a thirty-second check: print both rules, count the characters, and the longer one is the answer.

If you take one habit from this guide, make it this: any time you add a rule without a trailing slash, say out loud what else begins with those characters. The file will not warn you. It never warns anyone.

About the author

Hongtao Ren (任宏涛) — Developer based in Xi'an, China. Builds browser-based tools and JetBrains IDE plugins. He built and maintains SerpPrism.

Corrections are the most useful thing you can send. If a tool or guide here gives you a wrong answer, that is a bug, not a judgement call — use the contact page.

Questions

How long until a robots.txt change takes effect?

Google usually rereads it within a day, but recrawling affected URLs takes longer. Allow up to a few weeks for the change to fully propagate.

Can robots.txt remove a page already indexed?

No. Use noindex (and allow crawling) or the removals tool in Search Console. A disallow only stops future crawling.

Is a missing robots.txt a problem?

No. Absence means everything is allowed, which is correct for most sites.

Related tools