Home › Tools › robots.txt Tester
robots.txt Tester
Paste a robots.txt and test any URL against any user-agent. See exactly which rule wins, which line matched, and whether Googlebot can crawl the page.
SerpPrism's robots.txt Tester is a free online tool that tests any URL against any user-agent using your robots.txt rules, then shows exactly which rule wins and which line matched. No sign-up is needed and the test runs in your browser. It is for site owners and SEOs who need to know whether Googlebot can crawl a page before a small rules mistake becomes an indexing problem. Paste your robots.txt on the left, then enter a URL and a user-agent. The tester tells you allow or disallow, and — more usefully — quotes the exact rule that decided it, so you can find the line that is quietly blocking a section of your site.
How to use it
- Paste the full contents of your robots.txt (fetch it from
/robots.txtfirst). - Enter the path you want to check and pick a user-agent.
- Read the verdict — it names the winning rule and the line number it came from.
Longest match wins, not first match
This is the single most misread part of the spec. Crawlers compare every matching Allow and Disallow rule and follow the longest one. So Disallow: /admin/ plus Allow: /admin/public/ means /admin/public/ is crawlable, even though the disallow line comes first.
If you have ever 'fixed' a block by moving a line up or down in the file, this is why it did not work. Order inside a group is irrelevant; path length is what decides.
robots.txt blocks crawling, not indexing
A disallowed URL can still appear in search results if something else links to it — Google will show it with a 'page not crawled' note because it never read the content. If you need a page gone from the index, use noindex, and do not disallow it at the same time, or the crawler can never see the noindex.
This pairing — disallow plus noindex — is one of the most common ways sites accidentally keep dead pages in the index indefinitely.
A worked example: the same URL, two different verdicts
A file has two groups. Googlebot's group says Disallow: /staging/; the * group says Disallow: — nothing blocked. Test /staging/new-page with Googlebot selected and the verdict is BLOCKED, with the winning rule quoted: /staging/, line 2. Test the same URL with * and the verdict is ALLOWED.
Same URL, opposite answers, and both are correct — because a crawler reads only its own group and ignores the rest of the file. That is why the tester shows the matched group next to the verdict: the answer is meaningless without knowing which group produced it.
The practical lesson is to test with the user-agent you actually care about. Checking a staging URL against * tells you nothing about what Googlebot will do, and checking it against Googlebot tells you nothing about Bingbot. Run the URL once per crawler that matters.
What this tool deliberately does not do
It does not fetch the robots.txt from your domain for you. Browsers block reading a file from another origin unless that origin explicitly allows it, and routing the request through a proxy would mean this site could see every URL you test — exactly what a privacy-respecting tool should not be able to do.
It also does not know about rules that live outside robots.txt: noindex in a meta tag, an X-Robots-Tag response header, or password protection. A URL can be fully allowed by robots.txt and still never appear in search for any of those reasons.
Treat the verdict as 'what your robots.txt says', which is one layer of the picture, not the whole one.
Questions
Does Disallow match by prefix?
Yes. A rule of /blog also blocks /blogging and /blog/anything. Add a trailing slash when you mean the directory only, and remember that wildcards * and $ are supported by Google.
What if no rule matches?
The URL is allowed. Absence of a matching Disallow is an implicit allow — you do not need Allow: / to permit crawling.
Do all crawlers obey robots.txt?
Major search engines do. Many AI training crawlers and scrapers do not. robots.txt is a request, not an access control mechanism — anything that must stay private needs real authentication.
How do I check if Googlebot can crawl my page?
Paste your robots.txt into a tester, enter the URL, and choose Googlebot as the user-agent. The tester applies the longest-match rule — the one crawlers actually follow — and tells you allowed or blocked, with the exact line responsible. SerpPrism's tester does this without touching your live site.
Should I block a page with robots.txt or use noindex?
They solve different problems. robots.txt stops crawling but the URL can still appear in results without a snippet; noindex removes the page from results but requires the crawler to reach it — so never block a URL you also need noindexed. Test the combination in the tester before shipping it.
Where should the robots.txt file be placed?
At the root of the host, e.g. https://yoursite.com/robots.txt, served as plain text. Subdomains and ports each need their own file, and a file anywhere else is simply not found by crawlers.
Can I test a robots.txt that is not live yet?
Yes — paste the draft. The tester never fetches anything; it evaluates the text you give it, so a file still in your editor behaves exactly like a live one. Test before you publish, not after.
Why does the tester disagree with Google Search Console?
Search Console tests the live file Google last fetched; this tester evaluates the text you pasted. If they differ, the live file changed since you copied it, or Google is working from a cached copy. Fetch /robots.txt fresh and paste it again.
Related tools
Robots.txt Generator
Build a robots.txt file rule by rule: choose crawlers, list the paths to block or allow, add your sitemap, then copy or download the file.
Meta Tag Generator
Generate clean page title, meta description, canonical, Open Graph and Twitter card tags.
Heading Structure Analyzer
Paste HTML and get a clean H1–H6 outline plus a list of real problems: missing or duplicated H1, skipped levels, and headings that are too long.
Related guides
robots.txt Mistakes That Silently Deindex Pages
Five robots.txt errors that silently remove pages from Google — a trapped noindex, blocked CSS and JS, wildcards that over-match — plus the five-minute check.
What 71 Sites Actually Put in robots.txt
We fetched the robots.txt of 78 high-traffic sites. A quarter declare no sitemap at all, and five still ship a directive Google ignores. The fix takes one line.