Free SEO Tool

Free Robots.txt Checker

Read any site's robots.txt, validate its rules, and test whether a specific URL is blocked. Works on any domain, no verified property and no sign-up required.

How to check a robots.txt file

Three steps, and only the first is required:

1. Enter a domain

Just the domain, for example rankspot.ai. We find the file at /robots.txt for you, so there is no need to paste the full path.

2. Optionally test a URL

Add a path such as /blog/my-post and pick a crawler. This is how you find out whether a specific page is actually blocked.

3. Read the verdict

You get the parsed rules, any problems we spot, the raw file, and for a tested URL the exact directive that decided its fate.

What is robots.txt?

robots.txt is a plain text file at the root of a domain that tells crawlers which parts of the site they should not request. It has to live at exactly one place, the root, so example.com/robots.txt is read and example.com/pages/robots.txt is ignored. Every subdomain needs its own.

The most important thing to understand about it is what it does not do. Disallow prevents crawling, not indexing. A URL blocked in robots.txt can still show up in search results if other pages link to it, usually with no description because the crawler was never allowed to read it. If a page must stay out of the index, use a noindex tag, and if it must stay private, put it behind authentication.

It is also a public file, on every site. That is by design, and it is why you can check a competitor's rules as easily as your own.

What this robots.txt tester reports

More than the raw file. It parses the rules, validates them, and answers the question you actually came with:

The file itself

Whether robots.txt exists at all, the HTTP status it returned, its size, and the raw contents with a copy button.

Rules grouped by crawler

Every allow and disallow, grouped by the user-agent it applies to, plus any crawl-delay. Groups sharing a user-agent are merged, as crawlers merge them.

A verdict on any URL

Test a path against the rules and get allowed or blocked, plus the exact directive and line number that decided it.

Problems worth knowing about

An accidental site-wide block, rules written before any user-agent line, missing colons, unknown directives and a missing sitemap declaration.

How robots.txt rules are actually applied

Rules are not read top to bottom, which surprises people. Google resolves conflicts by specificity, and this checker implements the same logic:

Rule	Example	Outcome
Longest matching rule wins	Disallow: /admin/ and Allow: /admin/public/	/admin/public/x is allowed
Allow wins an exact tie	Disallow: /x/ and Allow: /x/	/x/y is allowed
Nothing matches	No rule covers the path	Crawling is allowed by default
Most specific user-agent wins	Rules for both * and Googlebot	Googlebot ignores the * group entirely

Wildcards follow the same idea: * matches any run of characters and a trailing $ anchors the match to the end of the URL, so /*.pdf$ blocks /report.pdf but not /report.pdf?v=2. Paths are case-sensitive even though directive names are not.

Fixing a blocked by robots.txt error

If Search Console reports that a page is blocked, work through it in this order:

1Find the rule. Enter the blocked path in the URL test above. It names the exact directive and line number responsible, which is usually the whole problem.
2Decide whether the block is wrong. Plenty of blocked URLs are blocked deliberately. The error is only an error if you wanted that page crawled.
3Fix it by narrowing, not deleting. Remove the rule, tighten it so it no longer matches, or add a more specific Allow. A longer matching rule always beats a shorter one.
4Re-check. Run the same URL again and confirm the verdict flipped before waiting on a recrawl.

If the file needs rewriting rather than patching, our free robots.txt generator builds a valid one from scratch. To see whether the affected pages currently rank at all, try the SERP checker.

Common robots.txt mistakes

Six that cost real traffic, the first far more than the rest combined:

Disallow: / left in after launch

The single most expensive robots.txt mistake. A staging file copied to production blocks the entire site from every crawler. We flag this as an error.

Expecting robots.txt to hide a page

Disallow stops crawling, not indexing. A blocked URL can still appear in results if other pages link to it. Use a noindex tag or authentication instead.

Blocking CSS and JavaScript

Google renders pages to understand them. Blocking the assets a page needs to render can make it look broken and hurt how the page is assessed.

Assuming rules are case-insensitive

Directive names are not case-sensitive but paths are. Disallow: /Admin does not block /admin.

Forgetting the sitemap line

Declaring your sitemap in robots.txt is a free, reliable way to help crawlers find your pages. We report when it is missing.

Putting robots.txt somewhere else

It only works at the root of a domain. A file at /pages/robots.txt is ignored, and subdomains each need their own.

Frequently asked questions

How do I check if robots.txt exists?

Enter your domain in the checker above and it will tell you, including the HTTP status the file returned. You can also visit yourdomain.com/robots.txt directly in a browser. A 404 there means no file exists, which is not an error: with no robots.txt, crawlers treat the whole site as crawlable.

How to see a robots.txt file?

Add /robots.txt to the end of any domain and open it in a browser. It is a public file by design, so you can read it for any site, including competitors. The checker above does the same thing and additionally parses the rules into something readable.

How do I view a robots.txt file?

The same way for any site: yourdomain.com/robots.txt. It is plain text, so it opens straight in the browser. If you want the rules grouped by crawler and validated rather than raw, paste the domain into the tool at the top of this page.

Where can I find the robots.txt file on my website?

It lives at the root of your domain and nowhere else. On most sites that means the root directory of the web server, though platforms like WordPress, Shopify and Webflow often generate it from a settings screen instead of a real file. A file anywhere other than the root is ignored, and every subdomain needs its own.

How to fix blocked by robots.txt error?

Find the rule causing it, which is exactly what the URL test above does: enter the blocked path and it names the directive and line number responsible. Then either remove that rule, narrow it so it no longer matches the URL, or add a more specific Allow, since a longer matching rule beats a shorter one. Re-check afterwards to confirm.

Why is my page blocked when my robots.txt looks correct?

Almost always because a different rule than the one you are looking at is winning. Rules are not applied top to bottom: the longest matching pattern wins, and a rule in a crawler-specific group overrides the wildcard group entirely. Enter the blocked path in the test above and it will name the directive and line number actually responsible.

Does this checker follow Google's matching rules?

Yes. It implements the behaviour Google documents: wildcards with *, end-of-URL anchoring with $, the longest matching rule winning, Allow beating Disallow on an exact tie, and the most specific user-agent group taking precedence over the wildcard group. An empty Disallow value is treated as allowing everything rather than as a rule.

What happens if robots.txt is missing or returns an error?

A 404 means no restrictions, so crawlers treat the whole site as crawlable, and this tool reports that as a normal result rather than a failure. A 5xx is different: crawlers may treat a persistently failing robots.txt as though everything is disallowed, so a server error on that file can quietly stop your site being crawled. The checker shows the exact status code returned so you can tell the two apart.

Can I check the robots.txt of another website?

Yes. robots.txt is public on every site that has one, so you can check a competitor's without any special access. That is a genuine advantage over Google Search Console's tester, which only works on properties you have verified.

Which crawlers can I test against?

Googlebot, Googlebot Image, Bingbot, DuckDuckBot and the wildcard group, plus the AI crawlers GPTBot, ClaudeBot and PerplexityBot. The crawler matters because a site can set different rules for each, and the most specific matching user-agent group wins outright.

Is this robots.txt checker free?

Yes. Several free checks per day, no account and no sign-up, with the full parsed output every time.

Start free today

Millions of people will ask AI about your category this week

RankSpot researches, writes and publishes daily, and sends you the short list of what's left. Free for 3 days.

Start free trial