Free SEO Tool

Free Robots.txt Generator

Build a valid robots.txt file in seconds. Set your crawl rules, add your sitemap, and copy or download the result. No sign-up required.

Start from a template

Crawl rules

Block AI crawlers

Blocking these stops the crawler from reading your pages, which also removes you from the AI answers they generate.

Sitemap URL

One URL per line if you have more than one sitemap.

Crawl delay

Seconds

Your robots.txt

User-agent: *
Disallow:

2 lines, 25 characters. Upload this file to the root of your domain at /robots.txt.

How to use the robots.txt generator

1. Pick a template

Start from Allow all, Block all, WordPress or Shopify, then adjust the rules to match your site.

2. Add your rules

Set a user-agent, choose Allow or Disallow, and enter the path. Add your sitemap URL and decide which AI crawlers to block.

3. Copy or download

Your robots.txt builds as you type. Copy it or download the file, then upload it to the root of your domain.

What is a robots.txt file?

A robots.txt file is a plain text file that lives at the root of your domain and tells search engine crawlers which parts of your site they may request. It is the first file Google, Bing and most AI crawlers look for, and it is always public at https://example.com/robots.txt.

The format is deliberately simple. You declare a User-agent, which names the crawler the rules apply to, then list Allow and Disallow paths beneath it. An asterisk as the user-agent means the rules apply to every crawler that does not have its own block.

Robots.txt controls crawling, not indexing. A page you disallow can still show up in search results if other sites link to it, because Google knows the URL exists even though it never fetched the content. When you need a page kept out of the index entirely, allow crawling and add a noindex meta tag so the crawler can actually read the instruction.

It is also not a security tool. The file is public, so listing a private folder in it advertises that folder to anyone curious enough to look. Protect sensitive areas with authentication.

Robots.txt syntax explained

Five directives cover almost every real-world file.

User-agent: names the crawler the following rules apply to. User-agent: * targets every crawler, User-agent: Googlebot targets only Google.
Disallow: blocks any URL starting with the path you list. Disallow: /admin/ blocks the admin folder.
Allow: carves an exception out of a broader Disallow, which is how you block a folder but keep one file inside it crawlable.
Sitemap: points crawlers at your XML sitemap. It needs an absolute URL and applies to the whole file, not to a single user-agent group.
Crawl-delay: asks a crawler to wait a set number of seconds between requests. Bing and Yandex honour it, Googlebot ignores it entirely.

Two patterns come up constantly. To allow everything, use an empty Disallow value:

User-agent: *
Disallow:

For the opposite case, robots txt disallow all is a single slash, which blocks every URL on the domain. Keep this on staging sites only:

User-agent: *
Disallow: /

Paths are case sensitive and match by prefix. You can use * as a wildcard and $ to anchor the end of a URL, so Disallow: /*.pdf$ blocks every PDF.

Adding your sitemap to robots.txt

The robots txt sitemap directive is the simplest way to tell every search engine where your XML sitemap lives. It takes one line and works without you submitting anything anywhere:

Sitemap: https://example.com/sitemap.xml

Two rules matter. The URL must be absolute, including the protocol, because a relative path like /sitemap.xml is invalid and gets ignored. And the directive is independent of user-agent groups, so it can sit anywhere in the file, though the convention is to put it at the bottom.

If you publish several sitemaps, add one Sitemap line each, or point at a sitemap index file that references them all. The generator above accepts one URL per line and writes them out correctly.

Declaring the sitemap here does not replace submitting it in Google Search Console, but it does cover every other crawler that will never see your Search Console account.

Robots.txt for WordPress and other CMS platforms

Most content management systems generate a virtual robots.txt file that you can override by uploading a real one to your web root. The rules worth setting differ by platform.

A robots txt generator for WordPress should block the admin area while keeping the AJAX endpoint open, because plugins and themes rely on it to render content that Google needs to see:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /wp-login.php
Disallow: /?s=

Blocking /?s= keeps internal search result pages out of the crawl, which is one of the most common sources of wasted crawl budget on WordPress sites. Do not block /wp-content/ or /wp-includes/, since that hides the CSS and JavaScript Google needs to render your pages.

On Shopify, the useful blocks are /cart, /checkout, /admin and /search. Both presets are built into the generator above, so you can select one and adjust from there.

Should you block AI crawlers?

Robots.txt is now the main lever site owners have over AI crawlers. Bots like GPTBot, ClaudeBot, PerplexityBot, Google-Extended and CCBot all read the file and honour what it says.

The trade-off is real, and it runs in both directions. Blocking them keeps your content out of model training and out of AI answers. Allowing them means your pages can be read, summarised and cited, which is quickly becoming a meaningful source of qualified traffic.

A useful distinction: some crawlers gather training data, while others fetch pages live to answer a question a user just asked. Blocking the second kind removes you from AI answers your customers are reading right now. Most publishers who want AI visibility allow retrieval crawlers and treat training access as a separate decision.

Whichever way you go, decide deliberately rather than by default, and check the outcome. Tick the crawlers you want to block in the tool above and the correct directives get written for you.

Common robots.txt mistakes

Four errors account for most of the traffic lost to a misconfigured file.

Blocking your whole site by accident

Disallow: / under User-agent: * removes every page from search results. It is the single most expensive robots.txt mistake, and it usually happens when a staging file is pushed to production.

Using robots.txt to hide a page

A disallowed URL can still appear in search results if other sites link to it. To keep a page out of the index, allow crawling and use a noindex meta tag instead.

Blocking CSS and JavaScript

Google renders your pages before ranking them. Blocking /wp-includes/ or your asset folders can make a page look broken to the crawler and cost you rankings.

Using a relative sitemap path

Sitemap: /sitemap.xml is invalid. The Sitemap directive needs a full absolute URL, so write Sitemap: https://example.com/sitemap.xml.

How to publish your robots.txt file

Once the generator has produced your file, four steps put it live.

1Download the file, or copy the output and save it as robots.txt in plain UTF-8 text. The filename is all lowercase.
2Upload it to the root directory of your domain so it resolves at https://example.com/robots.txt. A file in a subfolder is never read.
3Open the URL in a browser to confirm it returns a 200 status and shows the expected content rather than a 404 page.
4Open the robots.txt report in Google Search Console to check Google has fetched the new version and found no parsing errors.

Frequently asked questions

What is a robots.txt generator?

A robots.txt generator is a free tool that builds a valid robots.txt file for you from a simple form, so you do not have to remember the syntax. You choose which crawlers to allow or block, which paths to disallow, and which sitemap to declare, and the generator writes the correctly formatted directives. This robots.txt generator builds the file live in your browser, flags risky rules before you publish them, and lets you copy or download the result with one click.

How to create a robots.txt file?

To create a robots.txt file: (1) open a plain text editor or use the free robots txt generator above, (2) declare a User-agent line naming the crawler the rules apply to, using * for all crawlers, (3) add Allow and Disallow lines listing the paths, (4) add a Sitemap line with the absolute URL of your XML sitemap, (5) save the file as robots.txt with plain UTF-8 encoding, and (6) upload it to the root of your domain so it is reachable at https://example.com/robots.txt. It must sit in the root, because search engines do not look for it anywhere else.

What is robots.txt used for?

Robots.txt tells search engine crawlers which parts of your site they may request. It is used to keep crawlers out of areas that waste crawl budget or should not be indexed, such as admin screens, internal search results, cart and checkout pages, and duplicate faceted URLs. It is also where you declare your sitemap location, and increasingly where site owners decide whether AI crawlers like GPTBot and ClaudeBot may read their content. Robots.txt is a crawling control, not a security measure or a privacy tool.

What does robots.txt disallow mean?

Disallow tells the named crawler not to request URLs that begin with the path you list. Disallow: /admin/ blocks everything under the /admin/ folder. An empty value, written as Disallow: with nothing after it, means the opposite: nothing is blocked, so the crawler may request every URL. Robots txt disallow all is written as Disallow: / under User-agent: *, which blocks the entire site and should only ever be used on a staging or development environment.

What does blocked by robots.txt mean?

Blocked by robots.txt is a Google Search Console status meaning Google wanted to crawl a URL but your robots.txt file told it not to. If the page is genuinely private or low value, that is the correct outcome and no action is needed. If an important page shows this status, a Disallow rule is matching it by mistake, often because the path is broader than intended. Find the matching rule, narrow or remove it, then request indexing again.

How do I see the robots.txt of a website?

Add /robots.txt to the end of any domain and open it in your browser, for example https://example.com/robots.txt. The file is public by design, so you can view the robots.txt of any website, including competitors. If you get a 404, the site has no robots.txt file, which means all crawlers are free to request every URL.

How to remove a robots.txt block?

Open your robots.txt file, find the Disallow line matching the blocked path, and either delete it or make it more specific so it no longer catches the URL you want crawled. You can also override a broad rule with a narrower Allow line, which is how WordPress sites keep /wp-admin/ blocked while still allowing /wp-admin/admin-ajax.php. Save and upload the updated file, then use the URL Inspection tool in Google Search Console to request a fresh crawl. Changes are usually picked up within a day.

Does robots.txt still work?

Yes. Every major search engine, including Google, Bing, DuckDuckGo and Yandex, still reads and follows robots.txt, and Google formalised the protocol as the RFC 9309 standard. It matters more now than it did a few years ago, because the same file is how you grant or refuse access to AI crawlers such as GPTBot, ClaudeBot, PerplexityBot and Google-Extended. What robots.txt has never done is enforce anything: compliance is voluntary, and badly behaved scrapers ignore it entirely.

Is robots.txt legal?

Yes, publishing a robots.txt file is entirely legal and is the long-standing convention for telling crawlers what they may request. It is a published request rather than a legal contract, so it carries no automatic penalty when ignored. Reputable search engines and AI companies honour it as a matter of policy.

Is it illegal to ignore robots.txt?

Ignoring robots.txt is not automatically illegal, because the file is a voluntary convention rather than an access control. That said, courts in various jurisdictions have treated it as evidence of intent in disputes over scraping, and separate laws on copyright, contract and computer access can still apply to how scraped data is used. If you need to genuinely prevent access rather than request it, use authentication, IP blocking or rate limiting, because robots.txt will not stop anyone who chooses to ignore it.

Start free today

Millions of people will ask AI about your category this week

RankSpot researches, writes and publishes daily, and sends you the short list of what's left. Free for 3 days.

Start free trial