Free Robots.txt Tester
Five robots.txt mistakes that quietly cost traffic
robots.txt looks harmless, which is exactly why mistakes survive for months. Each of these has taken real sites out of search. Paste any of them into the tester above and see for yourself.
1. A staging rule that went live
User-agent: *
Disallow: /Perfect for a test site, disastrous on the real one. Every crawler is told to stay out of everything. Test / and a few real pages after every launch.
2. Blocking CSS and JavaScript
User-agent: *
Disallow: /wp-content/Google needs your styles and scripts to understand a page. Blocking them can make pages look broken to the crawler.
3. A short path that matches too much
Disallow: /pRules match from the start of the path, so this also blocks /products/, /pricing/ and /portfolio/. Use /p/ to block a folder.
4. Thinking the * group still applies to a named bot
User-agent: *
Disallow: /private/
User-agent: Googlebot
Allow: /Googlebot has its own group, so it ignores the * group completely and can crawl /private/.
5. Using robots.txt to hide a page
It stops crawling, not indexing. A blocked URL can still appear in results if other sites link to it. To keep a page out, allow crawling and use a noindex tag.
How the matching rules really work
Once you know three rules, most results stop being surprising.
The longest matching rule wins. If a file says Disallow: /shop/ and Allow: /shop/sale/, the page /shop/sale/shoes is allowed, because the Allow rule is longer and more specific. If an Allow and a Disallow tie in length, Allow wins.
Wildcards. A * matches any run of characters, and a $ at the end means the URL must end there. So Disallow: /*.pdf$ blocks PDF files but not /pdf-guide/.
Case matters in paths, not in directives. Disallow: /Private/ does not block /private/, although disallow and Disallow are treated the same.
Things that live outside the rules
- Each host has its own file.
blog.example.comneeds its own robots.txt. The one onexample.comdoes not cover it. - The file must be at the root, exactly
/robots.txt. A copy in a subfolder is ignored. - Size. Google reads roughly the first 500 KB of the file, so keep it lean.
- If the file returns a server error, crawlers may stop crawling your site until it is fixed. A missing file, by contrast, simply means no restrictions.
A worked example, line by line
Here is a small file and what a tester would tell you about it.
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /search
Disallow: /*.pdf$
User-agent: GPTBot
Disallow: /For a general crawler, /blog/my-post/ is allowed because nothing matches it. /wp-admin/ is blocked. /wp-admin/admin-ajax.php is allowed, because the Allow line is longer than the Disallow line. /search?q=shoes is blocked, since /search matches the start of the path. /files/price-list.pdf is blocked by the wildcard rule, but /files/pdf-guide/ is not, because the rule ends with a dollar sign.
For GPTBot, everything is blocked, and none of the rules in the star group apply to it at all.
Rules that suit a typical WordPress site
- Block
/wp-admin/but allowadmin-ajax.php, which some themes use on the public side. - Consider blocking internal search results, which create endless thin URLs.
- Do not block
/wp-content/uploads/. Your images live there, and blocking them stops them appearing in image search. - Add a line pointing to your sitemap.
A launch day checklist
- Open
yoursite.com/robots.txtand read it in your browser. - Paste it here and test the home page, one article, one product or service page, and your sitemap.
- Make sure there is no
Disallow: /under the star group. - Check Search Console shows the file as fetched with no errors.
Test before and after every change
Paste the file, choose the crawler (Googlebot first), and test your home page, an important product or article, and anything you meant to block. The result table names the exact line that decided each URL. When the file is right, double-check it in Search Console too, since that is Google’s own view. To write a new file, use the robots.txt generator, and for AI bots see the AI crawler access checker.
FAQ
No. You paste the content, so you can also test changes before publishing. Nothing is sent anywhere.
A longer Allow rule may match, or the crawler may use a more specific group that has no such rule.
It follows the same published matching rules, but it is independent. Use Search Console for Google’s own view.
Googlebot for Google Search, Bingbot for Bing, and the exact name of any other bot you want to allow or block.
Crawlers then assume they may visit everything. A missing file is not an error, but a simple one with your sitemap is good practice.
Yes. It follows the standard matching rules, including the * wildcard, the $ end marker and the longest matching rule wins principle.
Crawlers then assume they may visit everything. A missing file is not an error, but a simple one with your sitemap is good practice.
Yes. It follows the standard matching rules, including the * wildcard, the $ end marker and the longest matching rule wins principle.