Home / SEO Tools / Robots.txt Generator

Robots.txt Generator

Create a robots.txt file with your rules, sitemap and optional extra crawler blocks, and download it in one click.

Save the file as robots.txt in the root of your site so it loads at yourdomain.com/robots.txt. Google ignores the crawl-delay line, though some other crawlers respect it. Blocking a page in robots.txt stops crawling, but it does not reliably keep the page out of search results. Use a noindex tag for that.

What robots.txt does

A robots.txt file sits at the root of your site and tells crawlers which paths they may or may not request. It is the first file well-behaved bots read when they visit.

How to use this generator

Choose a default rule, add any paths to block or allow, list extra bots to block and add your sitemap URL. The WordPress button adds the common defaults. Download the file and upload it to your site root.

Common mistakes

Do not block CSS or JavaScript files that pages need to render. Do not use robots.txt to hide private content, since the file itself is public. To keep a page out of search results use a noindex tag instead of a block.

FAQ

At the root of your domain, so it loads at yourdomain.com/robots.txt.

No. Google ignores crawl-delay, though some other search engines respect it.

More free tools

Scroll to Top