Robots.txt Generator
Website URL (optional)
Rules
1
User-agent
Directive
Path
Sitemap URLs (one per line)
Crawl Delay (seconds)
How to use Robots.txt Generator
- 1
Add crawl rules
Click Add Rule and set the User-agent, choose Allow or Disallow, and enter the path each rule should apply to.
- 2
Set optional directives
Add your sitemap URLs (one per line) and an optional crawl-delay in seconds to control how search engines crawl your site.
- 3
Generate the file
Click Generate robots.txt to build a correctly formatted file from your rules and directives.
- 4
Copy or download
Use Copy to Clipboard or Download robots.txt, then upload the file to the root of your domain (yoursite.com/robots.txt).
Frequently asked questions
What is a robots.txt file for?
A robots.txt file tells search engine crawlers which parts of your site they may or may not access. It uses User-agent, Allow, and Disallow directives and is placed at the root of your domain so bots read it before crawling.
How do Allow and Disallow rules work?
Disallow blocks crawlers from a path, while Allow explicitly permits one, which is useful for opening a subfolder inside an otherwise blocked directory. You can target all bots with the User-agent for all robots, or write separate rules for specific crawlers like Googlebot.
What does crawl-delay do?
Crawl-delay suggests how many seconds a crawler should wait between requests, which can reduce server load. Note that not all search engines honor it; for example, Google ignores crawl-delay and prefers crawl-rate settings in Search Console.
Why add a sitemap line?
Including a Sitemap directive points crawlers to your XML sitemap so they can discover your pages more efficiently. The tool lets you list multiple sitemap URLs, one per line, and writes them into the file for you.
Does robots.txt guarantee a page stays out of Google?
No. Disallow prevents crawling but a blocked URL can still be indexed if other pages link to it. To keep a page out of search results, use a noindex meta tag or header on a page that is crawlable, rather than relying on robots.txt alone.
Related tools
Keep going with these handy tools