Robots.txt Generator
1
User-agent
Directive
Path
Crawl Delay (seconds)
How to use Robots.txt Generator
- 1
Add crawl rules
Click Add Rule and set the User-agent, choose Allow or Disallow, and enter the path each rule should apply to.
- 2
Set optional directives
Add your sitemap URLs (one per line) and an optional crawl-delay in seconds to control how search engines crawl your site.
- 3
Generate the file
Click Generate robots.txt to build a correctly formatted file from your rules and directives.
- 4
Copy or download
Use Copy to Clipboard or Download robots.txt, then upload the file to the root of your domain (yoursite.com/robots.txt).
How robots.txt Really Works: A Practical Guide
What robots.txt is and where it must live
robots.txt is a plain text file that implements the Robots Exclusion Protocol, a voluntary standard that tells web crawlers which parts of your site they should not request. It must be served from one exact location: the root of your domain, at yoursite.com/robots.txt. A robots.txt placed in a subdirectory, such as yoursite.com/blog/robots.txt, is simply ignored. Each subdomain and each protocol has its own scope, so blog.yoursite.com needs its own file separate from the one on yoursite.com.
Compliance is honor-system. Googlebot, Bingbot, and other mainstream crawlers obey robots.txt, but malicious scrapers and many AI bots may not. Never treat robots.txt as a security control. It is a politeness mechanism for well-behaved crawlers, not a lock on private content; anything truly sensitive needs authentication, not a Disallow line.
The four directives you actually use
A robots.txt is organized into groups, each beginning with one or more User-agent lines that name the crawler the rules apply to, followed by Allow and Disallow rules. User-agent: * targets every crawler that does not have a more specific group. Disallow: /admin/ blocks a path, and an empty Disallow (Disallow: with nothing after it) means allow everything. Allow is used to carve an exception out of a broader block, for example permitting one file inside an otherwise disallowed folder.
The Sitemap directive is special: it is independent of any User-agent group and can appear anywhere in the file, conventionally at the top or bottom. It should point to the full absolute URL of your XML sitemap, such as https://yoursite.com/sitemap.xml, and you can list several sitemap lines if you have more than one. This is the cheapest way to help search engines discover all of your URLs.
The critical distinction: crawling versus indexing
This is the single most misunderstood point about robots.txt, and getting it wrong causes real damage. robots.txt controls crawling, not indexing. Disallowing a URL stops crawlers from fetching its contents, but it does not remove the URL from search results. If other pages link to a blocked URL, Google can still index it as a bare link, often shown with the unhelpful note that no description is available because the page is blocked by robots.txt.
Worse, blocking a page in robots.txt prevents Google from ever seeing a noindex tag on that page, because it cannot crawl the page to read the tag. So the correct way to remove a page from search is the opposite of what many people assume: leave it crawlable and add a noindex meta tag or X-Robots-Tag header. Only block in robots.txt when your goal is to save crawl budget, not to hide a page from the index.
Wildcards and pattern matching
Modern crawlers support two pattern characters. The asterisk (*) matches any sequence of characters, and the dollar sign ($) anchors a match to the end of the URL. So Disallow: /*.pdf$ blocks every URL ending in .pdf, and Disallow: /*?sort= blocks any URL containing a sort query parameter, which is a common way to keep faceted-navigation duplicates out of the crawl. Paths are matched as prefixes, so Disallow: /private blocks /private, /private/, and /private-files alike.
Patterns are powerful but easy to over-apply. Test any non-trivial rule before deploying it; a single misplaced asterisk can block far more than intended. Google Search Console includes a robots.txt tester that shows whether a given URL is allowed or blocked by your rules, which is worth using for anything beyond the simplest file.
Crawl-delay and crawl rate
The Crawl-delay directive asks a crawler to wait a number of seconds between successive requests, which can ease load on a small or fragile server. The catch is that support is inconsistent: Bing and Yandex honor Crawl-delay, but Google ignores it entirely. Google instead determines crawl rate automatically and historically exposed a manual override in Search Console. If your concern is Googlebot specifically hammering your server, Crawl-delay will do nothing; you need server-side rate limiting or Search Console settings instead.
For most sites, you should not set a crawl delay at all. Slowing crawlers down also slows how quickly new and updated content gets discovered and indexed. Reserve it for genuine capacity problems, and keep the value modest.
Common mistakes that quietly tank SEO
The most catastrophic error is shipping a staging file that contains Disallow: / under User-agent: *. That single line blocks your entire site from every crawler, and it routinely happens when a site goes live and nobody swaps out the development robots.txt. After any launch or migration, the first thing to check is that your production robots.txt does not contain a blanket Disallow.
The second classic mistake is blocking CSS and JavaScript directories. Google renders pages like a browser, and if it cannot fetch your stylesheets and scripts it may see a broken, unstyled page and judge it as having a poor mobile experience or thin content. Let crawlers reach your assets. Other frequent slip-ups include using robots.txt to try to deindex a page (use noindex instead), forgetting that the file is case-sensitive in its paths, and putting the file anywhere other than the domain root.
A sensible default for most websites
Many sites are best served by a deliberately permissive robots.txt: a single group of User-agent: * with an empty Disallow so everything is crawlable, plus a Sitemap line pointing to your XML sitemap. From that open baseline you then add narrow Disallow rules only for things that genuinely should not be crawled, such as internal search result pages, admin areas, cart and checkout URLs, and parameterized duplicate URLs.
Resist the temptation to over-engineer. A short, correct robots.txt that you fully understand is far safer than a long one copied from elsewhere with rules you cannot explain. After generating yours, fetch it in a browser to confirm it is served as plain text at the root, then validate the important paths in a tester before relying on it.
Frequently asked questions
What is a robots.txt file for?
How do Allow and Disallow rules work?
What does crawl-delay do?
Why add a sitemap line?
Does robots.txt guarantee a page stays out of Google?
Related tools
Keep going with these handy tools