robots.txt Generator
Build a valid robots.txt with per-crawler allow/disallow rules, wildcard paths, sitemap locations and crawl-delay — with a live preview and warnings for the mistakes that actually bite. 100% in your browser.
Pick a preset or start from scratch: add crawler groups (Googlebot, Bingbot, the AI-training bots, or your own),
set Allow/Disallow rules with * and $ wildcards, point at your
sitemap, and watch the finished file appear on the right. The checker flags invalid syntax and the classic
mistakes — blocking your whole site, regex characters crawlers ignore, missing leading slashes —
before you upload anything.
Crawler rules
Each group starts with a User-agent line; every rule below it applies to that crawler.
* means “all crawlers”.
Sitemaps
Absolute URLs of your XML sitemaps (or sitemap index). Crawlers use these to find your pages.
Crawl-delay (optional, seconds)
Asks a crawler to wait between requests. Google and Bing ignore it; Baiduspider, YandexBot and some others honour it.
Your robots.txt
What is robots.txt?
robots.txt is a plain-text file at the root of your site —
https://your-site.com/robots.txt — that tells search-engine crawlers which parts of
the site they may request. Crawlers fetch it before they crawl and then cache it for a while (hours to
days, depending on the engine and how long the Cache-Control header says), so changes to
the file are not applied instantly.
It is a polite request, not a command. Well-behaved crawlers such as Googlebot and Bingbot follow it; nothing technically stops a determined crawler from ignoring it.
⚠️ robots.txt is not a security control
Three things people regularly get wrong:
-
It does not hide content. Anyone can fetch
/robots.txtand read exactly what you are blocking — listing a path often tells scanners precisely where the interesting stuff is. -
It does not deindex pages. If a blocked page is linked from elsewhere (or
appears in a sitemap), Google can still index it without crawling it — and a blocked page
can even get a more prominent “mystery page” title in results. To keep a page out
of search results, use a
noindexmeta tag or HTTP header, and keep the page crawlable so the tag can be read. -
It does not protect sensitive data. Put anything that must not be public
behind authentication (login walls, IP restrictions, server-level rules) — never behind
Disallow. robots.txt only manages crawling behaviour for bots that choose to cooperate.
How the rules work
| Directive | Meaning |
|---|---|
User-agent: * | The rules below apply to every crawler. A specific name (e.g. Googlebot) applies only to that crawler. |
Disallow: /path/ | Do not request anything under this path. An empty value (Disallow:) allows everything. |
Allow: /path/ | Do request this path, even if a broader Disallow covers it. |
Sitemap: https://… | Where your XML sitemap lives. Google, Bing and the other major engines read this line. |
Crawl-delay: 5 | Ask a crawler to wait 5 seconds between requests. Google and Bing ignore it; Baiduspider, YandexBot and some others honour it. |
Wildcards
Only two wildcard characters exist in robots.txt: * (matches any string, including an
empty one) and $ (matches the end of the path). They are not regular expressions —
?, +, | and brackets are silently ignored by crawlers.
| Rule | Blocks / allows |
|---|---|
Disallow: /private/ | Everything under /private/ |
Disallow: /*.pdf$ | All PDF files, at any depth |
Disallow: /images/ + Allow: /images/logo.png | The whole images folder except one file |
Disallow: /search?q= | Nothing — ? is not a wildcard; the rule is literal (and usually not what you meant) |
Which rule wins?
When a URL matches both an Allow and a Disallow rule, the longest
(most specific) path wins. If both are exactly the same length, Allow wins.
That is how “block a folder but keep one file public” works.
Common mistakes
- Blocking everything on a live site —
Disallow: /under*keeps every page out of search results. It is the right move for a staging site and the worst move for a production one. - Forgetting the leading slash — paths must start with
/;admin/is not a valid path. - Using regex syntax —
?,+,|and friends are ignored; only*and$work. - Thinking robots.txt is security or deindexing — it is neither (see above). Use authentication and
noindexfor those jobs. - Blocking CSS, JavaScript or images — search engines need them to render and understand your pages; blocking them can hurt rankings.
- Not listing your sitemap — the
Sitemap:line is the most reliable way to get every page discovered. - Expecting instant effect — crawlers cache robots.txt; give changes a few hours (or a few days) to take hold.
Where to test it
After you upload the file, verify how the engines actually see it: in Google Search Console
use URL Inspection (it shows the robots.txt the crawler fetched), and in Bing Webmaster Tools
use the robots.txt checker. The quickest sanity check is to open
https://your-site.com/robots.txt in a browser and read it the way a crawler would.
Suggested
