Robots.txt Generator

Building a File That Blocks the Right Things

• Never leave Disallow: / active for User-agent: * before launch. It's the default starting point on many staging environments and the single most common way an entire site accidentally disappears from search results.
• Group bots that need identical rules instead of repeating the same block. Multiple User-agent lines in a row share whatever Allow, Disallow and Crawl-delay lines follow — no need for a separate block per bot.
• Use Allow for narrow exceptions inside a broad Disallow, not as a replacement for it. Blocking /wp-admin/ while allowing just /wp-admin/admin-ajax.php keeps the area locked down without breaking site functionality that depends on that one file.
• Don't rely on Disallow to remove a page from search results. A blocked page can still be indexed with no description if it's linked from elsewhere — use a noindex tag on a crawlable page for that instead.

Features

The Mistake That Silently Costs the Most

A robots.txt file is checked by exactly nobody before it ships in most workflows — there's no build step that fails, no linter that flags it, no visible error on the page itself when it's wrong. Disallow: / for User-agent: * is completely valid syntax, reads as an intentional-looking line to anyone skimming the file, and blocks literally every well-behaved crawler from the entire site at once. It's also exactly the line a staging environment's robots.txt often contains by design, meant to keep an unfinished site out of search results — and exactly the line that occasionally survives, unnoticed, into the production deploy. Flagging it the moment it appears, rather than leaving it to be discovered days later in a sudden traffic drop, is the single highest-value check a robots.txt generator can run.

Blocking AI Crawlers Is a Newer, Genuinely Different Decision

GPTBot, CCBot, ClaudeBot and the handful of other major AI training crawlers are a meaningfully different category from Googlebot or Bingbot — allowing a search crawler serves the site's own interest, since it's how the site gets found, while allowing an AI training crawler serves a third party's interest in building a model, with no direct traffic or discoverability benefit flowing back. That's exactly why treating AI crawlers as their own rule group, separate from general search bots, makes sense as a default rather than an afterthought — a site can keep welcoming Googlebot fully while opting specific AI crawlers out, using precisely the multi-user-agent grouping this generator is built around.

Robots.txt Requests; It Doesn't Enforce

Every rule in this file is fundamentally a request, not a lock — it works because major search engines and most legitimate crawlers have agreed, as a matter of established web convention, to check and honor it before fetching anything. Nothing about the HTTP protocol actually prevents a crawler from ignoring robots.txt entirely and requesting whatever it wants; the file has no technical enforcement mechanism behind it at all. This matters for anyone treating robots.txt as a security or access-control tool: it keeps well-behaved crawlers from wasting crawl budget on areas that don't need indexing, and it's the correct, respected way to opt certain content out of AI training crawlers that publish and honor a token — but genuinely private or sensitive content needs real authentication, not a Disallow line, since anyone can simply fetch a disallowed URL directly regardless of what the file says.

Crawl-delay Is a Directive Google Quietly Doesn't Honor

Crawl-delay was originally introduced to let a site owner slow down an aggressive crawler that was consuming too much server bandwidth, and Bing, Yandex and several other engines still respect it as written. Googlebot is the notable exception — Google has stated plainly that it ignores the Crawl-delay directive entirely and instead adjusts its own crawl rate automatically based on how the server actually responds, with manual crawl-rate controls available separately through Search Console instead. Setting Crawl-delay is still worth doing for the crawlers that do honor it, but it's not a reliable way to slow Google down specifically, which is worth knowing before assuming a slow crawl-delay value is protecting server resources from every crawler equally.

An Unofficial Noindex Directive Used to Work Here — It Doesn't Anymore

For years, an undocumented Noindex: line inside robots.txt was quietly honored by Google alongside the two official directives, and a fair number of older tutorials and even some still-running production configs rely on it out of habit. Google formally stopped supporting it in September 2019, alongside several other unsupported rules it had never officially documented in the first place, specifically to standardize on the actual robots.txt specification rather than continuing to support informal extensions indefinitely. Any Noindex: line still present in a live file today is simply inert — it's neither an error nor a working directive, just dead text a crawler skips over, which makes it worth actively replacing with a real noindex meta tag on the page itself rather than assuming it's still doing anything.

From a Parsed or Built File to a Deployed One

A typical path: either start from the platform preset closest to the actual stack, or paste an existing robots.txt into Import if one is already live and just needs updating rather than rebuilding from scratch. From there, add or adjust rule groups directly — checking the warnings panel after every change catches a malformed path or an accidental full-site block before it matters, not after. Once the file looks right, it needs to be saved as robots.txt at the exact root of the domain — /robots.txt, not nested in any subfolder — which is a strict requirement crawlers won't work around. Pairing the finished file with a correctly configured meta tag setup covers both what gets crawled and how what does get indexed actually appears once it's found.

It's worth re-checking the file again after any major site restructuring, not just at launch. A Disallow rule written against an old URL structure can quietly stop protecting anything once paths change during a redesign, while a rule meant to open up a new section can be left blocked by an older, broader Disallow nobody remembered was still there. Robots.txt tends to be written once and forgotten, which is exactly the kind of file most likely to drift out of sync with a site that's still actively changing underneath it. Reviewing it as a routine part of any larger site change, rather than only when something visibly breaks, is the difference between the file actively serving the site and quietly working against it.

Frequently Asked Questions