Robots.txt Generator

How to Use the Robots.txt Generator

Define per-crawler rules, list the paths you want to allow or block, attach your sitemap, and copy the generated file straight into the root of your website. The entire generator runs in your browser, so no URL patterns or site structure are transmitted anywhere.

1. Choose a user-agent — Start with * to apply rules to every crawler, or specify a named bot such as Googlebot, Bingbot, DuckDuckBot or GPTBot when you need per-bot behavior.
2. Add Disallow paths — List the URL prefixes you want to keep out of crawlers, one per line. A path of /admin/ blocks everything under that folder; a bare / blocks the entire site.
3. Add Allow exceptions — Use Allow entries to carve out specific paths that should remain crawlable even though a broader Disallow applies. Allow rules take precedence when they are more specific than the matching Disallow.
4. Attach your sitemap — Paste the absolute URL of your sitemap.xml. The Sitemap directive is global, so a single entry applies to all user-agents.
5. Copy or download — Copy the generated text or download the file, then upload it to the root of your domain so it is reachable at https://yourdomain.com/robots.txt.

About the Robots Exclusion Protocol

The Robots Exclusion Protocol (REP) began as an informal convention proposed by Martijn Koster in 1994 after his own server was overwhelmed by an early web crawler. For nearly three decades it remained a de facto standard, and every major search engine agreed on the file format and location. In September 2022 the IETF published RFC 9309, which finally standardized the protocol and formalized the syntax for user-agent groups, allow and disallow rules, and path matching.

Robots.txt works on trust — well-behaved crawlers read the file and obey its directives, but it is not an access-control mechanism. A determined scraper or a malicious bot can ignore the file entirely, which is why sensitive material must be protected by authentication or IP restrictions rather than a Disallow rule.

The file must be served as plain UTF-8 text from the root of the host, with a maximum size of 500 kibibytes according to RFC 9309. Crawlers typically cache robots.txt for up to 24 hours, so changes can take a day to take effect across all bots.

Examples

A minimal file that allows full crawling and points to a sitemap:

User-agent: *
Disallow:
Sitemap: https://example.com/sitemap.xml

A file that blocks the admin area and private uploads for every crawler, but still lets Googlebot reach a public PDF inside the blocked folder:

User-agent: *
Disallow: /admin/
Disallow: /uploads/private/
User-agent: Googlebot
Allow: /uploads/private/whitepaper.pdf
Disallow: /uploads/private/
Sitemap: https://example.com/sitemap.xml

How Path Matching Actually Works

Paths are matched as case-sensitive prefixes against the part of the URL after the host, so Disallow: /Admin does not block /admin, and a rule of /blog sweeps up /blog/, /blog-archive and /blogroll alike. For the folder alone, write Disallow: /blog/ with the trailing slash. The asterisk * matches any run of characters and $ anchors a URL end, so /*?sort= blocks sorted duplicates and /*.css$ re-allows stylesheets. When an Allow and a Disallow both match, RFC 9309 gives the win to the rule with more path characters, ties going to Allow.

A Worked Example and Common Mistakes

On WordPress, block only /wp-admin/ while allowing /wp-admin/admin-ajax.php, and leave /wp-content/ crawlable because it holds the assets Google needs to render the page. On a faceted catalog, block the parameter URLs that multiply infinitely (/*?color=, /*?sort=) while clean canonical URLs stay open. Three classic mistakes: a leftover Disallow: / from a staging build silently deindexes a site; a public Disallow advertises the very file it tries to hide; and blocking an already-indexed page freezes the stale listing because Google can no longer fetch the noindex that would drop it.

When to Use a Robots.txt File

• Keep crawl budget focused — Steer bots away from infinite faceted-search URLs, internal search result pages, or archive calendars that waste crawl budget.
• Block staging or dev environments — Prevent a pre-production mirror from being indexed by adding a site-wide Disallow on that host.
• Control AI training crawlers — Disallow bots such as GPTBot, ClaudeBot, CCBot, Google-Extended or PerplexityBot if you do not want content used for model training.
• Reference your sitemap — Advertise sitemap.xml so crawlers discover new URLs quickly even when your internal linking is sparse.
• Throttle aggressive bots — Use Crawl-delay (honored by Bing and Yandex, ignored by Google) to slow non-critical crawlers on a resource-constrained server.
• Keep duplicate URLs out — Block tracking-parameter URLs or printer-friendly duplicates that would otherwise dilute your crawl-to-index ratio.