Robots.txt Generator and Tester, with AI Crawler Presets

Build a robots.txt with presets for Googlebot, GPTBot, ClaudeBot and other AI crawlers, check it for mistakes and test URLs using Google rules.

Runs in your browser. Nothing is uploaded.

SEO

Start from a preset

Group 1

User-agents (one per line)

Add a crawler to this group

Pick one and press Add

Rules (Allow: or Disallow: per line)

Crawl-delay in seconds (optional)

Google ignores Crawl-delay. Some other crawlers read it.

Sitemaps

Sitemap URLs (one per line)

Full URLs. They apply to the whole file, not to one group

Import an existing file

Paste a robots.txt

robots.txt

robots.txt2 lines · 32 bytes

User-agent: *
Disallow: /admin/

robots.txt asks well-behaved crawlers not to fetch URLs. It is not access control, and a blocked URL can still appear in search results if other sites link to it. To keep a page out of results, let it be crawled and use a noindex robots meta tag or header.

Test a URL against these rules

URL or path

Crawler

Enter a URL to see whether the crawler may fetch it, and which rule decides.

Matching follows Google's documented rules: the longest matching path wins, and Allow wins a tie. Other crawlers may differ in edge cases.

Crawler names

User-agent tokens used in the presets

Token	Vendor	Type	What blocking it does
`Googlebot`	Google	Search	Google Search. Blocking it removes the site from crawling by Google Search.
`Googlebot-Image`	Google	Search	Google Images and image features.
`Googlebot-News`	Google	Search	Google News.
`Googlebot-Video`	Google	Search	Video features in Search.
`Google-Extended`	Google	AI training	Controls use of crawled content for Gemini training and grounding. It does not affect Search inclusion or ranking.
`GoogleOther`	Google	Other	A generic Google crawler used by product teams, for example for research.
`Bingbot`	Microsoft	Search	Bing Search.
`GPTBot`	OpenAI	AI training	Content that may be used to train OpenAI models.
`OAI-SearchBot`	OpenAI	AI search	ChatGPT search. Blocking it keeps the site out of ChatGPT search answers.
`ChatGPT-User`	OpenAI	AI, on user request	Fetches pages when a ChatGPT user asks. OpenAI says robots.txt rules may not apply to user-initiated actions.
`ClaudeBot`	Anthropic	AI training	Content that may be used to train Claude models.
`Claude-SearchBot`	Anthropic	AI search	Improves the quality of search results in Claude.
`Claude-User`	Anthropic	AI, on user request	Fetches pages when a Claude user asks a question that needs them.
`PerplexityBot`	Perplexity	AI search	Surfaces and links sites in Perplexity search results; Perplexity says it is not used for training.
`Perplexity-User`	Perplexity	AI, on user request	User-initiated fetches. Perplexity says it generally ignores robots.txt.
`Applebot-Extended`	Apple	AI training	Opts content out of Apple generative AI training. It does not crawl and does not affect Search.
`CCBot`	Common Crawl	AI training	Common Crawl's open web archive, widely used as AI training data.

Names checked against Google, OpenAI, Anthropic, Perplexity and Apple documentation. Vendors add and rename crawlers, so check their pages before relying on a list.

Example

Input

User-agent: *
Disallow: /admin/
Allow: /admin/public/
Test: /admin/public/logo.png

Result

Allowed: Allow: /admin/public/ is the longest matching rule

Building the file

A robots.txt file sits at the root of a host and lists groups of rules. Each group starts with one or more User-agent lines naming a crawler, followed by Allow and Disallow lines whose values are URL paths. Pick a preset (allow everything, block everything, a typical site, opt out of AI training, block all AI crawlers), then edit the groups, add Sitemap URLs, or paste an existing file to load it into the builder. The file updates as you type and can be copied or downloaded as robots.txt.

The crawler list is checked against vendor documentation: Googlebot, Googlebot-Image, Googlebot-News, Googlebot-Video and Google-Extended from Google, GPTBot, OAI-SearchBot and ChatGPT-User from OpenAI, ClaudeBot, Claude-SearchBot and Claude-User from Anthropic, PerplexityBot and Perplexity-User from Perplexity, and Applebot-Extended from Apple. Bingbot and CCBot (Common Crawl) are included as widely used names. Vendors add and rename crawlers, so recheck their pages before relying on the list.

How rules are matched

The tester follows Google's documented behaviour. A crawler uses only the group with the most specific matching user-agent, and falls back to the * group when it has none; names are case-insensitive but paths are case-sensitive. For a URL, the rule with the longest matching path wins, and if an Allow and a Disallow tie, Allow wins. A * in a path matches any run of characters, a trailing $ anchors the end, and a trailing * is ignored. So Allow: /admin/public/ overrides Disallow: /admin/ for files inside it.

The linter flags the mistakes that matter: paths that do not start with / or *, rules before any User-agent line, relative Sitemap URLs, a blanket Disallow: / for all crawlers, files over Google's 500 KiB limit, and rules Google ignores such as Crawl-delay and noindex. Other crawlers can differ in edge cases, so the tester describes Google's behaviour only.

What robots.txt cannot do

robots.txt is a request to well-behaved crawlers, defined as the Robots Exclusion Protocol in RFC 9309. It is not security: anyone can read the file, and listing a private path in it advertises that the path exists. Use authentication for anything private. It also does not remove pages from search. A URL blocked by robots.txt can still be indexed, without a description, if other sites link to it. To keep a page out of results, allow crawling and serve a noindex robots meta tag or X-Robots-Tag header, because a blocked page's noindex is never read.

AI crawler rules are requests too. Blocking GPTBot or ClaudeBot signals that content should not be used for training, but fetchers that act on a user's request, such as ChatGPT-User and Perplexity-User, may not follow robots.txt, and blocking search crawlers such as OAI-SearchBot or Claude-SearchBot can keep your site out of those products' answers. Blocking Google-Extended does not affect Google Search.

Frequently asked questions

Does robots.txt remove a page from Google?

No. It stops crawling, not indexing. A blocked URL can still appear in results without a snippet if other pages link to it. Use a noindex meta tag or header on a crawlable page, or remove the page, to take it out of results.

How do I block AI crawlers?

Add a group for each crawler token, for example User-agent: GPTBot with Disallow: /, or use the AI presets. Compliant training crawlers will honour it. Some user-triggered fetchers may ignore robots.txt, so use server-side rules if you need to enforce a block.

Does Google support Crawl-delay?

No. Google ignores it. To slow Google down, have your server answer with 500, 503 or 429 status codes when it is overloaded. Some other crawlers, such as Anthropic's ClaudeBot, read Crawl-delay.

Where do I put the robots.txt file?

At the top level of each host, for example https://example.com/robots.txt. It applies only to that host, protocol and port, so a subdomain needs its own file.

Which rule wins when Allow and Disallow both match?

The one with the longer path. If they are equally long, Allow wins. The tester shows the deciding rule for any URL.