Skip to content

Robots.txt for AI crawlers: which bots to allow, which to block, and how

By the Growli teamUpdated 8 min read

Short answer

AI companies use separate user agents for training, search indexing and user-requested fetches. In robots.txt you can allow the search bots (such as OAI-SearchBot, Claude-SearchBot and PerplexityBot) so you can be cited, while disallowing training bots (such as GPTBot, ClaudeBot and Google-Extended). Firewalls and CDNs can still block bots your robots.txt allows.

Step by step

  1. Step 1: Open your robots.txt

    Go to yourdomain.com/robots.txt in your browser to see the rules you have today. No file means every bot is allowed.

  2. Step 2: Decide which AI bots to let in

    Allow the search bots that can cite you: OAI-SearchBot, Claude-SearchBot and PerplexityBot. Block the training bots only if you do not want your content used for training: GPTBot, ClaudeBot and Google-Extended.

  3. Step 3: Add the rules to robots.txt

    For example "User-agent: OAI-SearchBot" on one line and "Allow: /" on the next. Edit the file in your SEO plugin, your shop's theme settings or your hosting file manager.

  4. Step 4: Check your firewall or CDN

    Bot protection can block AI crawlers even when robots.txt allows them. On Cloudflare, check your AI bot settings in the dashboard.

    Cloudflare AI bot settings

  5. Step 5: Test what AI crawlers see

    A free Growli audit reads your robots.txt like each AI crawler and also detects firewall blocks.

    Run a free audit

Why AI crawlers need their own robots.txt rules

Robots.txt is a plain text file at the root of your domain (for example example.com/robots.txt) that tells automated visitors which paths they may fetch. Each set of rules starts with a User-agent line naming the bot it applies to. Well-behaved crawlers read this file before they crawl.

Most AI companies now run more than one bot, and each bot does a different job. One may collect pages for training models, another builds a search index so the assistant can cite you, and a third fetches a single page because a user asked about it. Blocking the wrong one can quietly remove you from AI answers while doing nothing about training.

If you are new to how assistants pick sources at all, start with what AI SEO is. This article is about the access layer: who gets in.

Training, search and user-triggered fetches: the three jobs

It helps to sort every AI bot into one of three groups before you write a single rule.

  • Training crawlers collect public pages that may be used to train future models. Blocking them is how you opt out of training. It does not, on its own, remove you from AI search answers.
  • Search crawlers build an index the assistant uses to find and cite pages when it answers. Blocking them makes it less likely your pages are shown or linked in that assistant's answers.
  • User-triggered fetchers visit a specific URL because a person asked the assistant to read it or a question needed it. They do not crawl the web on their own, and some vendors say robots.txt may not apply to them.

Some companies also publish control tokens. These are names you can use in robots.txt that do not belong to a crawler at all. They tell the company how it may use content its existing crawlers already fetched.

Which AI user agents exist and what each one does

The table below is based on each company's own documentation as of October 2026. Vendors add and rename bots fairly often, so check the linked pages before you rely on a name.

  • Company
    OpenAI
    User agent
    GPTBot
    Job
    Training
    Notes
    Disallowing it signals your content should not be used to train OpenAI's foundation models.
  • Company
    OpenAI
    User agent
    OAI-SearchBot
    Job
    Search
    Notes
    Used to surface sites in ChatGPT's search features. Set independently of GPTBot.
  • Company
    OpenAI
    User agent
    ChatGPT-User
    Job
    User-triggered
    Notes
    Visits pages when a user asks. OpenAI says robots.txt rules may not apply.
  • Company
    Anthropic
    User agent
    ClaudeBot
    Job
    Training
    Notes
    Collects content that may be used to train Claude models.
  • Company
    Anthropic
    User agent
    Claude-SearchBot
    Job
    Search
    Notes
    Indexes content to improve Claude's search results.
  • Company
    Anthropic
    User agent
    Claude-User
    Job
    User-triggered
    Notes
    Fetches pages when Claude users ask questions that need web content.
  • Company
    Perplexity
    User agent
    PerplexityBot
    Job
    Search
    Notes
    Surfaces and links sites in Perplexity results. Perplexity says it is not used for training foundation models.
  • Company
    Perplexity
    User agent
    Perplexity-User
    Job
    User-triggered
    Notes
    Fetches pages for a user's question. Perplexity says it generally ignores robots.txt.
  • Company
    Google
    User agent
    Google-Extended
    Job
    Control token
    Notes
    Not a crawler. Controls use of content for Gemini training and grounding. No effect on Google Search.
  • Company
    Apple
    User agent
    Applebot
    Job
    Search
    Notes
    Powers search features in Spotlight, Siri and Safari.
  • Company
    Apple
    User agent
    Applebot-Extended
    Job
    Control token
    Notes
    Does not crawl. Disallowing it opts out of training Apple's generative models.
  • Company
    Meta
    User agent
    meta-externalagent
    Job
    Training and indexing
    Notes
    Used to train AI models or improve products by indexing content.
  • Company
    Meta
    User agent
    meta-externalfetcher
    Job
    User-triggered
    Notes
    Fetches links on request. Meta says it may bypass robots.txt.
  • Company
    Common Crawl
    User agent
    CCBot
    Job
    Open archive
    Notes
    Builds a public web archive that many organizations, including AI developers, use.
  • Company
    Microsoft
    User agent
    Bingbot
    Job
    Search
    Notes
    Bing's crawler. Blocking it removes you from Bing.

Sources: OpenAI's crawler overview, Anthropic's crawler help article, Perplexity's bot documentation, Google's list of common crawlers and Apple's Applebot support page. Meta lists its crawlers, including a newer Meta-WebIndexer for Meta AI search, in its developer documentation for webmasters.

Google-Extended and Applebot-Extended are not crawlers

Google-Extended has no user agent string of its own. Googlebot does the fetching, and the Google-Extended token only decides whether that content may be used for Gemini model training and grounding. Google states it does not affect inclusion or ranking in Google Search.

That also means Google-Extended does not keep you out of AI Overviews, because those are part of Google Search and follow your Googlebot rules. Applebot-Extended works the same way for Apple: pages that disallow it can still appear in Apple's search features.

Where Bing fits

Bingbot is Bing's search crawler, and Microsoft's Copilot draws on Bing's web index for its web answers. Microsoft does not publish a separate AI training user agent for robots.txt, so for most sites the right move is simply to keep Bingbot allowed. Our guide to Bing Webmaster Tools and IndexNow covers how to get pages into Bing faster.

How robots.txt groups work (the rule that trips people up)

A crawler follows only the most specific group that matches its name. If you have a group for User-agent: GPTBot, GPTBot ignores everything in the catch-all group for all bots. Rules do not stack across groups.

So if your general group blocks /admin/ or /cart/, and you then add a named group for an AI bot, repeat those private paths in the named group too. Otherwise you may open up paths you meant to keep closed. Google explains the matching rules in its Search Central guide to creating a robots.txt file.

User agent names are matched without regard to case, but the path in Allow and Disallow is case-sensitive. /Blog/ and /blog/ are different paths.

Copy-paste robots.txt examples

Replace example.com with your domain and keep any existing rules your platform needs. These examples assume you want to be found by people, so search engines stay fully allowed.

robots.txt: allow everything
# Allow every crawler, including AI search and training bots
User-agent: *
Disallow: /admin/

Sitemap: https://www.example.com/sitemap.xml

The most common choice for businesses that want AI visibility without contributing to training is to block the training bots and tokens by name, and leave search and user fetchers allowed.

robots.txt: allow AI search, block training
# Opt out of AI training, stay visible in AI search
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: meta-externalagent
Disallow: /

User-agent: CCBot
Disallow: /

# Everyone else, including OAI-SearchBot, Claude-SearchBot,
# PerplexityBot, Googlebot and Bingbot
User-agent: *
Disallow: /admin/

Sitemap: https://www.example.com/sitemap.xml

If you want the opposite, blocking a single private area from AI search bots while leaving the rest open, give each bot its own group and repeat your normal private paths.

robots.txt: keep a members area out of AI search
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Disallow: /admin/
Disallow: /members/

User-agent: *
Disallow: /admin/

Several User-agent lines can share one group, as in the last example. Remember that robots.txt is public and voluntary. It is not access control, so anything truly private needs a login.

Allowing search is not the same as allowing training

OpenAI states that GPTBot and OAI-SearchBot settings are independent, and Anthropic and Perplexity also document separate bots for search and for user requests. That is why the split example above works: you can be eligible for citations in ChatGPT, Claude and Perplexity without offering your content for model training.

There are two limits to keep in mind. Blocking a training bot today affects future crawling, not data already collected. And user-triggered fetchers sit outside this split: OpenAI says robots.txt may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores it, because a person asked for the page.

Being allowed is only the entry ticket. Whether an assistant actually recommends you depends on your content, structured data and reputation, which we cover in how to get recommended by ChatGPT.

Check that your firewall or CDN is not overruling robots.txt

Robots.txt is a request, not a gate. Your CDN, firewall or hosting provider sits in front of it and can block a bot outright, even when robots.txt says the bot is welcome. The bot then gets an error page or a challenge and never sees your content.

Cloudflare is the most common example. It offers settings that block AI bots by category (training, agent and search), and it has been changing its defaults for new domains, so a site can end up blocking AI crawlers without the owner choosing to. See Cloudflare's AI bot blocking documentation and check your own dashboard. Security plugins and bot protection on other hosts can do the same.

  1. Open yourdomain.com/robots.txt in a browser and confirm it loads with the rules you expect.
  2. Look at your CDN or firewall settings for AI bot, bot fight or bot protection options.
  3. Check your server or CDN logs for requests from the bots you allowed and the status codes they received.
  4. Re-test after any change to your CDN, theme or security plugin.

Anthropic also notes that blocking its bots by IP address may not work reliably, so robots.txt is the supported way to opt out. A free tool can help with the checking: when you run a free Growli audit, it reads your robots.txt the way AI crawlers do and also flags AI crawler requests blocked by a firewall, with step-by-step fixes for WordPress, Shopify, Wix and other platforms.

Frequently asked questions

Does blocking GPTBot remove my site from ChatGPT?

Not by itself. GPTBot is OpenAI's training crawler, while OAI-SearchBot handles ChatGPT's search features. If you want to appear in ChatGPT search answers, keep OAI-SearchBot allowed even if you block GPTBot.

Does Google-Extended stop AI Overviews from using my content?

No. Google says Google-Extended does not affect Google Search, and AI Overviews are a Google Search feature controlled by your Googlebot rules. Google-Extended only governs use of your content for Gemini training and grounding.

Should a small business block AI crawlers?

Most businesses that want customers to find them should allow AI search bots, because those bots are how assistants find and cite pages. Blocking training bots is a separate choice about how your content is reused, and it does not affect search visibility on the platforms that split the two.

Is there an llms.txt rule I need for AI bots?

No. llms.txt is a separate, optional file that points language models to your most useful pages, and it does not grant or deny access. Access is still controlled by robots.txt and your server. Our llms.txt guide explains when it is worth adding.

Keep reading