Robots.txt File: What It Can and Can't Do About AI Crawlers

A robots.txt file can stop well-behaved AI crawlers from fetching your pages, but it can't control what happens to content they've already collected, and it won't keep a page out of Google. That distinction got sharper last week. On October 21, Google updated its documentation on controlling what you share with Google Search so that the robots meta tag, data-nosnippet, X-Robots-Tag and the Search generative AI control sit side by side. The same day, Google's Gary Illyes said in a video that robots.txt "doesn't dictate what a system does with the data after it has already crawled it," as Search Engine Journal reported.

If someone on your team has asked "are we blocking AI?", here is what each tool does and what we'd recommend.

What does a robots.txt file actually control?

It controls which URLs a named crawler may request, and nothing more. The file sits at the root of your domain and holds groups of rules: a User-agent line naming a crawler, then Allow and Disallow lines listing paths. Google's robots.txt specification supports four fields (user-agent, allow, disallow and sitemap) and ignores anything else.

Three limits matter:

• It's a request, not a lock. Google's introduction to robots.txt says crawlers decide whether to obey it. Reputable ones do.
• It doesn't hide pages from search. A disallowed URL can still appear in Google if other sites link to it.
• It's per host. A robots.txt file on www.example.com doesn't cover shop.example.com.

Can a robots.txt file stop AI training?

Only indirectly, by blocking the crawler that collects training data by its own name. Once a page has been fetched, a robots.txt file has no say over how the data is used.

So a line such as "No AI training" does nothing: it isn't a supported field, and Google's parser skips it. An IETF proposal would add a standard way to state usage preferences, but it's still a draft.

What works today is naming tokens. To keep Google from using your content for Gemini training while staying in Search, add two lines: User-agent: Google-Extended, then Disallow: /.

What do Google-Extended and the Search Console AI setting do?

They are two different switches, and they're easy to mix up.

Google-Extended has no crawler of its own. Google's list of common crawlers describes it as a control token. Blocking it tells Google not to use your content to train future Gemini models or to ground answers in Gemini Apps and Vertex AI. Google says it has no effect on whether you appear in Search or how you rank.

The Search generative AI control sits in Search Console under Settings, then Search generative AI. According to Google's help page, excluding your site removes your links and content from AI Overviews, AI Mode and generative AI features in Discover, and you lose the traffic and impressions those features send. Google says it isn't a ranking signal for the rest of Search and doesn't affect AI training.

Blocking Googlebot itself is different again: it takes you out of Google Search, which is almost never what a business wants.

Which AI crawlers should a business site block?

For most service businesses, very little. If your site exists to bring in leads, being cited in AI answers is worth more than keeping service pages away from AI systems. You want your pricing page and case studies read.

A simple rule:

1. Allow search and AI answer crawlers if visibility drives revenue. That covers most law firms, local services and B2B companies.
2. Block training-only tokens such as Google-Extended if your content is the product, such as paid courses or original research.
3. Put anything private behind a login. Your robots.txt file is public, so listing a secret folder in it advertises the folder.

Meta deserves a look this week. Search Engine Journal reports that Meta's revised Commercial Terms, which include a website crawling clause, take effect October 30, and that Meta documents separate crawlers for AI training and for Meta AI search. If you run ads or a Facebook Page, read the terms yourself and decide how your robots.txt file treats each one.

How do you check your robots.txt file today?

Five checks cover it:

1. Open yourdomain.com/robots.txt and read it line by line.
2. Look for User-agent: * followed by Disallow: /. That blocks everything and is often a leftover from a staging site.
3. Open the robots.txt report in Google Search Console to confirm Google can fetch the file.
4. Write your AI decision in one sentence, then add one group per crawler you want to block.
5. On a hosted CMS, check its search settings first. Google notes that platforms such as Wix may manage the file for you.

Frequently asked questions

Does blocking AI crawlers hurt my Google rankings?

Blocking Google-Extended doesn't, according to Google. The Search generative AI control removes you from AI Overviews and AI Mode but isn't a ranking signal elsewhere.

Will a robots.txt file remove my pages from Google?

No. A blocked URL can still be indexed from links on other sites. Use noindex or a password instead.

Do all AI bots obey robots.txt?

No. Compliance is voluntary, and Google warns that user agent strings can be faked.

How quickly do changes take effect?

Google generally caches a robots.txt file for up to 24 hours. The Search Console AI control usually takes a few days.

Does every site need a robots.txt file?

Not always. If the file is missing and the server returns a 404, Google assumes there are no crawl restrictions, which suits many small sites.

Decide once, then write it down

Most robots.txt file problems come from nobody owning it: a leftover staging rule, a copied list of bot names, a block someone added and forgot. Decide what your site allows, record why, and review the file whenever you change platforms. If you'd like it checked alongside your hosting and site setup, our development team can review it with you.