Does blocking AI crawlers hurt my Google ranking?

Blocking Google-Extended does not. Google states it "does not impact a site's inclusion in Google Search nor is it used as a ranking signal". Blocking Googlebot itself would remove you from Search entirely, which is a different and much worse decision.

Can robots.txt actually stop my content being used for training?

Only for crawlers that choose to obey it. robots.txt is a request, not a lock — the major AI companies publish and honour their tokens, but a badly behaved scraper simply ignores the file.

Why do some AI bots ignore robots.txt?

Because a person asked for that specific page. Perplexity documents that Perplexity-User generally ignores robots.txt, and OpenAI says robots.txt rules may not apply to ChatGPT-User, since both are user-initiated fetches rather than crawling.

Should I block Common Crawl's CCBot?

That depends on how strongly you feel about training. Common Crawl publishes an open corpus that many model builders draw on, so blocking CCBot is the broadest single opt-out available — and it also removes you from academic and search research datasets.

Do I need llms.txt as well as robots.txt?

They do different jobs. robots.txt says who may crawl what; llms.txt is a short curated brief telling assistants which pages actually matter. Publishing both takes about fifteen minutes and they never conflict.

robots.txt for AI crawlers: allow search, block training

On this page

"AI is scraping my site" is really two separate questions wearing one coat. Do you want AI assistants to find and cite you? Almost certainly yes — that is where a growing share of your visitors comes from. Do you want your writing folded into the next model's training data? That is a business decision, and it is yours to make.

The good news is that the big AI companies use different bot names for those two jobs, so you can say yes to one and no to the other.

Three kinds of AI crawler

Bot token	Company	What it is for	If you block it
`GPTBot`	OpenAI	training foundation models	out of training data
`OAI-SearchBot`	OpenAI	ChatGPT search results	you disappear from ChatGPT search answers
`ChatGPT-User`	OpenAI	a user asked ChatGPT to open your page	robots rules may not apply
`ClaudeBot`	Anthropic	training	out of training data
`Claude-SearchBot`	Anthropic	search result quality	less visible in Claude search
`Claude-User`	Anthropic	a user asked Claude to open your page	reduces what it can quote
`Google-Extended`	Google	Gemini training and grounding	no effect on Google Search
`PerplexityBot`	Perplexity	Perplexity search results	you disappear from Perplexity
`Perplexity-User`	Perplexity	a user asked for that page	generally ignores robots.txt
`CCBot`	Common Crawl	an open crawl corpus	out of the broadest public dataset

Sources, in the vendors' own words: OpenAI, Anthropic, Google, Perplexity, Common Crawl.

Two facts worth pinning up. Google is explicit that Google-Extended "does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search" — blocking it costs you nothing in Search or AI Overviews. And the user-initiated agents are a genuine exception: when a person pastes your URL into a chat, that fetch is not crawling, and the rules may not apply.

The starter pack

Add this to the robots.txt at the root of your site. It keeps every AI search bot in and takes the training bots out.

text

# Search and citation bots — welcome
User-agent: Googlebot
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /

# Training crawlers — no thanks
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: CCBot
Disallow: /

Sitemap: https://[your-site]/sitemap.xml

Want the opposite? Delete the second block. Being in training data is how a model learns your brand exists between search calls — plenty of businesses take that trade deliberately.

Now pair it with the file that does the inviting. Save this as llms.txt next to robots.txt:

markdown

# [Business name]

> [One sentence: what you do, for whom, where.]

- Opening hours: [Mon-Fri 9-18]
- Service area: [city or delivery radius]
- Contact: [phone or email]

## Start here
- [Services and prices](https://[your-site]/services): what we offer, with prices
- [Book](https://[your-site]/book): online booking
- [FAQ](https://[your-site]/faq): shipping, returns, cancellations

Why both? Because robots.txt only ever says no. The curated brief is what says start here — why your website needs an llms.txt covers the format, and you can check yours with the llms.txt checker. Ours is at /llms.txt if you want a finished example.

Or have your assistant write both

text

Ask your AI assistant:
Read my website at https://[your-site]. Write me a robots.txt that allows search and
citation crawlers (Googlebot, OAI-SearchBot, Claude-SearchBot, PerplexityBot) and
disallows training crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot), plus my
sitemap line. Then write a matching llms.txt listing my five most useful pages with a
one-line note each, using only facts from the site. Mark guesses with [CHECK].

Upload both files, then open https://your-site.com/robots.txt in a browser to confirm it is really being served — a file sitting in the wrong folder is the most common mistake here.

What this does and does not buy you

It does not stop a determined scraper: robots.txt is a published request that well-behaved crawlers honour, nothing more. It does not change your rankings. What it does is make your position explicit and machine-readable, so the companies that do respect it have no excuse.

And it is only the reading half. Being findable is one thing; being usable is the next one — WebMCP lets an agent book, check and update through your pages instead of only reading them, and the page tools guide is the short version. If you would rather skip the file editing entirely, build an assistant from your URL and it will read your site the way an AI does — the knowledge guide explains what it keeps, and pricing is metered per business.

Questions

Does blocking AI crawlers hurt my Google ranking?: Blocking Google-Extended does not. Google states it "does not impact a site's inclusion in Google Search nor is it used as a ranking signal". Blocking Googlebot itself would remove you from Search entirely, which is a different and much worse decision.
Can robots.txt actually stop my content being used for training?: Only for crawlers that choose to obey it. robots.txt is a request, not a lock — the major AI companies publish and honour their tokens, but a badly behaved scraper simply ignores the file.
Why do some AI bots ignore robots.txt?: Because a person asked for that specific page. Perplexity documents that Perplexity-User generally ignores robots.txt, and OpenAI says robots.txt rules may not apply to ChatGPT-User, since both are user-initiated fetches rather than crawling.
Should I block Common Crawl's CCBot?: That depends on how strongly you feel about training. Common Crawl publishes an open corpus that many model builders draw on, so blocking CCBot is the broadest single opt-out available — and it also removes you from academic and search research datasets.
Do I need llms.txt as well as robots.txt?: They do different jobs. robots.txt says who may crawl what; llms.txt is a short curated brief telling assistants which pages actually matter. Publishing both takes about fifteen minutes and they never conflict.