Does blocking AI crawlers hurt my Google ranking?

Refusing training crawlers such as GPTBot or Google-Extended has no effect on classic search ranking. Refusing Googlebot or bingbot does: they serve both the web index and the AI answers built on top of it. That is why the “no AI at all” option here leaves the catch-all group open and never touches your web indexing.

Is robots.txt enough to keep a crawler out?

No. It is a published preference. Well-behaved operators honour it; several of the 37 crawlers listed here have never documented that they do, and nothing in the file stops them. If you need enforcement, block by user-agent or network at your CDN — and remember the reverse trap: a CDN can refuse a crawler your robots.txt welcomes.

Do assistants actually read llms.txt?

Some do, not all yet: it is a young convention, picked up first by developer tools and AI agents. It costs a few minutes, and gives any assistant that looks a clean map of your site instead of your navigation menu. It never replaces robots.txt: a crawler you refuse there will not fetch it.

What is tdmrep.json for?

In the EU, anyone may mine publicly available content unless the rights holder has reserved that right in a machine-readable way (article 4 of directive 2019/790). robots.txt is a crawl preference; TDMRep is the W3C community protocol written for that reservation. Publishing both says the same thing to the crawler and to the lawyer.

Do you store my site or my files?

No. This page ships the crawler registry to your browser and builds every file there, the zip included. Nothing is submitted, logged or kept. The scan is the only part of Crawlable that makes requests, and only when you ask it to.

No sign-up

AI configuration generator

One form, every file AI crawlers read: a robots.txt with the 37 crawler tokens that matter in 2026, an llms.txt that points assistants at your best pages, the EU text-and-data-mining opt-out and the identity snippet for your <head>. Download them all in one zip.

Your site

robots.txtllms.txt<head>

Filled in once, reused in every file.

Address

Used for the sitemap line and to complete your links. Nothing is sent anywhere.

Name

Your brand or organisation, the way you want AI answers to write it.

In one sentence

What you do and for whom. It opens your llms.txt and describes you in the JSON-LD.

What AI may do

robots.txt

This decides your robots.txt: who may crawl, and what stays private.

What should AI be allowed to do?

Visible, but not training dataRecommendedIn AI answers and citations; out of the training corpora. What most sites want.Everything allowedAnswers, assistants and model training. Maximum reach.No AI at allRefuse every AI crawler. Classic Google and Bing indexing is left untouched.Crawler by crawlerDecide for each of the 37 crawlers in the registry.

Paths to keep private (one per line, optional)

Repeated in every group, so named crawlers honour them too. Most files state them once, where AI crawlers never look.

Declare my sitemap

Leave empty for /sitemap.xml. WordPress with Yoast: /sitemap_index.xml.

The pages AI should read first

llms.txt

Your llms.txt: a short map an assistant reads before your site. Five to fifteen well-described links beat a hundred.

A section called “Optional” tells assistants they may skip it when short on context. Your sitemap is added there automatically.

What an assistant should know (optional)

Plain text placed after the summary: service area, opening hours, what you do not do.

Rights and identity

tdmrep.json<head>

The EU opt-out from text and data mining, and the name AI answers attach to your site.

Reserve my text and data mining rights (EU)Writes /.well-known/tdmrep.json and a meta tag: the machine-readable opt-out article 4 of the EU copyright directive asks for. robots.txt alone is a crawl preference, not a rights reservation.Ticked automatically when every training crawler is refused in step 2.

Logo (optional)

A full URL or a path on your site. Goes into the JSON-LD.

Official profiles (optional)

One per line: LinkedIn, X, Wikipedia, Wikidata… They tell AI which entity you are.

Every file in its place, with a README saying where each one goes.

/robots.txt

#=======================================================================
#  robots.txt — generated by Crawlable on 2026-09-23
#  Regenerate or verify: https://crawlable.fr/en/config-generator
#  AI crawler registry: version 2026.09.1
#=======================================================================

# --- Crawlable [begin] · generated block ------------------------------
# This block can be regenerated: re-import this file into the generator
# and everything outside the two markers is kept as it is.

# --- Everyone else ----------------------------------------------------
User-agent: *
Allow: /

# --- AI search, assistants and agents: allowed ------------------------
# These are how you show up in ChatGPT, Gemini, Copilot, Perplexity
# and Claude. Blocking one removes you from its answers.
User-agent: Googlebot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: bingbot
User-agent: PerplexityBot
User-agent: Claude-SearchBot
User-agent: Perplexity-User
User-agent: Applebot
User-agent: ChatGPT Agent
User-agent: Claude-User
User-agent: MistralAI-User
User-agent: Amazonbot
User-agent: DuckAssistBot
User-agent: Bravebot
User-agent: ExaSearchBot
User-agent: Google-NotebookLM
User-agent: GoogleAgent-URLContext
User-agent: meta-externalfetcher
User-agent: FirecrawlAgent
User-agent: YouBot
User-agent: Andibot
User-agent: Claude-Code
User-agent: PetalBot
Allow: /

# --- Model training and datasets: refused -----------------------------
# Independent of the blocks above: you stay visible in AI answers,
# your content simply does not end up in a training corpus.
User-agent: AI2Bot
User-agent: Ai2Bot-Dolma
User-agent: Applebot-Extended
User-agent: bedrockbot
User-agent: Bytespider
User-agent: DoubaoBot
User-agent: CCBot
User-agent: ClaudeBot
User-agent: anthropic-ai
User-agent: Claude-Web
User-agent: cohere-ai
User-agent: cohere-training-data-crawler
User-agent: DeepSeekBot
User-agent: Diffbot
User-agent: Diffbot-User
User-agent: ERNIEBot
User-agent: Google-Extended
User-agent: GPTBot
User-agent: meta-externalagent
User-agent: FacebookBot
User-agent: Timpibot
Disallow: /

# Add your sitemap here: Sitemap: https://example.com/sitemap.xml

# --- Crawlable [end] --------------------------------------------------

23 allowed14 refusedNo visibility given up

Where it goes: At the root of your domain: https://example.com/robots.txt. A file in a subfolder is never read. WordPress: Yoast or Rank Math → Tools → File editor. Shopify: Settings → robots.txt. Next.js: app/robots.ts.

These do not document honouring robots.txt, so the file is a request they may ignore: Diffbot, Bytespider, DeepSeekBot, cohere-ai, Timpibot, ERNIEBot. Blocking those needs a rule at your CDN.
No sitemap declared. Add your address above and we add the line: it is the cheapest way to tell a crawler what exists.

Why these files are not the usual ones

One group per crawler, repeated on purpose

A crawler obeys exactly one group: the one naming it. Name OAI-SearchBot and it stops reading your “*” group — including the Disallow you wrote there. That is how a site ends up serving /admin to ChatGPT while believing it is blocked. We repeat your paths in every group.

Answers, training and rights are three choices

Blocking GPTBot does not remove you from ChatGPT: that is OAI-SearchBot. And neither is a legal opt-out from text and data mining — that is TDMRep. Most “block AI” snippets blur all three; we keep them apart, and we never score you for opting out of training.

37 crawlers, four files that agree

The sitemap robots.txt declares is linked from llms.txt; refusing every training crawler reserves your TDM rights too. Written separately, these files drift apart. Built from one form, on a versioned crawler registry, they cannot.

All crawlers →

Questions

Does blocking AI crawlers hurt my Google ranking?: Refusing training crawlers such as GPTBot or Google-Extended has no effect on classic search ranking. Refusing Googlebot or bingbot does: they serve both the web index and the AI answers built on top of it. That is why the “no AI at all” option here leaves the catch-all group open and never touches your web indexing.
Is robots.txt enough to keep a crawler out?: No. It is a published preference. Well-behaved operators honour it; several of the 37 crawlers listed here have never documented that they do, and nothing in the file stops them. If you need enforcement, block by user-agent or network at your CDN — and remember the reverse trap: a CDN can refuse a crawler your robots.txt welcomes.
Do assistants actually read llms.txt?: Some do, not all yet: it is a young convention, picked up first by developer tools and AI agents. It costs a few minutes, and gives any assistant that looks a clean map of your site instead of your navigation menu. It never replaces robots.txt: a crawler you refuse there will not fetch it.
What is tdmrep.json for?: In the EU, anyone may mine publicly available content unless the rights holder has reserved that right in a machine-readable way (article 4 of directive 2019/790). robots.txt is a crawl preference; TDMRep is the W3C community protocol written for that reservation. Publishing both says the same thing to the crawler and to the lawyer.
Do you store my site or my files?: No. This page ships the crawler registry to your browser and builds every file there, the zip included. Nothing is submitted, logged or kept. The scan is the only part of Crawlable that makes requests, and only when you ask it to.

Now check the files actually work

Correct files are half the answer. Your CDN can refuse a crawler before it ever reads robots.txt, a catch-all route can serve your llms.txt as HTML, and a JavaScript-only page can be reachable yet unreadable. A scan sends one real request per crawler and tells you what actually happened.

Free, no account, about 5 seconds.