Tous les crawlers IA, et ce que coûte leur blocage

Builds the index ChatGPT searches when it answers. Block it and you cannot be cited in ChatGPT search results at all.

The same crawl feeds Google Search, AI Overviews and AI Mode. Still the largest single source of AI answers.

Microsoft Copilot answers are grounded in the Bing index. No Bingbot, no Copilot citations.

Builds the index behind Perplexity answers. Perplexity sends real referral traffic, so this one converts.

Indexes pages so Claude can find and cite them when it searches the web.

Feeds Siri, Spotlight and Apple Intelligence answers across every Apple device.

Feeds Alexa and the Rufus shopping assistant. Matters for retail.

Exa is the retrieval layer behind a long tail of AI products and agents.

Brave Search sells its index to AI products, so one crawl reaches several assistants.

Indexes pages for You.com answers.

Relevant if you care about Huawei devices and Asian markets.

Small but growing generative search engine.

Fetches your page live when a user asks ChatGPT about it. Blocked means ChatGPT tells the user it cannot open your site.

Fetches a page a Perplexity user explicitly pointed at.

Opens your page when a Claude user asks about it.

Fetches pages for Le Chat. Worth allowing if French and EU audiences matter to you.

Powers the AI answers shown above DuckDuckGo results.

Pulls a page a NotebookLM user added as a source.

Opens a link a Meta AI user shared in a conversation.

Browses and acts for a user — filling a form, comparing products, starting a checkout. Blocking it removes you from agent-driven journeys.

Retrieves your page when a Gemini API caller passes its URL as context.

Extraction layer used by thousands of AI apps to read your pages on demand.

Coding agents read docs live. Relevant if developers are your audience.

Collects content to train future OpenAI models. Separate from search: blocking it does not remove you from ChatGPT search.

Collects content to train Anthropic models. Independent from Claude-SearchBot.

A robots.txt control token, not a crawler. Disallowing it keeps your content out of Gemini training and grounding without touching Search.

A control token, not a crawler. Disallow it to stay out of Apple model training while keeping Siri and Spotlight visibility.

Collects content for Meta AI products and Llama training.

Aggressive crawler; widely reported to ignore robots.txt. Block at the edge if you mean it.

Collects training data for DeepSeek models.

Retrieves pages for Cohere model training and grounding.

Collects public content for Baidu ERNIE models.

Builds a commercial knowledge graph resold to AI companies.

One crawl, many models: Common Crawl is the training corpus behind a large share of open-weight LLMs.

Non-profit research crawler feeding open, documented datasets.

Builds a decentralised search index sold on to AI consumers.

Crawls pages into customer-built Bedrock knowledge bases.