AEO glossary

AI Crawler

An AI crawler is a bot operated by an AI company to fetch web pages for training, indexing or live answer retrieval.

Technical explanation

Different bots serve different purposes: some only gather training data, others fetch pages at answer time. Blocking a retrieval bot in robots.txt can remove a site from live answers even when its content is excellent.

Example

GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and Google-Extended are checked separately because their rules and roles differ.

How it differs

Blocking Google-Extended affects AI training use, not classic Google indexing by Googlebot.

Related terms

Where Varazm measures this:

Site-wide AEO Audit Methodology

Frequently asked questions

What is AI Crawler?: An AI crawler is a bot operated by an AI company to fetch web pages for training, indexing or live answer retrieval.
How does AI Crawler differ from related terms?: Blocking Google-Extended affects AI training use, not classic Google indexing by Googlebot.
Can you give an example of AI Crawler?: GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and Google-Extended are checked separately because their rules and roles differ.