Marketing Index

AI crawler

An AI crawler is a bot an AI company uses to visit and read websites. The content is used either to train language models or to fetch answers when a user asks a question.

Examples of AI crawlers

  • GPTBot and OAI-SearchBot from OpenAI, the company behind ChatGPT

  • ClaudeBot from Anthropic, the company behind Claude

  • PerplexityBot from Perplexity

  • Google-Extended, which is not a separate crawler but a setting that controls whether Google may use your content for its AI models

How do they differ from Googlebot?

Googlebot indexes your website for Google's search results. AI crawlers collect content for AI services. Some are used for training, others for searching in real time when a user asks.

Should you block AI crawlers?

You decide that in your robots.txt. If you block them, AI services may find it harder to read and cite your pages. If you want to be visible in AI answers, you should generally allow the crawlers used for search. Also check that your website does not block them by mistake, for example through a firewall.

Can AI services read your website?

Searchmind checks technical access for AI crawlers as part of the work with AI visibility. Read more about GEO and AEO at Searchmind.