An AI crawler is a bot an AI company uses to visit and read websites. The content is used either to train language models or to fetch answers when a user asks a question.
Examples of AI crawlers
GPTBot and OAI-SearchBot from OpenAI, the company behind ChatGPT
ClaudeBot from Anthropic, the company behind Claude
PerplexityBot from Perplexity
Google-Extended, which is not a separate crawler but a setting that controls whether Google may use your content for its AI models
How do they differ from Googlebot?
Googlebot indexes your website for Google's search results. AI crawlers collect content for AI services. Some are used for training, others for searching in real time when a user asks.
Should you block AI crawlers?
You decide that in your robots.txt. If you block them, AI services may find it harder to read and cite your pages. If you want to be visible in AI answers, you should generally allow the crawlers used for search. Also check that your website does not block them by mistake, for example through a firewall.
Can AI services read your website?
Searchmind checks technical access for AI crawlers as part of the work with AI visibility. Read more about GEO and AEO at Searchmind.
Write to us
Want to know whether AI services can read your website? Fill out the form and we will get back to you.