AI crawler

An AI crawler is an automated bot that fetches web pages on behalf of an AI company, either to train models or to answer live user questions.

OpenAI documented GPTBot in August 2023. Others include ClaudeBot from Anthropic, PerplexityBot, and Google-Extended, which is a robots.txt token rather than a separate crawler. Several companies also run distinct user-triggered fetchers, such as ChatGPT-User, that visit a page only when someone asks about it.

Training crawlers and live fetchers matter differently. Blocking a training bot keeps your text out of future models; blocking a live fetcher can stop an assistant from reading and citing your page today.

Server logs show which bots arrive and what they request, which is the most reliable way to know whether answer engines can see your content at all.

Related terms

Go further