AI Crawler List 2026: Every AI Bot, User Agent, and How to Verify Them

Over fourteen days, one GEO-focused site logged 3,392 fetches from 13 distinct AI crawlers — and the single largest slice was not training. Live, in-session agent bots (ChatGPT-User, Perplexity-User, Claude-User) were 44.3% of all hits, and retrieval bots that ground AI answers added another 15.1%, so the citation-relevant majority — bots fetching a page to answer a real person right now — was 59.4%. Training crawls were 29.6%. This page is the directory those numbers come from: every AI crawler we saw, its company, its purpose, the exact user-agent string it sent, whether it obeys robots.txt, and how to verify it is genuine. It is built on first-party server logs, not a vendor panel.

Most "AI crawler list" articles are copied from each other and rarely say which bots actually show up, how often, or which claims of identity hold up under verification. This one is different in two ways. The directory below is checked against each company's official documentation. The hit counts are first-party: every request to lumengeo.co from a known AI user agent over a two-week window, classified by purpose and cross-checked against published bot IP ranges. Where the official record is contested — Perplexity's robots.txt behavior is the clearest case — we say so plainly instead of repeating a vendor claim.

Last updated: June 2026

Data window June 2026

This is the companion reference to our most-fetched page, robots.txt for AI crawlers. That guide tells you what to allow and block; this one tells you exactly who is knocking.

The four kinds of AI crawler

The phrase "AI bot" collapses four very different jobs into one word. Get the categories right and every robots.txt and analytics decision downstream gets easier. Get them wrong and you will block the traffic that earns citations while leaving the traffic you were worried about untouched.

• Agent (live, in-session). Fetched in real time because a person asked an assistant something that requires reading your page. ChatGPT-User, Perplexity-User, and Claude-User are agent bots. This is the highest-intent AI traffic there is — a potential reader, by proxy, right now.
• Retrieval (answer grounding). Fetched to build or refresh the index an AI search engine draws from when it composes a cited answer. OAI-SearchBot, PerplexityBot, Claude-SearchBot, and DuckAssistBot are retrieval bots. This is the machinery that decides whether you can be cited at all.
• Search (index). Traditional search crawling that also feeds AI answers. Bingbot indexes for Bing and, in doing so, supplies Microsoft Copilot and parts of ChatGPT search.
• Training (bulk). Wide crawling that collects content into model-training corpora. GPTBot, ClaudeBot, CCBot, Bytespider, Meta-ExternalAgent, and Amazonbot are training bots. These send no users and produce no citations; the visit is pure cost.

The reason the split matters: agent and retrieval bots are the ones tied to being cited in live answers. Training bots are not. Blocking one class has nothing to do with the other.

The master AI crawler list

Every AI crawler worth knowing in 2026, grouped by company. The Bot column shows the robots.txt user-agent token verbatim — the token you write a Disallow rule against is the name in this column. Google-Extended and Applebot-Extended are the two exceptions: they are robots.txt tokens only, not crawlers, and make no HTTP requests. Rows marked "Yes" under "In our logs" are bots we actually observed; the rest are documented by their vendors but did not appear in our window.

A few notes the table can't carry. Perplexity documents that PerplexityBot obeys robots.txt, but in August 2025 Cloudflare reported that Perplexity was using stealth, undeclared crawlers to reach content that site owners had disallowed; Perplexity has acknowledged that a user-supplied URL may be fetched even when robots.txt would block it. Treat its compliance as stated-but-disputed. Bytespider (ByteDance) publishes no vendor documentation page and has repeatedly been observed fetching paths under Disallow: /, so if you actually need it gone, robots.txt alone is not enough — you need an edge or WAF rule. Anthropic historically said it did not publish IP ranges and has since released a verification IP list; reverse DNS remains a fallback. If a bot you want to exclude only honors robots.txt as an honor system, the firewall is the enforcement layer.

User-agent strings reference

These are the exact user-agent strings we observed in our logs, reproduced verbatim. Use them to build log filters and detection rules — but remember a user agent is just a text header, and anyone can send one that says GPTBot. The string is a starting point for identification, never proof. For proof you verify the IP, which is covered in how to verify AI crawlers.

One detail worth flagging: Claude-User now arrives carrying a claude-code/2.1.x token, which means the fetch was triggered by a developer working inside Claude or Claude Code, not a consumer chatting on the web. The agent class is broadening beyond chat windows into coding tools and automations.

Which crawlers actually showed up (and how often)

Here is the part panels can't give you: who actually fetched our pages, in what volume, and how much of it survived verification. The window is 14–28 June 2026 — 3,392 logged AI-bot hits across 146 paths and 25 countries, roughly 225 per day.

By purpose the split was Agent 1,503 (44.3%), Training 1,003 (29.6%), Retrieval 513 (15.1%), and Search 373 (11.0%). Agent and retrieval together — the citation-relevant traffic — came to 2,016 hits, or 59.4% of everything. OpenAI's three crawlers alone (ChatGPT-User, OAI-SearchBot, GPTBot) accounted for 1,966 hits, about 58% of the log. No other company was close.

The most useful behavioral finding is about re-crawling. ChatGPT-User did not visit our reference content once and move on. It re-fetched the robots.txt for AI crawlers article on 13 of roughly 14 days — essentially daily — and hit the homepage 946 times across the window. Explainer and reference pages get pulled again and again as live questions touch them, which is exactly why an evergreen, well-structured reference page is such durable GEO real estate. For how to watch this on your own site, see how to track AI search traffic and our 14-day AI crawler traffic study.

Verification is what separates a trustworthy bot log from a guess. Of the 3,392 hits, 1,987 (58.6%) cryptographically verified against the relevant company's published IP ranges. Sixty-seven (2.0%) actively failed: they claimed a checkable identity but came from an IP outside that company's ranges. The rest are simply unverifiable by us, because we do not run an IP check for every vendor and some legitimate bots don't expose a method we use. Read that honestly: the 67 failures are the provable floor of impostor traffic, not a ceiling, and "unverifiable" is not a synonym for "fake." The highest fail rate among checkable bots was GPTBot at 10.8% (17 of 158) — a reminder that the most-spoofed identities are the most famous ones. We cover the impostor problem in fake AI bot traffic.

Should you block them?

There is no single "block AI bots" decision. Each class carries a different consequence, so decide per class.

• Training bots (GPTBot, ClaudeBot, CCBot, Bytespider, Meta-ExternalAgent, Amazonbot). Blocking these is a pure content-rights choice with zero citation cost. If you do not want your content in model-training corpora, disallow the training tokens and add the two opt-out tokens, Google-Extended and Applebot-Extended, which cover Gemini and Apple Intelligence training without touching your search visibility. The catch: Bytespider and, by some reports, Meta's crawlers do not reliably honor robots.txt, so enforcing a block on those requires an edge or WAF rule, not just a Disallow line.
• Retrieval bots (OAI-SearchBot, PerplexityBot, Claude-SearchBot, DuckAssistBot). These build the indexes AI engines cite from. Block them and you remove yourself from that engine's cited answers. This is almost never what a brand actually wants, and it is the most common accidental self-inflicted wound in AI search.
• Agent bots (ChatGPT-User, Perplexity-User, Claude-User, Meta-ExternalFetcher). Blocking these means refusing to load a page for a person who explicitly asked an assistant to read it. It is the highest-intent traffic on the list. Block it only with a very specific reason.
• Search (Bingbot). Blocking Bingbot removes you from Bing — and, because Bing's index feeds Microsoft Copilot and parts of ChatGPT search, from those AI surfaces too. Block only if you genuinely want out of Bing entirely.

The principle to carry away: blocking training does not block citations, and blocking agent or retrieval bots removes you from live AI answers. They are different levers with opposite effects. For the exact copy-paste configurations, the CDN edge-block gotcha, and how to confirm your rules work, use robots.txt for AI crawlers.

FAQ

What is the difference between ChatGPT-User and GPTBot?

They are two different OpenAI crawlers with opposite purposes. GPTBot is a training crawler that bulk-collects content for model training, and you can opt out of it without affecting anything else. ChatGPT-User is an agent crawler that fetches a specific page in real time because a person asked ChatGPT to read it. Blocking GPTBot keeps you out of training data but has no effect on citations; blocking ChatGPT-User refuses to load pages for live users. OpenAI's third bot, OAI-SearchBot, is separate again and handles search retrieval for ChatGPT.

How do I identify AI crawlers in my server logs?

Start with the user-agent string — most legitimate AI bots self-identify with a token like GPTBot, ClaudeBot, PerplexityBot, or ChatGPT-User and a URL pointing to the operator's documentation. But a user agent is just a header and is trivially spoofed, so the name alone is not proof. For confirmation, check the source IP against the company's published range file (for example openai.com/gptbot.json or perplexity.ai/perplexitybot.json) or run a forward-confirmed reverse DNS lookup for bots that use it, such as Bingbot resolving to search.msn.com. Verified hits you can trust; unverified ones you should treat with caution.

Do AI crawlers respect robots.txt?

Most documented crawlers from OpenAI, Anthropic, Microsoft, Amazon, and Common Crawl state that they honor robots.txt, and in practice they generally do. The notable exceptions are user-initiated agent fetches and a few bad actors: Perplexity has acknowledged that a user-supplied URL may be fetched even when robots.txt would block it, and Bytespider has repeatedly been observed crawling disallowed paths. Because robots.txt is an honor system, anything you truly need to keep out should be enforced at the firewall or CDN edge, not just declared in a text file.

Which AI crawler visits websites most often?

In our 14-day first-party log, ChatGPT-User was by far the most active at 1,434 of 3,392 total hits — more than every non-OpenAI bot combined. It is an agent crawler, fetching pages in real time as ChatGPT users ask questions, and it re-fetched our main reference article on 13 of about 14 days. OpenAI's three crawlers together made up roughly 58% of all AI-bot traffic, which is a strong signal that ChatGPT visibility carries outsized weight in 2026.

Does blocking AI crawlers stop me from being cited?

It depends entirely on which crawlers you block. Blocking training crawlers like GPTBot, ClaudeBot, and CCBot has no effect on citations — they only feed model training. Blocking retrieval and agent crawlers like OAI-SearchBot, PerplexityBot, and ChatGPT-User does stop citations, because those are the bots that build AI answer indexes and fetch your pages for live answers. A blanket "block all AI bots" rule is the common mistake: it removes you from AI search while saving only a little training bandwidth.