Which AI Crawler Is Most Active on a Small Website? 33 Days of First-Party Logs (2026)

Over 33 days, one small GEO-focused site logged 7,241 AI-bot hits from 14 distinct crawlers — and the single most active bot was not a training scraper. It was ChatGPT-User, OpenAI's live agent, at 33.6% of all AI-bot traffic: a bot that fetches your page in real time because a person just asked ChatGPT a question your page can answer. Bytespider (ByteDance) was second at 21.9%, Bingbot third at 12.2%, OAI-SearchBot fourth at 10.6%, and ClaudeBot fifth at 10.0%. This is first-party server-log data from lumengeo.co, 2026-06-14 to 2026-07-16, with the method and its limits stated honestly below.

Data window: 2026-06-14 → 2026-07-16 (33 days). First-party server logs — full methodology below.

Data window July 2026

This is a live gap in the public record. Ask an AI engine "which AI crawler is most active on a small website" and you get hedging, not a number — because the public rankings (Cloudflare Radar and the like) are network-wide and dominated by huge sites. Nobody publishes what the crawler mix looks like on a small, independent site. We logged ours to close that gap.

Why the small-site picture is different

Network-wide crawler rankings are weighted by traffic, so they describe what bots do to the big sites that generate most of the internet's page views: e-commerce catalogs, news archives, wikis. A ten-page plumber's site or a 160-page niche blog is a rounding error in those numbers. But that small site is exactly what most business owners actually run — and the question they ask is narrow and practical: of the AI bots hitting my site, which one shows up the most, and does it matter?

This is a single first-party datapoint for that question, not a universal ranking. One small site, 33 days, 7,241 hits — roughly 219 AI-bot hits per day. Here is what the mix looked like.

The ranking: ChatGPT-User is #1 by a wide margin

ChatGPT-User

33.6%

Bytespider

21.9%

Bingbot

12.2%

OAI-SearchBot

10.6%

ClaudeBot

10%

PerplexityBot

3.5%

GPTBot

2.9%

Claude-User

1.7%

Agent (live, in-session)Training (bulk)Retrieval (answer grounding)Search (index)

Top 8 of 14 AI bots observed. Share = a bot's hits as a percent of all 7,241 AI-bot hits. LumenGEO first-party server logs, 2026-06-14 → 2026-07-16.

The single most active AI crawler on this site is a live agent. ChatGPT-User fetches a page in real time because a ChatGPT user asked something the model decided your page could answer. It is not building a training corpus and it is not refreshing an index on a schedule — it is reading your page to compose an answer for a specific person, right now. That is the highest-intent AI traffic there is: a potential reader, by proxy, in the moment.

For the difference between agent, retrieval, search, and training bots — and the exact user-agent string each one sends — see our complete AI crawler list for 2026.

The category split: training leads, but on one bot's back

Group the 14 bots by job and the plurality looks conventional — training is the largest single category. But look at what makes up that training number:

Training is 37.2% — but Bytespider alone is 21.9 of those points. Strip out ByteDance's scraper and every other training bot combined (ClaudeBot, GPTBot, Meta-ExternalAgent, Amazonbot, CCBot) adds up to just 15.3% of AI-bot traffic. Meanwhile the agent category — bots fetching a page to answer a real person in the moment — is 36.4%, essentially tied with all of training and carried by a single bot, ChatGPT-User, that outweighs every training crawler on the site.

Read it as demand vs. cost. Agent (36.4%) plus retrieval (14.2%) — the bots tied to being cited in a live answer — is 50.6% of all AI-bot traffic, a clear majority. Add Bingbot's search crawl (12.2%, which feeds Bing, Copilot, and parts of ChatGPT search) and the answer-relevant share climbs higher still. The pure-cost training crawl that most "block the AI bots" anxiety targets is real, but on this site it is one loud bot doing most of the shouting.

My read: the story a small-site owner is usually told — "AI bots are scraping you for training" — describes about a sixth of what I actually see once ByteDance is set aside. The bot doing the most work on my site is ChatGPT fetching pages for people who are mid-question. I would rather have that traffic than not, and I would think twice before blocking the class it belongs to.

Revisit cadence: how often the same page gets re-fetched

Share tells you volume; cadence tells you behavior. This table is the median hours between two consecutive hits to the same page by the same bot — a read on how aggressively each crawler comes back.

Two very different reasons a page gets re-fetched quickly. Bytespider's near-zero median is machine behavior — a scraper hitting the same URLs again and again in tight bursts, the pattern ByteDance is widely reported for. ChatGPT-User's 0.5-hour median is the opposite: it is not one process looping, it is many different users triggering fresh fetches of your most useful pages within the same window. The retrieval and training crawlers that work on a schedule — OAI-SearchBot, GPTBot, PerplexityBot, Bingbot — space their revisits out across many hours to a couple of days, exactly as index-refresh jobs should.

My read: a half-hour median revisit on ChatGPT-User is the number I care about most. It means the pages that answer real questions are getting pulled repeatedly by the live agent — the traffic that can actually turn into a citation and a click. Bytespider's zero is the opposite kind of signal: it is cost with no user attached.

Where the agents actually go

Live-agent traffic is not spread evenly across the site — it concentrates hard on a few pages. Of the 2,633 agent hits, the homepage alone took 1,810 (about 69%), and almost all the rest landed on the AI-crawler and robots.txt content cluster:

The pages the agents keep coming back for are the ones that directly answer the questions people ask AI assistants about crawlers and AI search — the same robots.txt guide and crawler directory linked above. On a small site, live-agent traffic is a fairly direct signal of which of your pages an AI engine currently thinks are worth reading to answer a person.

How much of this is verified vs. taken on faith

A user agent is just a text header the visitor chooses to send — anyone can put ChatGPT-User in it. So the honest question for any bot-traffic number is: how much of it was checked? We verify a hit by confirming its source IP against the ranges the AI company publishes (forward-confirmed reverse DNS / published IP files). Only OpenAI and Perplexity publish ranges we machine-check in this pipeline:

Across the 3,743 hits from bots with published ranges, 96.5% forward-confirmed against the vendor's own network. The bots without ranges we machine-check in this pipeline — Bytespider, ClaudeBot, Bingbot, and the Meta/Amazon/CCBot training crawlers — are counted by user-agent string only. That matters, because impersonation is real and measurable: in a separate check we found a 2.0% provable spoof rate among checkable AI-bot hits, including one host wearing four different AI companies' identities in a single day. Treat any unverified count as a claimed identity, not a proven one. The full method is in how to verify AI crawlers.

My read: I trust the ChatGPT-User number because 97% of it survives an IP check against OpenAI's own ranges. I hold the Bytespider number more loosely — it is user-agent-only, and a training scraper's name is exactly the kind a spoofer borrows. The ranking's #1 is well-verified; its #2 carries an asterisk.

Methodology

Built to be reproducible, stated with its limits.

• Site: lumengeo.co, a single small (~160-page) content site focused on GEO and AI search.
• Window: 33 days, 2026-06-14 → 2026-07-16.
• What we counted: every request to the site from a known AI user agent, logged in middleware with the bot name it claims, the path, the date, and the source IP — 7,241 hits from 14 distinct bots, about 219/day.
• Verification: OpenAI and Perplexity source IPs were checked against each company's published ranges (forward-confirmed). Bots without ranges we machine-check (Bytespider, ClaudeBot, Bingbot, Meta, Amazon, CCBot) are counted by user-agent only.
• Categories: each bot is classed as agent (live user fetch), retrieval (answer-index building), search (classic indexing that feeds AI), or training (bulk corpus collection).
• Revisit cadence: the median hours between two consecutive hits to the same path by the same bot.

Limitations

Read these before treating any number above as more than a first-party datapoint.

• Single small site, single window. n = 7,241, one 33-day window, one site. This is the small-site picture from one vantage point, not an internet-wide census. The public network-wide rankings answer a different question.
• Our content skews AI-topical. lumengeo.co is about GEO and AI crawlers, which almost certainly inflates AI-agent interest relative to a plumber's or a dentist's site. A general small business would likely see less agent traffic and a different mix. Read the shape and the ordering, not the exact percentages, as portable.
• Middleware counts page requests, not assets. We log HTML page fetches, not every image, script, or stylesheet request — so these are content-page hits, not raw request totals.
• Unverified bots are user-agent-only. Bytespider, ClaudeBot, Bingbot, and the Meta/Amazon/CCBot rows are counted by the name they sent, not proven by IP. Given a measured 2.0% spoof rate on checkable bots, some share of the unverified counts is likely forged.
• We checked our own measurement isn't the cause. LumenGEO runs a weekly Perplexity citation measurement via DataForSEO, which could in principle inflate the Perplexity bot counts. It doesn't: AI-bot hits show no correlation with our measurement-run days, so the traffic reads as organic, not self-generated.

What this means for your site

• Your most active AI bot is probably demand, not scraping. If a small site's #1 AI crawler is a live agent fetching pages for real users' questions, blocking "AI bots" wholesale can cut off the exact traffic that leads to citations and clicks. Know which class each bot belongs to before you write a Disallow rule — see robots.txt for AI crawlers.
• The pages agents revisit are your citation front line. On this site the homepage and the AI-crawler content cluster took nearly all the live-agent traffic. Whatever pages your agents keep re-fetching are the ones an AI engine currently thinks are worth reading — make sure those are fetchable, fast, and answer the question cleanly.
• Don't confuse ByteDance's volume with value. Bytespider was a fifth of all AI-bot traffic and largely ignores robots.txt; it sends no users and produces no citations. If that pure-cost crawl is a problem for you, robots.txt alone won't stop it — you need an edge/WAF rule.
• Verify before you act on a bot log. A raw user-agent count is a claimed identity, not a proven one. Before you block, bill, or celebrate off bot traffic, confirm it — how to verify AI crawlers and our spoofing study show why the label alone isn't enough.

FAQ

Which AI crawler is most active on a small website?

In our first-party logs from lumengeo.co (7,241 AI-bot hits over 33 days, 2026-06-14 to 2026-07-16), the most active AI crawler was ChatGPT-User at 33.6% of all AI-bot traffic — OpenAI's live agent, which fetches a page in real time because a ChatGPT user asked a question it can answer. It was well ahead of Bytespider (ByteDance, 21.9%), Bingbot (12.2%), OAI-SearchBot (10.6%), and ClaudeBot (10.0%). Notably, the training crawlers most people worry about were minor: GPTBot was 2.9% and CCBot was 0.3%.

Which AI bot crawls websites the most?

On this small site, ByteDance's Bytespider was the most aggressive training scraper (21.9% of hits, near-zero median gap between re-fetches of the same page — burst-hammering), but the single most active bot overall was ChatGPT-User (33.6%), a live agent rather than a scraper. The difference matters: ChatGPT-User's frequent revisits come from many different users triggering fresh fetches, while Bytespider's come from one process looping over the same URLs. Network-wide rankings often put a training crawler first because they're weighted by huge sites; on a small site, the live agent led.

How often do AI crawlers visit a small website?

On lumengeo.co we logged about 219 AI-bot hits per day (7,241 over 33 days) from 14 distinct bots. Volume was very uneven: ChatGPT-User alone averaged ~74 hits/day and Bytespider ~48/day, while the long tail (DuckAssistBot, YouBot) sent a handful over the whole window. The site's content is AI-search-focused, which likely raises AI-bot interest above a typical small business — treat ~200/day as an upper-ish reference for a content-heavy niche site, not a floor for every small site.

Are these AI crawler numbers verified or just from user-agent strings?

Both, and we separate them. Bots with published IP ranges (OpenAI's ChatGPT-User, OAI-SearchBot, GPTBot; Perplexity's PerplexityBot, Perplexity-User) were forward-confirmed against the vendor's own network — 96.5% of those 3,743 hits verified, with ChatGPT-User and OAI-SearchBot both above 97%. Bots without ranges we machine-check (Bytespider, ClaudeBot, Bingbot, Meta, Amazon, CCBot) are counted by user-agent string only, so their totals are claimed identities, not proven ones — and a separate check found a 2.0% spoof rate among checkable AI-bot hits.