I Logged Every Ai Crawler For 34 Days. Chatgpt Outreads Googlebot

In mid-July, my Google clicks in my home market (Israel) dropped by almost half. Buyer-intent queries that used to bring steady leads just evaporated from Search Console. While I was staring at GSC dashboards trying to figure out what broke, I finally did the thing I should have done months earlier: I stopped looking at dashboards and started reading raw server logs.

What I found there was a parallel universe. Google Search was sending me less than ever — but AI systems were reading my site constantly. Not "someday this will matter" constantly. Right-now constantly: an AI assistant was fetching a page of mine roughly every 26 minutes, around the clock, because a real human had just asked it a question.

So I built a small log analyzer and let it run. Here's what 34 days of complete Caddy logs from a small business site (about 70 real human visitors a day) actually look like.

The numbers

All counts are HTTP 200 responses only (more on why below), over 34 days:

Three things in that table genuinely surprised me.

ChatGPT-User outreads Googlebot. 40.3 fetches a day versus 35.8. This isn't a crawler building an index for later — ChatGPT-User is the user-agent OpenAI sends when a human is mid-conversation and ChatGPT decides to pull a live page to answer them. On my site, that now happens more often than Googlebot visits. For a tiny business site in a niche market, I did not expect that.

Bing crawls 4.4x harder than Google. 158 requests a day versus 36. Nobody optimizes for Bing. Bing's referral traffic is a rounding error. But Seer Interactive found that 87% of SearchGPT's citations match Bing's top organic results (versus 56% for Google) — Bing's index is a structural input to what ChatGPT can retrieve and cite. Suddenly that 4.4:1 ratio reads very differently: the "irrelevant" search engine is the supply chain for the assistant that's actually fetching my pages 40 times a day.

Live retrieval is the biggest AI bucket that matters. Grouping by meaning rather than by bot name:

???? Live user retrieval (a human asked right now)   1,907  (~56/day)  
???? AI-search indexing (OAI-SearchBot, Perplexity,  
   bingbot, Applebot)                               6,096  (~177/day)  
???? Training crawls (GPTBot, ClaudeBot, Amazonbot)     897  (~26/day)  
???? Classic Google                                   1,233  (~36/day)  

1,907 live retrievals in 34 days. Each one of those is a moment where a real person asked an AI assistant something, and the assistant chose my site as a source. That's the leading indicator for citations — and it's the number I now track weekly.

What they actually read (this is the useful part)

Per-bot top pages turned out to be more interesting than the totals:

• ChatGPT-User loves the homepage (115 fetches), a WhatsApp automation guide (108), and — third place — our WhatsApp bot pricing guide. People are asking ChatGPT what a WhatsApp bot costs, and ChatGPT is pulling my pricing page to answer them. That's a sales conversation I'm part of without knowing it.
• Claude-User is obsessed with exactly one post: a spam-detection write-up got 233 of its 519 total fetches — 45% of everything Claude reads on my site is one article. One strong, specific technical post apparently becomes a go-to source for one assistant.
• GPTBot (the training crawler) spends its time on /signin and /forgot-password. Twelve visits to my login page. Enjoy the training data, I guess.
• Perplexity-User: zero. PerplexityBot indexes me (103 requests), but not a single live user-triggered retrieval in 34 days. Every assistant has its own taste in sources, and Perplexity's taste apparently doesn't include me. Yet.

Methodology, and the trap that almost ruined the data

The analyzer is ~60 lines of Python that runs over Caddy's JSON access logs. The core is just user-agent patterns with one ordering rule — the *-User patterns must be matched before the generic bot patterns, because the loop stops at first match:

BOTS = {  
    "ChatGPT-User":    re.compile(r"ChatGPT-User", re.I),   # live retrieval  
    "Claude-User":     re.compile(r"Claude-User", re.I),    # live retrieval  
    "Perplexity-User": re.compile(r"Perplexity-User", re.I),  
    "OAI-SearchBot":   re.compile(r"OAI-SearchBot", re.I),  
    "GPTBot":          re.compile(r"GPTBot", re.I),          # training  
    "PerplexityBot":   re.compile(r"Perplexity", re.I),  
    # ... bingbot, Googlebot, ClaudeBot, Applebot, Amazonbot  
}  

The trap: count only HTTP 200s. My first version counted every request, and the numbers were inflated with garbage — security scanners spoof OpenAI user-agents (a scanner claiming to be GPTBot while probing for /wp-admin on a site that doesn't run WordPress) and land on 404s. Filtering to 200-only removed the impostors and left actual content fetches. If you run this on your own logs and skip that filter, your "AI traffic" will be part vulnerability scan.

Why your analytics can't see any of this

None of these 8,900 AI-related requests appear in GA4. No JavaScript execution, no consent banner click, no pageview event. And it's worse than that: on my site, GA4 undercounts humans by roughly 9x too — only ~11% of visitors accept the cookie banner (I measured this two independent ways over 11 days: ~780 real browser visitors in the logs and widget-loads, 89 in GA4).

So the fastest-growing reader of my site — AI assistants answering buyer questions in real time — is completely invisible unless you read server logs. If your hosting gives you raw access logs, that's where this story lives. If it doesn't, you can't see it at all, and you might be making content decisions while blind to a quarter of your real readership.

What I'm doing differently now

1. Watching *-User agents weekly, not rankings daily. Live retrievals are the citation pipeline.
2. Taking Bing seriously for the first time in my life. Bing Webmaster Tools went from "never opened it" to the top of this week's list — if 87% of ChatGPT citations flow through Bing's index, it's the cheapest GEO lever there is.
3. Writing more single-topic, specific pages. Claude's 233-fetch obsession with one spam-detection post convinced me: assistants latch onto pages that answer one question thoroughly, not pillar pages that answer twelve questions shallowly.

Now I want your denominator. Pull one month of access logs and count bingbot vs Googlebot hits — my ratio is 4.4:1. If you run a small site, especially a non-English one, I'd genuinely like to know whether that ratio holds outside my niche: drop your two numbers (and your site's language) in the comments.