Monthly AI Crawler Report: May 2026 — Bytespider Surges to #4 as Applebot Reverses

This report is updated monthly with fresh Cloudflare Radar data. Bookmark this page to track how AI crawlers are reshaping web traffic each month.

Last month I told website owners to update their Applebot policy because Apple's crawler had leapfrogged Bingbot into the top five. One month later, that move has already unwound: Applebot fell from 9.1% to 7.0% and dropped back to #7. The crawler that took its place at the top is one almost nobody flagged a quarter ago — ByteDance's Bytespider surged +61% to 10.5%, vaulting past ClaudeBot, Bingbot, and Applebot to become the fourth-largest AI crawler on the web. Meanwhile Googlebot kept sliding (29.6% → 27.1%), and the "Training will exceed 55% by June" prediction is now in trouble: training crawling plateaued at 51.8%, flat month-over-month, while Search crawling quietly jumped from 7.7% to 9.6%.

I analyzed the latest 28 days of Cloudflare Radar AI Insights data — the window covering May 5 through June 2, 2026 — and compared it against the preceding 28-day window (April 7 through May 5) so every month-over-month figure below is computed on an identical basis. The picture is the same diversification story I've tracked all year, but with two sharp reversals: the bots everyone added to their robots.txt last month (Applebot) cooled, and the model everyone wrote off as a curiosity (Moonshot's Kimi K2.6) exploded from a single-digit new entrant to the #2 model on Workers AI at 18.4%. If you want the foundational primer on the pipelines behind this data, our explainer on how search engines really work walks through crawl budgets, inverted indexes, and learning-to-rank — all the systems these bots are feeding.

📊 Stats Alert: Bytespider grew +61% month-over-month (6.5% → 10.5%) to become the #4 AI crawler. Applebot's April top-five surge reversed, falling from 9.1% to 7.0%. Googlebot dropped to 27.1%. Training crawling plateaued at 51.8% while Search crawling jumped to 9.6%. On Workers AI, Kimi K2.6 rocketed to 18.4% to become the #2 model.

Who Are the Top AI Crawlers in May 2026?

Googlebot remains the largest AI-related crawler globally, but its decline accelerated again — from 29.6% to 27.1%, a -2.5 pp drop and a new all-time low in Cloudflare Radar's tracking. Meta-ExternalAgent stays in second at 13.1%, down for the second straight month. The big reshuffle happened just behind them: GPTBot recovered to 11.5% (#3), Bytespider exploded to 10.5% (#4), and ClaudeBot slipped to 9.5% (#5) — meaning ByteDance's crawler now out-crawls Anthropic's. Applebot's April leap reversed completely, falling -2.0 pp to 7.0% and surrendering the #5 spot it briefly held.

The other quiet headline is in the search tier: Claude-SearchBot tripled to 2.4% and overtook OpenAI's OAI-SearchBot (which fell back into the long tail). Anthropic's dedicated Claude web search crawler is now the single largest AI search crawler on the web — a notable lead for Anthropic in real-time retrieval, distinct from ClaudeBot's training role.

The concentration of crawling power among the traditional top five companies — Google, Meta, OpenAI, Anthropic, and Microsoft — fell to 69.5% in May (counting each company's primary crawler), down from 73.9% in April and continuing a five-month slide from 84.5% in January. The displacement is no longer a "long tail" story — it's ByteDance. Bytespider alone (10.5%) is now the fourth-largest individual AI crawler on the web, out-crawling every bot except Googlebot, Meta-ExternalAgent, and GPTBot — including both Anthropic's ClaudeBot (9.5%) and Microsoft's Bingbot (8.3%). Add ByteDance's second crawler (TikTokSpider, in the long tail) and ByteDance sits firmly among the top handful of AI crawler operators, a status it didn't hold a quarter ago. The diversification I've documented all year has a clear new winner, and it isn't Apple.

If you're managing AI bot access on your site, the operator that most needs a fresh look this month is ByteDance — Bytespider's traffic has nearly tripled since March (3.6% → 10.5%). For context on how crawling translates into actual referral traffic back to websites, check our companion Search Engine Referral Report on crawl-to-refer ratios — and see the new crawl-to-refer breakdown further down this report.

Which AI Crawlers Gained or Lost Ground This Month?

The headline shift in May is a single-crawler surge offsetting broad declines at the top. Googlebot (-2.5 pp), Applebot (-2.0 pp), ClaudeBot (-2.0 pp), and Meta-ExternalAgent (-1.4 pp) all gave up ground — and almost all of it flowed to Bytespider (+4.0 pp), with smaller gains to GPTBot (+1.3 pp), Amazonbot (+0.5 pp), and the search tier (Claude-SearchBot). This is the inverse of April, when the gains dispersed across many crawlers; in May they concentrated in one.

Six trends stand out from this comparison:

1. Bytespider is the story of the month. ByteDance's crawler jumped from 6.5% to 10.5% (+4.0 pp, +61% relative) — the single largest absolute and relative gain of any AI crawler in May, and its third consecutive month of growth (3.6% in March → 6.5% in April → 10.5% in May). Bytespider now out-crawls ClaudeBot, Bingbot, Applebot, and Amazonbot individually. ByteDance has gone from a rounding error to the fourth-largest AI crawler on the web in one quarter.

2. Applebot's surge reversed. Last month's most-confident call — "Applebot has entered the top five, update your policy" — didn't hold. Applebot fell -2.0 pp to 7.0%, dropping back below Bingbot to #7. April's spike looks increasingly like a one-month training burst (a backfill of content for Apple Intelligence) rather than a sustained ramp. This is a useful reminder that a single month of crawl data can mislead; the trend matters more than the spike.

3. ClaudeBot slipped below GPTBot. ClaudeBot fell -2.0 pp to 9.5% (#5) while GPTBot recovered +1.3 pp to 11.5% (#3). Anthropic's training crawler lost ground even as its search crawler (Claude-SearchBot) gained — suggesting Anthropic may be shifting effort from bulk training collection toward real-time retrieval.

4. Googlebot's decline accelerated again. After a -1.6 pp dip the prior month, Googlebot dropped -2.5 pp to 27.1% — no sign of the floor I speculated about last month. Google has now lost more than 11 percentage points of AI-crawler share since January's 38.7%.

5. The search tier flipped to Anthropic. Claude-SearchBot tripled to 2.4% and overtook OAI-SearchBot, which fell out of the top nine. For the first time, Anthropic operates the largest dedicated AI search crawler — a mirror image of the training market, where OpenAI's GPTBot leads Anthropic's ClaudeBot. If you're building apps that rely on real-time retrieval, understanding what a web search API is and how these bots work under the hood matters more than ever.

6. Meta kept contracting. Meta-ExternalAgent fell a second straight month, -1.4 pp to 13.1%. The "plateau at 18-20%" I predicted in the Q1 review is now decisively wrong in the opposite direction — Meta has lost 3.6 pp across April and May combined, the clearest sign that its aggressive Q1 Llama-training crawl was a finite campaign, not a new baseline.

What Are AI Bots Actually Doing With the Content They Crawl?

This is where last month's confident extrapolation broke. In April I projected Training would reach 55% by mid-Q2. Instead, Training plateaued at 51.8% in May — statistically flat (-0.04 pp) after seven straight months of gains. The category that actually moved was Search, which jumped +1.9 pp to 9.6%, its largest single-month gain on record.

Here's what the May data tells website owners:

Training plateaued at 51.8%. After climbing from 42.0% in January to 51.8% in April, the seven-month growth streak stalled in May. Training is still the clear majority — for every 100 AI bot requests, 52 are explicitly dedicated to training — but the "Training takeover" has reached a near-term ceiling. The simplest read: the Q1–Q2 land-grab of bulk training data is largely complete, and the major labs are crawling for refresh, not first acquisition. Bytespider's surge kept the category from declining; without it, Training would have fallen.

Mixed Purpose kept sliding to 35.4%. Crawlers that simultaneously index for search and collect training data have now lost ground for five consecutive months — from 48.3% in January to 35.4% in May, a cumulative -12.9 pp drop that tracks Googlebot's own decline almost exactly. You still can't separate the AI training from the search indexing with these crawlers — block Googlebot and you disappear from Google Search. For developers looking at how to ground AI responses with Google Search alternatives, this bundling problem keeps coming up.

Search crawling surged to 9.6% — the real shift this month. After holding flat for two months, AI search crawlers jumped +1.9 pp, driven almost entirely by Claude-SearchBot tripling to 2.4% (now the largest dedicated AI search crawler, having passed OAI-SearchBot). This is the strongest signal yet that the AI industry's center of gravity is shifting from "crawl everything once to train" toward "fetch pages in real time to answer queries." The growing ecosystem of AI search API alternatives is the developer-facing side of exactly this trend.

User Action ticked up to 2.6%. Real-time, user-triggered fetches (ChatGPT-User and equivalents) continued their slow climb. This is the category most directly tied to AI assistants' "browse" features, and it has now risen three months running.

The structural story of May is a two-front shift toward retrieval: Search (+1.9 pp) and User Action (+0.3 pp) both gained while Training stalled and Mixed Purpose fell. If April was the month Training crossed the majority line, May is the month that line stopped moving — and real-time retrieval started to.

Which Industries Are AI Bots Targeting Most?

The industry mix moved meaningfully in May, and in the opposite direction from April. Shopping & General Merchandise fell from 28.2% to 25.0% (-3.2 pp) — the largest single-month drop of any vertical this year — while Computer & Electronics (+1.2 pp) and News, Media & Publications (+1.1 pp) absorbed most of the redistributed crawling. The retail-content land grab that defined Q1 is cooling.

Shopping is still the most-crawled vertical, but its lead over Internet & Telecom narrowed to just 3.1 pp — the tightest gap on record, down from a double-digit lead at the start of the year. The News, Media & Publications rise (+1.1 pp) is worth watching: as Search and User-Action crawling grows (see the crawl-purpose section above), AI assistants are fetching more news and reference content in real time to answer time-sensitive queries — a different pattern than bulk e-commerce training scrapes.

The industry-level breakdown gets more granular:

The most notable industry-level move is Retail falling -2.9 pp to 22.5%, mirroring the Shopping vertical's decline, while Computer Software climbed +1.2 pp to 17.2% — AI crawlers are increasingly targeting documentation, code, and developer content over product catalogs. Adult Entertainment also jumped +1.0 pp to 4.6%, a recurring pattern when training crawlers do broad recrawls. If you maintain developer documentation, API references, or technical tutorials, this is why choosing the right AI web search API for your applications matters — these crawlers are the infrastructure behind the search results your users see, and software content is now the second-most-crawled industry on the web.

AI training crawlers aren't the only bots scanning the web at this scale, either. Technology detection platforms like Technologychecker.io crawl and fingerprint over 50 million domains using HTTP header analysis, JavaScript fingerprinting, DNS lookups, and headless browser rendering to identify 40,000+ technologies. Unlike AI training crawlers that take content for model weights, technology intelligence crawlers need to re-crawl frequently to track stack changes and new tech adoptions.

How Much Traffic Do AI Crawlers Give Back?

New this month, I pulled Cloudflare Radar's crawl-to-refer ratio — the number of pages an operator crawls for every one referral it sends back to a website. It's the single most honest measure of whether a given AI company is a fair exchange or a pure extractor, and the May numbers are stark.

According to Cloudflare Radar, Anthropic crawled roughly 11,992 pages in May for every single visitor it referred back — by far the most lopsided ratio of any major operator, and consistent with ClaudeBot's training-heavy footprint. OpenAI sits at 1,057:1, while Google returns the most traffic relative to what it takes, at just 5:1 — the structural advantage of running search and AI crawling through bundled infrastructure that still drives clicks. DuckDuckGo, which leans on others' indexes, is nearly even at 1.6:1.

For website owners, this is the number that reframes the "should I block AI crawlers?" question. A 5:1 ratio (Google) is a recognizable search-engine bargain — you give crawl access, you get visitors. A ~12,000:1 ratio (Anthropic) is not a bargain in the traditional sense; it's content acquisition with almost no traffic return. That gap is the clearest data-backed case for treating training crawlers and search crawlers differently in your robots.txt — exactly the separation Anthropic's own ClaudeBot/Claude-SearchBot split now makes possible.

Source: Cloudflare Radar — radar/bots/crawlers/summary/crawl_refer_ratio (radar.cloudflare.com), May 5 – June 2, 2026 vs. April 7 – May 5, 2026.

How Are Websites Fighting Back Against AI Crawlers?

⚠️ Methodology note: To sidestep the share-denominator problem that complicated last month's edition (Cloudflare refreshed its parsed robots.txt corpus in early Q2), this section reports raw domain counts — the number of distinct domains whose robots.txt explicitly names each crawler — rather than percentages. Counts are directly comparable month-to-month. May figures are the most recent Radar snapshot; April figures are last month's snapshot.

Most Referenced AI Crawlers in robots.txt — May 2026

Two rank changes stand out. Google-Extended overtook CCBot for #3, gaining +26 domains versus Common Crawl's +8 — website owners are increasingly opting out of Google's AI training (Google-Extended) while keeping search visibility (Googlebot). And PerplexityBot climbed to #7 (+24 domains), continuing the pattern where blocking activity tracks a crawler's visibility in logs rather than its raw traffic share.

The most telling non-move is Applebot-Extended, flat at 338 domains despite Applebot's traffic falling -2.0 pp this month. Robots.txt adoption lags traffic by a month or more — the owners who added Applebot-Extended directives during April's surge are still carrying them now that the surge has faded. Adoption is sticky; it accumulates on awareness and rarely reverses.

The meta-externalagent gap persists: despite being the #2 AI crawler by traffic at 13.1%, Meta's bot ranks only #8 by robots.txt references — still one of the largest disparities between traffic share and blocking attention among major crawlers. If you maintain a robots.txt and you're worried about training crawlers, Meta-ExternalAgent and the now-#4 Bytespider are the two highest-traffic bots most under-referenced relative to their footprint.

Two qualitative observations continue to hold:

1. Selective adoption, not blanket blocks. Website owners add rules for specific crawlers as they notice them, not site-wide AI bans. Google-Extended's rise and Applebot-Extended's stickiness both reflect awareness-driven, crawler-by-crawler decisions.
2. The blocking cohort is still small. The top-referenced crawler appears in 632 domains' robots.txt in Radar's parsed sample — meaningful and growing steadily (+35 month-over-month), but a tiny fraction of the global web. The overwhelming majority of sites still allow every AI crawler unrestricted access.

What's Happening on Cloudflare Workers AI?

Beyond crawlers, Cloudflare Radar tracks usage patterns on Cloudflare Workers AI — the platform that lets developers run AI models at the edge. If April reshuffled the model leaderboard, May rewrote it: Moonshot AI's Kimi K2.6 exploded from a low-single-digit new entrant to the #2 model at 18.4% — the largest single-month gain for any Workers AI model on record — while OpenAI's GPT-OSS 120B, last month's breakout, fell out of the top tier entirely.

Most Popular Models (by Account Share)

The headline is Kimi K2.6. Moonshot AI's model debuted at roughly 2% in April as "the first non-US model to crack the top 11" — a footnote. One month later it's the #2 model on Workers AI at 18.4%, a ~16 pp jump that is the single largest one-month gain Radar has recorded for any model. Combined with Google's Gemma 4 26B climbing to 13.0% (#3), two of the top three Workers AI models are now non-Meta — something that was unthinkable when Llama variants held ~60% of the platform in January.

Llama 3 8B keeps eroding. Still #1, but down to 21.5% from 28.0% — a third consecutive monthly decline, and less than half its 41.7% January peak. Meta's overall provider share fell to roughly 32–33% (Llama 3 8B + Llama 4 Scout + M2M-100 + Llama 3 AWQ), down from ~46% in April and ~60% in January. The Workers AI platform has gone from Meta-dominated to genuinely multipolar in two quarters.

The reversal that mirrors the crawler data: OpenAI's GPT-OSS 120B, which I called "the breakout adoption story of Q2" last month at 5.5%, fell out of the top nine (now under 2.6%). Just as Applebot's traffic surge didn't hold, GPT-OSS 120B's adoption spike didn't either. Both are reminders that a single month of momentum on Cloudflare's platform doesn't guarantee a trend — the developers experimenting with a new model in April moved on to Kimi K2.6 in May.

Task Distribution

Text Generation surged +12.0 pp to 66.1% — the largest single-month gain in any task category Radar tracks — directly attributable to Kimi K2.6 and Gemma 4 26B, both text-generation workloads, absorbing the share that fled Stable Diffusion XL. Text-to-Image fell -7.3 pp to 18.2% as image generation cooled across the board. The edge-AI workload mix is consolidating around text: between Text Generation and Translation, language tasks now account for more than 70% of all Workers AI inference.

Q1 2026 Quarterly Review: The Great Reshuffling

Q1 2026 was the most transformative quarter in the history of AI web crawling. This section compiles data from three consecutive monthly analyses to document the structural shifts reshaping how AI companies interact with the open web.

Q1 2026 at a Glance

How Each Crawler Moved Across the Quarter

Googlebot's sustained decline is the defining trend of Q1. Losing 7.1 pp over three months represents the largest quarterly share loss for any single crawler in Cloudflare Radar's tracking history. This doesn't necessarily mean Google is crawling less — it means competitors are crawling more, faster. At 31.6%, Googlebot is no longer 2x the size of the next-largest crawler; the ratio to Meta-ExternalAgent is now 1.9x and shrinking.

Meta-ExternalAgent was the biggest winner of Q1. Gaining +5.1 pp over three months, Meta's crawler overtook GPTBot in February and never looked back. The growth rate decelerated across the quarter (+3.1 pp → +3.7 pp → +2.3 pp), suggesting Meta may be approaching a near-term equilibrium.

Applebot's March surge was the quarter's biggest surprise. After modest growth in January (+0.2 pp) and February (+0.6 pp), Applebot exploded in March (+3.2 pp, +124% relative). Over Q1, Applebot more than doubled from 2.5% to 5.8%. Apple is now a top-six AI crawler operator — a status no one predicted at the start of the quarter.

The Training-Mixed Purpose Crossover

The most consequential structural shift of Q1 2026 happened in the crawl purpose data:

In January, Mixed Purpose led Training by 6.3 pp (48.3% vs 42.0%). By February, Training had overtaken Mixed Purpose for the first time (45.4% vs 43.9%). By March, the gap had widened to 10 pp (49.9% vs 39.9%).

This crossover represents a fundamental change in how AI companies interact with the web:

• Before Q1 2026: The majority of AI bot traffic came from dual-purpose crawlers (Googlebot, Bingbot) that bundled search indexing with AI training. Website owners couldn't separate the two.
• After Q1 2026: The majority comes from purpose-built training crawlers (GPTBot, ClaudeBot, Meta-ExternalAgent, Applebot) that exist solely to collect AI training data. Website owners can block these selectively.

The shift gives website owners more control. You can now block the majority of AI training crawling without sacrificing search visibility — something that wasn't possible when Mixed Purpose crawlers dominated.

Market Concentration Is Declining

The AI crawler market is diversifying. The traditional top five lost 4.3 pp of concentration over Q1, but when you include Apple as a sixth major player, the top-six concentration remained remarkably stable at ~86%. The diversification is happening within the top tier, not from outside it.

Quarterly Robots.txt Blocking Trends

ClaudeBot saw the largest blocking increase across Q1 at +0.39 pp, overtaking CCBot to become the second-most referenced AI crawler in robots.txt files behind GPTBot. The blocking wave is real but slow — over the entire quarter, GPTBot grew from 5.29% to just 5.52%. More than 94% of domains still allow all AI crawlers unrestricted access. At the current rate, it would take years for blocking rates to reach even 10%.

The Traffic-Blocking Gap

Meta-ExternalAgent's 13.4 pp gap is the most actionable — it's a dedicated training crawler with no search indexing benefit, yet it's blocked by fewer domains than GPTBot despite generating more traffic.

Workers AI Model Ecosystem Evolution

Every top model except FLUX.1 Schnell, GPT-OSS 120B, and Whisper Large V3 Turbo lost share across Q1. Meta's dominance is slowly eroding — from ~60% in January to 53.8% in March. GPT-OSS 120B is the breakout model of Q1, debuting in February and climbing to 2.1% in March. FLUX.1 Schnell is gaining on Stable Diffusion XL, emerging as a credible challenger in the text-to-image space.

Five Structural Shifts That Defined Q1 2026

1. The Training Takeover. Training crawlers went from minority (42.0%) to near-majority (49.9%) in a single quarter. The web's content is now primarily being consumed for AI model weights rather than search indexing. Website owners who want to opt out of AI training have clearer tools to do so.

2. The Google Erosion. Googlebot lost 7.1 pp across Q1. This is almost certainly driven by competitors growing faster rather than Google crawling less, but the proportional shift matters for every website owner's traffic analysis.

3. The Meta Ascendancy. Meta-ExternalAgent gained +5.1 pp, overtook GPTBot for #2 in February, and finished the quarter at 16.7%. Meta's aggressive Llama training pipeline is driving unprecedented data collection.

4. Apple's Arrival. Applebot went from 2.5% to 5.8% across Q1, with most growth concentrated in March. Apple is now a top-six AI crawler operator and should be included in any website owner's AI bot management strategy.

5. The Diversification of Workers AI. The model ecosystem became meaningfully more diverse. Llama 3 8B's share fell from 41.7% to 37.3% as developers adopted newer models and the long tail grew from ~15% to ~20%.

Q2 2026 Predictions (Made in the April Edition)

Based on Q1 trends, here's what I forecast for Q2 2026 — scored against May data in the tracker below:

1. Training crawlers will exceed 55%. The +7.9 pp Q1 trajectory suggests Training could reach 55-57% by June 2026, with Mixed Purpose falling below 35%.

2. Googlebot will drop below 30%. The current quarterly decline rate would put Googlebot in the 28-30% range by end of Q2.

3. Applebot will enter the top five. If Apple sustains even half of March's growth rate, Applebot could pass Bingbot (currently 8.2%) by May or June.

4. Meta-ExternalAgent will plateau around 18-20%. The deceleration trend suggests Meta's growth rate is stabilizing.

5. GPT-OSS 120B will enter the Workers AI top five. At its current growth rate, it could reach 3-4% by June, approaching FLUX.1 Schnell territory.

6. Robots.txt blocking will remain below 6% for all crawlers. The slow growth rate means meaningful blocking thresholds remain years away.

Q2 Predictions Tracker: May Check-In

With one month of Q2 already scored in April and a second now in, here's where each prediction stands. The pattern from last month inverted: the calls that "hit early" in April (Applebot top-five, GPT-OSS 120B) have since reversed, while the slow-burn structural calls (Googlebot decline) keep compounding.

The training prediction has broken. Last month I said 55% by June was "on track" at a ~+1.7 pp/month pace. May delivered +0.0 pp. Training didn't just decelerate — it stopped. Barring a June surge from Bytespider, this prediction will miss, and the more interesting story (real-time Search crawling rising to fill the gap) is the one I should have been forecasting.

Two predictions hit, then reversed — the lesson of the month. Applebot entered the top five in April (prediction #3, "hit early") and GPT-OSS 120B reached the Workers AI top five (#5, "near hit"). Both unwound in May: Applebot fell to #7, GPT-OSS 120B dropped out of the top nine. A single month above a threshold is not the same as crossing it durably. This is the clearest cautionary tale in two editions of this report — momentum-based "early hits" are the most fragile kind of prediction.

The structural calls keep compounding. Googlebot's sub-30% prediction (#2) didn't just hit — it overshot to 27.1%, with no floor in sight. The Meta plateau prediction (#4) keeps getting more wrong as Meta contracts further. The common thread: predictions grounded in a sustained, multi-month structural trend (Google's proportional erosion, Meta's finite training campaign) are far more reliable than predictions that extrapolate a single steep month.

Revised forecasts for the rest of Q2 (June):

1. Bytespider will challenge GPTBot for #3. At +4.0 pp/month it could pass GPTBot (11.5%) in June, making ByteDance's primary crawler the third-largest on the web.
2. Kimi K2.6 will challenge Llama 3 8B for #1 on Workers AI. An 18.4% share rising while Llama 3 8B falls to 21.5% puts a #1 upset within one month's reach.
3. Search crawl purpose will pass 10%. Claude-SearchBot's growth plus rising User-Action traffic should push real-time retrieval past the 10% line in June.
4. Training will stay flat (51–53%), not reach 55%. The plateau is real; June won't reverse it.

What Should Website Owners Do About May's Trends?

Based on what I've found in the May 2026 Cloudflare Radar data, here are the actions I'd prioritize:

Make Bytespider your top robots.txt priority this month. ByteDance's Bytespider surged to 10.5% — the #4 AI crawler, ahead of ClaudeBot, Bingbot, and Applebot, and nearly triple its March share. If your robots.txt was written around the "big four" of Google, Meta, OpenAI, and Anthropic, it now misses the fastest-growing major crawler on the web. Add an explicit Bytespider directive.

Don't over-react to last month's Applebot surge. I told you in April to update your Applebot policy because it had entered the top five. It has since fallen back to #7 (9.1% → 7.0%). Keep your Applebot-Extended directive if you added one — adoption is sticky and Apple may ramp again — but Applebot is no longer the urgent case it looked like a month ago. The urgent case is Bytespider.

Separate training from search — the crawl-to-refer data proves why. Anthropic crawls roughly 12,000 pages for every referral it sends back (vs. Google's 5:1). Claude-SearchBot (search, now 2.4% and the largest AI search crawler) is distinct from ClaudeBot (training, 9.5%). If you want Claude's search to surface your content while opting out of training, use separate directives — and apply the same logic to OpenAI's GPTBot/OAI-SearchBot split.

Watch the shift from training to retrieval. Training crawling plateaued at 51.8% while Search jumped to 9.6%. A robots.txt policy written purely to block bulk training scrapers will increasingly miss the real-time search and user-action fetches that actually drive AI-assistant citations back to your site. Decide deliberately whether you want to be in AI answers (allow search crawlers) or out of training sets (block training crawlers) — they're now genuinely separable.

Don't assume Meta-ExternalAgent will keep growing. Meta's bot fell again to 13.1% — its second straight monthly contraction (16.7% in March → 13.1% in May). Meta's aggressive Q1 Llama-training crawl looks like a finite campaign, not a permanent baseline.

Audit your Googlebot assumptions. Googlebot fell to 27.1%, down from ~39% in January. For every 10 AI bot requests hitting your site at the start of the year, Google accounted for ~4; now it's closer to ~3. The other 7 are increasingly dedicated training and search crawlers — not search-indexing bots.

How WebSearchAPI.ai Fits Into the AI Crawler Ecosystem

Every AI crawler in this report exists because AI companies need fresh, structured web data to power their models and search products. At WebSearchAPI.ai, we sit on the other side of this equation — providing developers and AI agents with a clean, fast, and affordable way to access real-time web data without running their own crawlers.

Here's why this matters in the context of May's trends:

• The balance is shifting from training to real-time retrieval. Training crawling plateaued at 51.8% while Search crawling jumped to 9.6% and Claude-SearchBot became the largest dedicated AI search crawler. The bulk-training land grab is maturing; the next phase is real-time fetching to answer live queries. Instead of building and maintaining crawling infrastructure for either job, WebSearchAPI.ai gives you instant access to structured search results, content extraction, and real-time web intelligence through a single API call.
• The crawler landscape is fragmenting faster than it's consolidating. With ByteDance's Bytespider now #4, Applebot reversing, Anthropic leading the search tier, and top-five concentration falling to 69.5%, the complexity of managing web data access grows every month. WebSearchAPI.ai handles the retrieval layer so you can focus on your application logic.
• Sub-second latency, 99.9% uptime, and structured responses mean your AI applications get the data they need without the infrastructure headaches that come with managing crawler fleets.

If the data in this report tells you anything, it's that the volume and complexity of AI web crawling is only accelerating — and the balance is now tilting toward real-time retrieval. WebSearchAPI.ai is purpose-built for developers who want to harness that web intelligence without becoming a crawling operation themselves. Learn more about what a web search API can do for your stack.

Frequently Asked Questions

What is an AI crawler?

An AI crawler (also called an AI bot or AI spider) is an automated program that visits websites to collect content for training artificial intelligence models or powering AI-powered search features. Unlike traditional search engine crawlers that index pages for search results, AI crawlers like GPTBot, ClaudeBot, and Meta-ExternalAgent specifically collect data to train large language models (LLMs). Some crawlers like Googlebot serve both purposes — indexing for search and collecting training data simultaneously. You can identify AI crawlers by their user-agent strings in your server logs or through tools like Cloudflare Radar.

How often is this AI crawler report updated?

This report is updated monthly with fresh data from Cloudflare Radar AI Insights. Each edition covers a rolling 28-day window and compares it against the immediately preceding 28-day window, so every month-over-month figure is computed on an identical basis. Quarterly editions add full-quarter trajectory analysis (the Q1 2026 review remains in this post as a historical anchor). Bookmark this page or check back at the beginning of each month for the latest analysis of AI crawler traffic patterns, market share shifts, and robots.txt directives.

Can I block AI crawlers from my website?

Yes. The primary method is adding disallow rules to your robots.txt file for specific AI crawler user agents. For example, adding User-agent: GPTBot followed by Disallow: / will request that OpenAI's crawler stop visiting your site. However, robots.txt is a voluntary protocol — crawlers are not technically required to obey it. As of May 2026, GPTBot and ClaudeBot remain the two most-referenced AI crawlers in robots.txt files (appearing in 632 and 539 domains respectively in Cloudflare Radar's parsed sample), with Google-Extended now third, but absolute adoption is still a small fraction of the web. Some CDN providers like Cloudflare also offer dashboard-level controls to block or rate-limit AI bots.

What is the difference between AI training crawlers and AI search crawlers?

AI training crawlers (like GPTBot, ClaudeBot, and Meta-ExternalAgent) collect web content to build and improve AI models. They typically scrape large volumes of content from many sites. AI search crawlers (like OAI-SearchBot and the newly-observed Claude-SearchBot) fetch specific pages in real time when a user performs a search query through an AI tool like ChatGPT or Claude. The key difference: training crawlers take your content to make the model smarter, while search crawlers fetch your content to answer a specific user question — and may drive traffic back to your site. As of May 2026, training crawling commands a clear majority at 51.8% of all AI bot traffic but has plateaued, while search crawling jumped to 9.6% — its strongest month yet, led by Anthropic's Claude-SearchBot.

Will blocking AI crawlers affect my SEO or search rankings?

Blocking dedicated AI training crawlers like GPTBot, ClaudeBot, Meta-ExternalAgent, Bytespider, or Applebot-Extended will not affect your rankings in Google, Bing, or other traditional search engines. These crawlers are separate from the search indexing bots. However, blocking Googlebot will remove your site from Google Search entirely since Google uses the same crawler for both search indexing and AI training. Google offers a middle ground with the Google-Extended user agent — blocking it opts you out of AI training while keeping your search presence intact. Apple offers the same kind of separation with Applebot-Extended, which sits in the top 12 most-referenced robots.txt user agents as of May 2026.

Why did Applebot's surge reverse in May 2026?

After Applebot leapt from 5.8% (March) to 9.1% (April) and briefly entered the top five, it fell back to 7.0% in May, surrendering the #5 spot and dropping to #7. The most likely explanation is that April's spike was a finite training burst — a backfill of web content to support Apple Intelligence — rather than a sustained crawling ramp. It's a useful reminder that a single month above a threshold doesn't establish a durable trend. Website owners who added Applebot-Extended directives during the April surge can keep them (adoption is sticky and Apple may ramp again), but Applebot is no longer the urgent case it appeared to be a month ago; ByteDance's Bytespider, now the #4 crawler at 10.5%, is.

What changed with Anthropic's crawlers in May 2026?

Anthropic's two crawlers moved in opposite directions in May. ClaudeBot (training) fell to 9.5%, slipping below GPTBot to #5, while Claude-SearchBot (search) tripled to 2.4% — overtaking OpenAI's OAI-SearchBot to become the single largest dedicated AI search crawler on the web. This mirrors OpenAI's GPTBot/OAI-SearchBot split but in reverse: OpenAI leads in training (GPTBot), while Anthropic now leads in search (Claude's web search functionality). The divergence suggests Anthropic is shifting effort from bulk training collection toward real-time retrieval, and it gives website owners a clear reason to set separate directives for the two bots.

How does Cloudflare track AI crawler traffic?

Cloudflare's global network spans 330+ cities in 125+ countries and processes over 81 million HTTP requests per second. Through its Radar platform, Cloudflare identifies and classifies AI bot traffic by analyzing user-agent strings, request patterns, and behavioral signatures across all sites on its network. The data in this report comes from Cloudflare Radar's AI Insights endpoints, which aggregate these signals into share-of-traffic percentages by bot, crawl purpose, industry, and region.

Which AI crawler is growing the fastest in 2026?

In May 2026, Bytespider had both the largest absolute and relative growth, +4.0 pp (+61%) (6.5% → 10.5%), making ByteDance's crawler the #4 AI bot on the web. Bytespider has now grown for three consecutive months (3.6% → 6.5% → 10.5%), the most sustained surge of any crawler this year. Applebot, last month's standout, reversed sharply (-2.0 pp to 7.0%), and Claude-SearchBot was the fastest riser in the search tier, tripling to 2.4% to become the largest dedicated AI search crawler.

What percentage of web traffic comes from AI bots?

The percentages in this report represent share of identified AI bot requests, not share of total web traffic. Cloudflare Radar tracks the proportion of AI-related crawler activity relative to other AI bots, providing a competitive landscape view. The actual percentage of total web traffic from AI bots varies by website, but industry estimates suggest AI crawlers now account for a meaningful and growing share of overall internet traffic, particularly for content-heavy sites in retail, technology, and media.

How can I monitor AI crawler activity on my own website?

Check your server access logs for known AI bot user-agent strings (GPTBot, ClaudeBot, meta-externalagent, Applebot, Bytespider, Amazonbot, etc.). Most web analytics platforms filter out bot traffic by default, so log-level analysis gives the most accurate picture. Cloudflare users can view AI bot activity directly in their dashboard. For a structured approach, consider using a web search API to understand how your content appears in AI-powered search results and ensure your most important pages are properly accessible.

Cloudflare Radar Data Source & Methodology

Understanding where this data comes from — and what it can and cannot tell you — is critical for interpreting the trends above. Here's a full breakdown of how Cloudflare Radar collects, classifies, and aggregates the AI crawler data used in this report.

Network Scale

This scale is what makes Cloudflare Radar one of the most comprehensive sources of internet traffic data available. The data in this report comes from two primary sources:

1. Cloudflare's global network — real-time traffic data from HTTP requests flowing through their infrastructure
2. 1.1.1.1 public DNS resolver — aggregated and anonymized DNS query data

For routing data, Cloudflare also uses RIPE RIS data from RIPE NCC (BGP route collectors).

How AI Bots Are Identified

Cloudflare uses a layered detection system to identify and classify AI crawlers:

1. User-agent string matching — the most basic method; identifies bots that transparently announce themselves (GPTBot, ClaudeBot, etc.)
2. Verified Bot Directory — manual approval process requiring bots to maintain public robots.txt commitments, use dedicated/verifiable IPs, unique user-agents, and honor crawl-delay settings
3. Machine learning — supervised ML system that assigns a Bot Score (1-99)
4. Heuristics — tailored rulesets for AI bot classification
5. Behavioral analysis — pattern recognition from request sequences
6. AI Labyrinth honeypot — hidden links to AI-generated decoy pages; bots that follow them are identified with high confidence since human visitors never see these links
7. ai.robots.txt list — used as the basis for which AI bots to track

💡 Expert Insight: The layered approach matters because not all AI crawlers identify themselves honestly. User-agent matching catches transparent bots like GPTBot and ClaudeBot. Behavioral analysis and honeypots catch crawlers that disguise themselves as regular browsers.

How Crawl Purpose Is Classified

Bots are categorized into these purpose buckets:

• Training: dedicated training crawlers (GPTBot, ClaudeBot, Meta-ExternalAgent)
• Search: AI search bots (OAI-SearchBot)
• User Action: bots fetching pages for real-time user queries (ChatGPT-User)
• Mixed Purpose: bots serving dual roles like search indexing + AI training (Googlebot, Bingbot)
• Undeclared: purpose not identifiable

Data Aggregation Methods

• 7-day trailing average used to smooth daily fluctuations
• IPv4 addresses aggregated into /20 prefixes for visualization
• HTML traffic is separately classified into human, AI bot, and non-AI bot categories
• Normalization: data expressed as percentage of total requests (not absolute counts)
• Methodologies remain unchanged year-over-year for valid comparisons
• API data available under CC BY-NC 4.0 license

Caveats & Limitations

⚠️ Warning: Keep these limitations in mind when interpreting the data in this report:

• Countries with insufficient data volume are excluded from trend reporting
• Some metrics are available only at worldwide level, not per-country
• Mobile device categorization relies on User-Agent headers (accuracy limitations)
• Speed test data excludes locations with fewer than 100 tests per week
• The "location" filter corresponds to the billing country of the Cloudflare customer whose site received the traffic, not where the crawler is physically located
• Cloudflare sees traffic only to sites behind its network, not the entire internet, so the data is representative but not exhaustive

Report Parameters

This edition uses data from Cloudflare Radar's AI Insights endpoint (/radar/ai/bots/summary/*), Workers AI inference endpoint (/radar/ai/inference/summary/*), web-crawler endpoints (/radar/bots/crawlers/summary/{vertical|industry|crawl_refer_ratio}), and robots.txt analysis endpoint (/radar/robots_txt/top/user_agents/directive). The May 2026 monthly data covers the rolling 28-day window of May 5 through June 2, 2026, with every month-over-month comparison computed against the immediately preceding 28-day window (April 7 through May 5, 2026) via the API's dateRange=28d and dateRange=28dControl parameters, so both columns share an identical methodology. The Q1 2026 Quarterly Review compiles data from three consecutive monthly analyses covering January through March 2026.

I queried bot traffic breakdowns by user agent, crawl purpose, industry, and vertical; the new crawl-to-refer ratio by operator; Workers AI model and task distribution by account share; and domain-level robots.txt directives. All percentages represent share of identified AI bot requests (for crawling data) or share of accounts (for Workers AI data), not share of total web traffic. The crawl-to-refer ratio is a RATIO (crawls per referral), and robots.txt figures are raw domain counts.

⚠️ Note on revised values: Cloudflare Radar aggregates and may revise data after publication, and it refreshed its parsed robots.txt corpus in early Q2 2026. To keep month-over-month deltas clean, this edition computes the prior-month ("April 2026") column from the API's 28dControl window rather than re-using last edition's published figures, and reports robots.txt as raw domain counts rather than percentages. Where last month's published numbers differ slightly, the difference reflects Radar's revisions, not a change in this report's method.

Data source: Cloudflare Radar AI Insights and Web Crawlers API endpoints (radar.cloudflare.com), May 5 – June 2, 2026 vs. April 7 – May 5, 2026. Last updated: June 2, 2026.

About the Author: I'm James Bennett, Lead Engineer at WebSearchAPI.ai, where I architect the core retrieval engine enabling LLMs and AI agents to access real-time, structured web data with over 99.9% uptime and sub-second query latency. With a background in distributed systems and search technologies, I've reduced AI hallucination rates by 45% through advanced ranking and content extraction pipelines for RAG systems. My expertise includes AI infrastructure, search technologies, large-scale data integration, and API architecture for real-time AI applications.

Credentials: B.Sc. Computer Science (University of Cambridge), M.Sc. Artificial Intelligence Systems (Imperial College London), Google Cloud Certified Professional Cloud Architect, AWS Certified Solutions Architect, Microsoft Azure AI Engineer, Certified Kubernetes Administrator, TensorFlow Developer Certificate.