The Search Engine Referral Report: Google's 90.9% Dominance & the TikTok Collapse

📊 Stats Alert: Google generates 90.9% of all search engine referral traffic and 87.9% of all web referrals globally — up from 81.6% in January. TikTok has stabilized at 3.2% after collapsing from 10.6%, leaving Bing as the #2 referrer. AI search bots rose to 6.6% of all verified bot traffic, and ChatGPT referrals resumed growth to 0.25%. Meanwhile, Anthropic still crawls ~11,992 pages for every single referral it returns — a 3.6x improvement from January but still the worst ratio on the web.

When I first published this report in February covering January 2026 data, the headline was Google's overwhelming 81.6% referral share and TikTok's surprise position as the second-largest referrer. Two editions later, the structural shifts have settled: Google's grip is tighter, not looser, TikTok's referral share has imploded and now sits flat near 3%, and the AI-search bot category keeps climbing. I re-analyzed the latest 28 days of Cloudflare Radar data — May 5 through June 2, 2026 — against the preceding 28-day window, and the new numbers sharpen the picture of one company's expanding control over how people discover websites.

💡 Author note from James Bennett: I update this report alongside the Monthly AI Crawler Report because the WebSearchAPI.ai team uses these crawl-to-refer ratios directly to calibrate our outbound bot policy. When ByteDance's ratio climbed from 2.6:1 in January to ~9:1 now, we saw the matching pattern in our own infrastructure logs — Bytespider's request rate against our origins climbed in lockstep, and in the May crawler data Bytespider became the #4 AI crawler on the web. The Cloudflare Radar dataset isn't an academic exercise for us; it's a real input into bandwidth and rate-limit decisions.

How Much of All Web Referral Traffic Does Google Control?

According to Cloudflare Radar web crawler referral data (/radar/bots/crawlers/summary/REFERER endpoint), Google properties now generate 87.9% of all identified referral traffic globally — up from 81.6% in January. The +6.3 percentage-point gain across the quarter represents the steepest tightening of Google's referral grip since Cloudflare Radar began tracking the metric.

gemini.google.com, claude.ai, perplexity.ai, and scholar.google. now sit within the sub-0.05% long tail (collapsed into "other" in the latest window); each remains a rounding-error referrer.

When I strip out non-search referrers like TikTok and isolate only traditional search engines, Google's share rises to approximately 90.9% of all search engine referral traffic — essentially unchanged from January's 91.2% in proportional terms, but on a much larger base because TikTok is no longer pulling traffic out of the search-engine denominator. Among every 100 visitors a website receives from a search engine, roughly 91 still come from Google.

The micro-shifts this quarter all favor Google. Bing edged up to 3.40% (+0.59 pp vs January), Yandex to 1.96%, and DuckDuckGo to 1.40%. The notable reversal is chatgpt.com referrals resuming growth to 0.25% — after a flat reading last edition, OpenAI's referral footprint ticked up again, the largest proportional move among the AI-search domains. The rest of the AI-search trio (Gemini, Claude.ai, Perplexity) remains in the sub-0.05% long tail: present in the data, but still rounding errors against Google's 87.9%.

💡 Expert Insight from James Bennett (Lead Engineer, WebSearchAPI.ai):

The 91% Google figure is what shows up in the report, but in production at WebSearchAPI.ai I treat it as effectively 100%. When we run our retrieval evaluation harness against real-world content publishers, we never see a customer whose organic referral mix includes more than a sliver from non-Google sources. The minor reshuffling at the bottom of the table (Yandex creeping up, Baidu reshuffling between mobile and desktop) is fine for trivia but irrelevant for retrieval strategy. If you're optimizing your content for "everyone except Google," you're optimizing for under 9% of search referrals. Build for Google, with explicit fallback handling for AI search bots, and you'll have covered 99% of the addressable surface.

What Happened to TikTok as a Referral Source?

The defining quarter-over-quarter story remains that TikTok's referral share collapsed from 10.59% in January to 3.25% now — a 69% relative decline. But the news in the latest window is that the collapse has stopped: TikTok held flat month-over-month (3.22% → 3.25%), confirming this is a new, lower floor rather than a continuing slide. Bing remains the #2 referrer at ~3.56% combined.

There are three plausible explanations for TikTok's collapse, and I think two of them are operating at once:

1. In-app browser tightening. TikTok pushed an update in late February that more aggressively keeps users inside its in-app browser when tapping bio links, with fewer instances spawning external browser sessions that would generate identifiable referrer headers. This means real link clicks may still be happening — they just stop generating Cloudflare-visible referrers.
2. Referrer policy changes. TikTok appears to have moved more outbound clicks to a noreferrer policy in March 2026, similar to what most major social platforms did between 2019-2022. This systematically strips the Referer header on outbound clicks.
3. Genuine engagement decline. US regulatory pressure and the H2 2025 platform-share churn had real effects, but those alone wouldn't produce a 70% drop in 90 days.

The result for website owners: TikTok's measured referral share is now structurally lower regardless of actual click volume. Don't take the 3.25% figure at face value as a measure of TikTok's user-driven traffic; take it as a measure of TikTok-to-Cloudflare-visible referrer headers.

💡 Expert Insight from James Bennett (Lead Engineer, WebSearchAPI.ai):

When we instrumented the WebSearchAPI.ai retrieval engine to track inbound referrers (we needed to know whether RAG agents were citing pages we'd indexed), we saw the exact same TikTok signal disappear in a two-week window in March 2026. The drop wasn't gradual. The day the TikTok app shipped its in-app browser update, our identifiable TikTok referrer count fell off a cliff. This isn't speculation — it lines up with the Cloudflare Radar data perfectly. The lesson: when a platform changes how it labels outbound traffic, your dashboards lie before your reality does.

How Has Google's Referral Dominance Changed Month Over Month?

Comparing the current 28-day data (May 5 - June 2, 2026) against the immediately preceding 28-day window (April 7 - May 5, 2026), the major shifts have already happened — month-over-month, the picture is now stable. According to Cloudflare Radar referral analytics:

Google's grip continues to tighten — the +0.31 pp gain is the largest single-month move in the table and consistent with the Q1 → Q2 trajectory that took Google from 81.6% to 87.9%. TikTok has stopped sliding: after months of decline following the March in-app-browser change, it held flat (+0.03 pp), confirming the referrer-policy effect has fully rolled out and 3.2% is the new floor.

The most interesting micro-trend is at the bottom: chatgpt.com referrals resumed growth, adding +0.05 pp to reach 0.25% — the largest proportional move of any AI-search referrer this month and a reversal of last edition's flat reading. After I flagged a possible plateau last month, OpenAI's referral footprint ticked back up, suggesting ChatGPT Search is sending users to source links more often, not less. The rest of the AI-search trio (Gemini, Claude.ai, Perplexity) stayed in the sub-0.05% long tail. I'll come back to what this means when discussing crawl-to-refer ratios.

💡 Expert Insight from James Bennett:

ChatGPT referrals ticking back up this month is a genuinely interesting wrinkle. At WebSearchAPI.ai we run continuous evaluations against ChatGPT, Claude, Perplexity, and Gemini, and we report a "citation-to-click ratio" alongside our retrieval quality metrics — the gap between "your URL was cited" and "a user actually clicked through" had widened all through Q1, so seeing chatgpt.com referrals reverse and grow is the first sign that gap may be narrowing for at least one operator. It's one data point, not a trend yet, but it's worth watching: the CTR of an AI-cited link behaves nothing like a Google SERP link, and the operators are clearly still tuning how aggressively they surface source links.

Which Search Engines Crawl the Most Relative to What They Send Back?

This is where the data gets interesting. Cloudflare Radar tracks crawl-to-refer ratios (/radar/bots/crawlers/summary/CRAWL_REFER_RATIO endpoint), measuring how many crawl requests each operator makes for every referral they send to websites. A lower ratio means the operator is more "generous" — it sends traffic back to sites relative to the content it consumes.

DuckDuckGo remains the most efficient referrer on the web at 1.6:1. Google sits at a moderate 5.0:1 — essentially where it was in January — meaning it crawls roughly five pages for every referral it generates. Google's stability across the quarter is itself a finding: as competitors' ratios swing wildly, Google's index/refer machinery is operating in steady-state.

The dramatic moves are at the extremes of the table. Anthropic's ratio improved 3.6x from 43,214:1 to 11,992:1 — by far the largest improvement in the dataset, and an extension of the trend I flagged last edition. This is consistent with Claude-SearchBot (Anthropic's search-specific crawler, which in the May AI Crawler Report tripled to 2.4% and became the largest dedicated AI search crawler) generating steadily more referral traffic. OpenAI's ratio also improved to 1,057:1, consistent with chatgpt.com referrals resuming growth this month.

Going the opposite direction, ByteDance's ratio worsened 3.5x from 2.6:1 to 9.1:1. The data tells a coherent story when you cross-reference: Bytespider's traffic share kept surging in the May crawler report (6.5% → 10.5%, now the #4 AI crawler), while TikTok's referral share stayed collapsed at 3.2%. Many more crawls + flat referrals = a much worse ratio. ByteDance is now one of the least efficient mainstream referrers, having been the second-most-efficient in January.

Mistral worsened 2.7x (21.9:1 → 60.2:1) and Baidu 3.0x (4.1:1 → 12.4:1) — both crawling more aggressively while sending little referral traffic back. Mistral's jump is the new entry to watch on origin logs.

⚠️ Warning: If you run a content-heavy website, audit your server logs for AI crawler traffic. Anthropic (11,992:1) and OpenAI (1,057:1) still consume bandwidth at ratios that offer minimal return in referral traffic, even after their improvements. ByteDance's 3.5x worsening is the most actionable change in this report — Bytespider became the #4 AI crawler on the web in May, so its footprint on your origin is now meaningfully larger than it was in January. Selective robots.txt rules and rate limits should be reviewed quarterly.

What Drove Anthropic's Massive 3.6x Improvement?

Anthropic's ratio fell from 63,331:1 (December 2025) → 43,214:1 (January 2026) → 11,992:1 (May 2026). Two compounding effects explain the trend:

1. Claude-SearchBot started referring real traffic. When I tracked Anthropic's crawlers in early Q1, only ClaudeBot existed. Cloudflare Radar later identified Claude-SearchBot as a separate user agent, and by May it had tripled to 2.4% to become the largest dedicated AI search crawler on the web. Search-purpose bots generate referrals (users clicking through to citations); training-purpose bots don't.
2. ClaudeBot's crawl volume eased. ClaudeBot's traffic share peaked at 13.0% in late 2025 and has since fallen to 9.5% in the May crawler data. The crawl numerator is shrinking while the referral denominator (Claude-SearchBot) grows — the ideal combination for a falling ratio.

OpenAI's improvement is smaller but follows the same pattern: even though GPTBot's training-crawl share recovered to 11.5% in May, its search-side referral traffic (chatgpt.com) grew faster, nudging the overall ratio down to 1,057:1.

💡 Expert Insight from James Bennett:

The 11,992:1 number is still extreme — it means a publisher gets one human visitor back for every ~12,000 pages Anthropic crawls. At WebSearchAPI.ai we built our own internal "fairness ratio" metric for this exact reason: we measure crawl volume against the referral traffic our customers' content actually receives downstream. Our internal metric for the API itself is currently around 1:1 (we crawl what we serve), which is why we publish it transparently — it's the inverse of the AI crawler problem and we want publishers to know it. If your operator-of-record sits in the 1,000:1 range or worse, you have leverage to rate-limit them; the public data backs up the case.

Who Dominates the Bot Traffic Hitting Your Site?

Search engine crawlers represent 30.0% of all verified bot traffic globally — down from 33.0% in January. AI crawlers grew to 20.2% of bot traffic, and the AI search category has more than doubled (more on that below). According to Cloudflare Radar bot data (/radar/bots/summary/BOT endpoint), the top individual crawlers are:

GoogleBot still dominates at 15.9% of all verified bot traffic — but the gap to AI crawlers has narrowed dramatically. Meta-ExternalAgent (7.7%), GPTBot (6.9%), and Applebot (4.2%) collectively generate 18.8% of bot traffic — meaningfully more than GoogleBot alone. Note two moves that track the May crawler report exactly: Baiduspider jumped to #4 (5.5%) as Baidu crawled more aggressively, while Applebot fell to 4.2%, the referral-side echo of Applebot's traffic reversal documented in the May AI Crawler Report.

When you add Google's full footprint (GoogleBot + Google AdsBot + image/video bots), Google still represents roughly 20-22% of all bot traffic. Combined with Google's 87.9% referral share and Chrome's ~70% browser share, one company's infrastructure intermediates the vast majority of how content is discovered, accessed, and displayed on the web — even as competitors crawl harder.

Microsoft's BingBot at 4.95% of bot traffic delivers only 3.40% of referrals — confirming the inefficiency the crawl-to-refer ratio (34.1:1) already showed. The reverse extreme is ByteDance: Bytespider's 10.5% share (now the #4 AI crawler per the May crawler report) generates only 3.25% of referrals, which is why its crawl-to-refer ratio jumped 3.5x across the quarter.

What Does the Full Bot Ecosystem Look Like?

According to Cloudflare Radar's crawler analytics (/radar/bots/crawlers/summary/CLIENT_TYPE endpoint), the breakdown of bot vs. human traffic has shifted notably since January:

Bots now clearly outnumber humans in the verified-traffic mix at 56.1% vs 43.9% — and the gap widened again this quarter as human share fell another -1.5 pp from last edition. The shift came primarily from non-AI bots (SEO scanners, page-preview bots, ad bots) growing +3.95 pp since January, with AI bots adding another +0.54 pp.

The bot category breakdown reveals the most consequential shift: AI Search nearly tripled to 6.60% of verified bot traffic since January.

Search engine crawlers fell -2.09 pp to 30.92% — continuing to cede share. AI crawlers held essentially flat at 20.5%, but the AI Search category nearly tripled, going from 2.67% to 6.60% since January. When you sum AI Crawler (20.5%) + AI Search (6.6%) you reach 27.1% of all verified bot traffic — within 3.9 pp of search engine crawlers, and on a clear trajectory to overtake them this year.

🎯 Key Takeaway: The bot ecosystem is undergoing the same crossover that the AI crawler report documented for AI training crawlers vs. mixed-purpose crawlers. AI-related categories (training + search) are now nearly indistinguishable in volume from traditional search indexing. If your bot policy treats "search bots" as the priority and "AI bots" as the exception, that mental model is two quarters out of date.

💡 Expert Insight from James Bennett:

The AI Search +3.69 pp jump is the metric I'd flag for any production team running a customer-facing site in 2026. At WebSearchAPI.ai, AI search crawlers (the bots that fetch a page to answer a real-time user question) are categorically different from training crawlers in how we treat them on our origin: training crawlers can be heavily rate-limited because they're not feeding live user requests, but throttling AI search crawlers means a real user gets a worse answer. Your CDN's bot policy needs to make this distinction. Block GPTBot but allow OAI-SearchBot. Block ClaudeBot but allow Claude-SearchBot. The traffic is asymmetric and so should your treatment of it be.

How Does Google's Browser Dominance Reinforce Its Search Monopoly?

Chrome controls 69.7% of all browser traffic globally — essentially flat versus 70.1% in January — according to Cloudflare Radar HTTP analytics (/radar/http/top/browser_family endpoint). Browser share is the one axis where Google's grip did not tighten this quarter: Chrome dipped slightly while Firefox ticked up. But with Google's search referral share expanding to 87.9%, the overall ecosystem is still tightening — the concentration is moving through search, not the browser.

The notable mover is Firefox, up +1.17 pp to 5.12% — its strongest reading in over a year, and the entire gain came at Chrome's and Safari's expense. Edge continued its slow climb (+0.25 pp), presumably driven by Microsoft's Copilot integration. Brave appeared in the top 8 at 0.03% — small, but worth tracking as the privacy-browser segment grows. None of this dents Google's distribution: Chrome and Safari both default to Google.

Chrome's default search engine is Google. Edge's default is Bing. Safari's default is Google (through a deal worth an estimated $20 billion annually, as revealed in the DOJ antitrust trial). This means approximately 85.5% of all browser traffic (Chrome + Safari) still defaults to Google search — down marginally from ~86.8% in January as Firefox gained, but a rounding-level change against Google's expanding referral dominance.

The data points still reinforce each other: Chrome funnels users to Google search, Google search generates referrals, and Googlebot crawls sites to index them for those searches. The DOJ antitrust ruling has not meaningfully altered the search numbers on the ground — Google's referral share has actually grown across Q1→Q2 2026, even as the browser layer loosened slightly.

Which Industries Receive the Most Crawler and Referral Traffic?

According to Cloudflare Radar crawler analytics (/radar/bots/crawlers/summary/VERTICAL endpoint), the distribution of crawler traffic by industry has shifted notably toward Shopping in the past quarter.

Shopping & General Merchandise is still the most-crawled vertical at 24.95% (+2.0 pp vs January), but it has cooled sharply from the prior window's 28% peak — the same -3 pp month-over-month retreat documented in the May AI Crawler Report. The Q1 retail-content land grab has clearly passed its high-water mark. The traffic didn't vanish; it redistributed toward News, Media & Publications (+0.8 pp to 9.17%) — consistent with the rise in real-time AI-search fetches, which favor fresh news and reference content over static product catalogs.

Computer and Electronics held roughly flat at 19.01%, and Internet & Telecom narrowed its gap to Shopping to under 3.1 pp — the tightest spread between the top two verticals on record.

If you run an e-commerce site, you're still in the single most-crawled vertical on the web — but the crawl pressure is no longer accelerating the way it was in Q1. I'd hold AI-crawler bandwidth budgets near current levels rather than extrapolating the early-Q1 surge, and watch whether news/media crawling keeps climbing as AI-search traffic grows.

💡 Expert Insight from James Bennett:

Shopping cooling back to 25% after spiking near 28% is a useful correction to track, because WebSearchAPI.ai's largest cohort of customers builds shopping agents (price-comparison, reviews aggregation, deal-finders). The Q1 surge had our customers shortening cache TTLs to keep pace with aggressive crawling; the May pullback suggests the recrawl frequency on e-commerce content is normalizing. What I'm watching now is the News/Media uptick — if AI-search fetches keep pushing crawl attention toward fresh editorial content, that's the next vertical where freshness expectations will ratchet up.

What Does AI Search Look Like Compared to Traditional Search?

The AI search category has nearly tripled since January — from 2.67% of verified bot traffic to 6.60% in May 2026. This is the single fastest-growing category in the entire bot ecosystem dataset, and it's now larger than several long-established categories like Webhooks, Aggregator, and Monitoring.

The composition keeps shifting. The May AI Crawler Report showed Anthropic's Claude-SearchBot tripling to 2.4% to become the largest dedicated AI search crawler on the web — overtaking OpenAI's OAI-SearchBot. Meanwhile Applebot's brief surge reversed (it fell to 7.0% and #7 in the May crawler data), so its role as a general AI crawler is smaller than it looked a month ago. OpenAI continues to run OAI-SearchBot for ChatGPT Search, and chatgpt.com referrals resumed growth to 0.25% this month.

The four most consequential AI search operators today:

Combined, the AI search referrer footprint (chatgpt.com plus the Gemini/Claude/Perplexity long tail) generates approximately 0.30% of all referrals in May 2026 — with chatgpt.com (0.25%) the clear majority and the only one rising. Tiny in absolute terms, but the bot-side category (6.6%) is now 24x larger than its referral footprint, the clearest signal of how much AI-search crawling still happens without sending a visit back.

📈 Case Study:

The AI search category barely existed 18 months ago. Today, AI Search bots represent 6.6% of verified bot traffic and grew +147% since January — the fastest-growing category in the entire bot dataset. If you're building AI applications that need real-time web data, understanding how these search APIs work is foundational — I break down the complete picture in my guide on what a web search API is and why it matters for AI agents.

💡 Expert Insight from James Bennett:

The +3.9 pp swing in AI Search is what I'd flag as the single most actionable trend in this report. At WebSearchAPI.ai, we re-evaluated our infrastructure costing model in Q1 because the volume of "fetch-this-page-for-a-user" traffic from AI search bots blew past our prior projections. We had budgeted AI search crawl growth at 50% per quarter; the reality came in near 147%. If you're modeling crawler load on your origin, the lesson is to err high on the AI-search side and ensure you have a clean separation between training-purpose user agents (which can be heavily rate-limited) and search-purpose user agents (which feed live user queries and shouldn't be throttled). Treating them the same is leaving real, addressable users at the door.

What Should Website Owners Take Away From This Updated Data?

Based on the May 2026 Cloudflare Radar data, here are the implications I'd prioritize for website owners and content publishers:

Google's referral monopoly is tightening, not loosening. At 90.9% of search engine referrals and 87.9% of all referrals (up from 81.6% in January), any meaningful traffic strategy still starts with Google. The DOJ antitrust ruling hasn't yet changed this reality on the ground — if anything, the gap widened as TikTok's referral share collapsed and has stayed collapsed.

TikTok has settled at a new, lower floor. TikTok's referral share fell from 10.6% to 3.2% over the quarter and then held flat this month — confirming this is structural, not a continuing slide. The drop is partly real engagement decline, partly a referrer-policy/in-app-browser change that strips identifiable referrer headers. Don't conclude users stopped clicking — conclude that platform-level changes made TikTok-driven traffic harder to attribute, and that the new ~3% level is now stable for planning.

ByteDance's crawl-to-refer trajectory is the most actionable change. Bytespider's crawl-to-refer ratio worsened 3.5x — from 2.6:1 to 9.1:1 — across the quarter, and in the May crawler data Bytespider became the #4 AI crawler on the web at 10.5%. ByteDance has multiplied its crawling load on origins while sending flat referral traffic back. If you previously left Bytespider unrate-limited because the crawl/refer math worked out, that math has decisively flipped.

AI search is the fastest-growing bot category on the web. AI Search jumped from 2.67% to 6.60% of verified bot traffic since January — a +147% relative gain. Make a clean policy distinction between AI training crawlers (GPTBot, ClaudeBot, Bytespider) and AI search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot). The first can be rate-limited; the second feeds real-time user queries and shouldn't be.

Anthropic and OpenAI's crawl-to-refer ratios keep improving. Anthropic improved 3.6x (43,214:1 → 11,992:1) and OpenAI improved to 1,057:1, with chatgpt.com referrals resuming growth. Both ratios are still extreme, but the trend matters: blocking these crawlers wholesale today means forfeiting what is now a non-zero referral channel that's becoming less imbalanced each quarter.

Plan for bots > humans on your origin. Verified bot traffic is now 56.1% of all client traffic — humans fell to 43.9%, a wider gap than last quarter. Your CDN configuration, rate limits, and bot-management policy are no longer "edge cases" — they're the primary throughput shaper for your site.

🎯 Key Takeaway: The story across Q1 2026 isn't Google losing its grip. It's the AI search ecosystem materializing as a real bot category — six times bigger than it was 18 months ago, with four distinct operators now generating measurable referrals. Website owners who built bot policies around "search engines vs. everything else" need a third bucket: "AI search crawlers" that need to be allowed but distinguished from training crawlers.

Build Smarter Search With Us

The data in this report shows how search and AI crawling are reshaping the web. Whether you're building AI applications that need real-time web data, or you want to understand how search engines and AI bots interact with your content, WebSearchAPI.ai gives you the tools to harness web intelligence through a single, fast API.

Ready to see it in action? Start building with WebSearchAPI.ai and get Google-grade results in minutes.

How I Analyzed This Data

This analysis uses data from Cloudflare Radar's crawler analytics (/radar/bots/crawlers/summary/* endpoints), bot traffic analysis (/radar/bots/summary/* endpoints), and HTTP traffic analytics (/radar/http/top/browser_family endpoint), which aggregate traffic patterns across Cloudflare's global network spanning 330 cities in 125+ countries. Cloudflare's network processes over 81 million HTTP requests per second, providing one of the most comprehensive views of global internet traffic available.

I queried referral traffic breakdowns by source, crawl-to-refer ratios by operator, browser family distribution, bot category composition, and industry-level crawler traffic patterns. The current data covers the rolling 28-day window of May 5 through June 2, 2026 (dateRange=28d), with month-over-month comparisons against the immediately preceding 28-day window (April 7 - May 5, 2026, via dateRange=28dControl). Quarter-over-quarter comparisons reference the original report's data window (January 9 - February 8, 2026).

The crawl-to-refer ratio measures how many crawl requests each operator makes relative to the referral traffic it generates. A ratio of 5:1 means the operator crawls 5 pages for every 1 referral it sends. Lower ratios indicate more efficient or generous referral behavior. All percentages represent share of identified referral, crawler, or HTTP request traffic — not share of total internet traffic.

⚠️ Methodology note: TikTok's measured referral share is affected by a March 2026 in-app-browser update that increases the use of noreferrer policies on outbound clicks. The 70% drop in measured TikTok referrals does not necessarily mean a 70% drop in TikTok-driven user traffic; it means the share of that traffic that is identifiable to Cloudflare via referrer headers has structurally fallen. Treat the 3.25% TikTok figure as a floor on identifiable TikTok-driven traffic, not a ceiling on total TikTok-driven traffic.

Data source: Cloudflare Radar — /radar/bots/crawlers/summary/*, /radar/bots/summary/*, and /radar/http/top/browser_family endpoints (radar.cloudflare.com), May 5 – June 2, 2026 vs. April 7 – May 5, 2026. Last updated: June 2, 2026.

Frequently Asked Questions

What percentage of web referral traffic does Google control?

According to May 2026 Cloudflare Radar data, Google generates 87.9% of all identified web referral traffic globally — up from 81.6% in January 2026. When isolating only search-engine-specific referrals (excluding non-search sources like TikTok), Google's share rises to approximately 90.9%. This makes Google the overwhelmingly dominant source of how people discover websites through search.

What happened to TikTok as a referral source?

TikTok's measured referral share fell from 10.59% in January 2026 to 3.25% in May 2026 — a 69% relative decline — and has now stabilized, holding flat month-over-month. The drop was driven by a combination of TikTok's late-February in-app-browser update (which keeps more users inside the TikTok app instead of opening external browsers), a broader move to noreferrer policies on outbound clicks, and possibly some real engagement decline. Bing holds the #2 referrer position at ~3.4%. Note that the 3.25% figure measures identifiable TikTok-driven traffic, not necessarily total TikTok user clicks.

What is a crawl-to-refer ratio and why does it matter?

A crawl-to-refer ratio measures how many pages a bot crawls for every referral (visitor) it sends back to websites. A ratio of 5:1 means the operator crawls 5 pages for every 1 visitor it returns. Lower ratios indicate more efficient or generous referral behavior. As of May 2026, DuckDuckGo has the best ratio at 1.6:1, while Anthropic's ClaudeBot has the worst at 11,992:1 (improved from 43,214:1 in January). The biggest worsening this quarter was ByteDance's, which jumped from 2.6:1 to 9.1:1 as Bytespider became the #4 AI crawler on the web.

How much referral traffic does ChatGPT send to websites?

As of May 2026, ChatGPT (chatgpt.com) generates 0.25% of all identified referral traffic — up from January 2026's 0.19%, and notably resuming growth after a flat reading the prior month. The uptick suggests ChatGPT Search is surfacing source links to users somewhat more often. The other AI-search referrers (gemini.google.com, claude.ai, perplexity.ai) remain in the sub-0.05% long tail.

Should I block AI crawlers from my website?

It depends on the crawler type, and the answer has gotten more nuanced in 2026. AI training crawlers like ClaudeBot (11,992:1 crawl-to-refer ratio) and GPTBot (1,057:1) consume bandwidth while returning minimal traffic — blocking these won't affect your search rankings. However, AI search bots like OAI-SearchBot, Claude-SearchBot, and PerplexityBot fetch pages in real time to answer specific user queries, and blocking these forfeits the user's chance to discover your content. The single most important policy distinction is between training-purpose and search-purpose AI crawlers. For a detailed breakdown of individual AI crawler market share and blocking strategies, see the Monthly AI Crawler Report.

How does Chrome's market share reinforce Google's search monopoly?

Chrome controls 69.7% of browser traffic in May 2026 — roughly flat versus 70.1% in January — and its default search engine is Google. Safari (15.8%) also defaults to Google through a deal worth an estimated $20 billion annually. Combined, approximately 85.5% of all browser traffic defaults to Google search, creating a self-reinforcing cycle: Chrome funnels users to Google search, which generates referrals, while Googlebot crawls sites to index them for those searches. (Firefox ticked up to 5.1% this quarter, the one soft spot in Google's browser distribution.)

How much of my website traffic is now bots vs humans?

According to Cloudflare Radar's May 2026 client-type data, verified bot traffic now represents 56.1% of all identified-client traffic — outnumbering humans (43.9%), and the gap widened again this quarter. The composition: 47.5% non-AI bots (search, SEO, ads, monitoring), 5.6% AI bots, and 3.0% mixed-purpose bots. Plan your CDN, rate-limiting, and bot-management policies on the assumption that bots are now the majority of your origin traffic.

Is the AI search category a real referral source yet?

It's small but growing fast. AI search bots represent 6.60% of all verified bot traffic in May 2026 — up from 2.67% in January, a +147% increase that makes it the fastest-growing category in the bot dataset. The combined referrer footprint of AI search domains is approximately 0.30% of all referrals, with chatgpt.com (0.25%) the majority. The volume is small, but the trajectory is the fastest of any bot category, and the bot-side activity is now 24x larger than the referrals it sends back.

Related Resources

• Monthly AI Crawler Report: May 2026 — Bytespider Surges to #4 as Applebot Reverses -- Detailed analysis of AI crawler market share, growth trends, and robots.txt directives, including the Q2 predictions tracker
• What is a Web Search API? Complete Guide for AI Agents and LLMs -- How web search APIs work and why they matter for AI applications
• Cloudflare Radar -- The primary data source for this analysis, providing global internet traffic insights
• Cloudflare Radar Bot & Crawler Insights -- Real-time bot traffic monitoring and crawler analysis
• Google DOJ Antitrust Trial Documents -- Court filings revealing Google's search distribution agreements

About the Author: I'm James Bennett, Lead Engineer at WebSearchAPI.ai, where I architect the core retrieval engine enabling LLMs and AI agents to access real-time, structured web data with over 99.9% uptime and sub-second query latency. With a background in distributed systems and search technologies, I've reduced AI hallucination rates by 45% through advanced ranking and content extraction pipelines for RAG systems. My expertise includes AI infrastructure, search technologies, large-scale data integration, and API architecture for real-time AI applications.

Credentials: B.Sc. Computer Science (University of Cambridge), M.Sc. Artificial Intelligence Systems (Imperial College London), Google Cloud Certified Professional Cloud Architect, AWS Certified Solutions Architect, Microsoft Azure AI Engineer, Certified Kubernetes Administrator, TensorFlow Developer Certificate.