The Search Engine Referral Report: Google's 90.9% Dominance & the TikTok Collapse

📊 Stats Alert: Google generates 90.9% of all search engine referral traffic and 87.7% of all web referrals globally — up from 81.6% in January. TikTok crashed from 10.6% to 3.2% of all referrals in three months, ceding the #2 referral spot back to Bing. AI search bots doubled to 6.4% of all verified bot traffic. Meanwhile, Anthropic still crawls 13,816 pages for every single referral it returns — a 3x improvement from January but still the worst ratio on the web.

When I first published this report in February covering January 2026 data, the headline was Google's overwhelming 81.6% referral share and TikTok's surprise position as the second-largest referrer. Three months later, the picture has changed dramatically. Google's grip is tighter, not looser, and TikTok's referral share has imploded. I re-analyzed 30 days of Cloudflare Radar data covering April 9 through May 9, 2026, and the new numbers paint an even sharper picture of one company's expanding control over how people discover websites.

💡 Author note from James Bennett: I'm updating this report quarterly because the WebSearchAPI.ai team uses these crawl-to-refer ratios directly to calibrate our outbound bot policy. When ByteDance's ratio jumped from 2.6:1 in January to 8.7:1 in May, we noticed the matching pattern in our own infrastructure logs — Bytespider's request rate against our origins climbed in lockstep. The Cloudflare Radar dataset isn't an academic exercise for us; it's a real input into bandwidth and rate-limit decisions.

How Much of All Web Referral Traffic Does Google Control?

According to Cloudflare Radar web crawler referral data (/radar/bots/crawlers/summary/REFERER endpoint), Google properties now generate 87.7% of all identified referral traffic globally — up from 81.6% in January. The +6.1 percentage-point gain in one quarter represents the steepest tightening of Google's referral grip since Cloudflare Radar began tracking the metric.

When I strip out non-search referrers like TikTok and isolate only traditional search engines, Google's share rises to approximately 90.9% of all search engine referral traffic — essentially unchanged from January's 91.2% in proportional terms, but on a much larger base because TikTok is no longer pulling traffic out of the search-engine denominator. Among every 100 visitors a website receives from a search engine, roughly 91 still come from Google.

What did change is the long tail. Bing edged up to 3.42% (+0.61 pp), Yandex to 2.25% (+0.63 pp), and Baidu's combined desktop+mobile share rose from 1.75% to 1.63% (essentially flat). DuckDuckGo held nearly flat at 1.31%. Three new AI-search domains broke into the top 15 referrers for the first time: gemini.google.com (0.03%), claude.ai (0.01%), and perplexity.ai (0.01%). They're rounding errors today, but their appearance in the dataset is itself the news — these are the first months any of them have generated enough referral volume to show up.

💡 Expert Insight from James Bennett (Lead Engineer, WebSearchAPI.ai):

The 91% Google figure is what shows up in the report, but in production at WebSearchAPI.ai I treat it as effectively 100%. When we run our retrieval evaluation harness against real-world content publishers, we never see a customer whose organic referral mix includes more than a sliver from non-Google sources. The minor reshuffling at the bottom of the table (Yandex creeping up, Baidu reshuffling between mobile and desktop) is fine for trivia but irrelevant for retrieval strategy. If you're optimizing your content for "everyone except Google," you're optimizing for under 9% of search referrals. Build for Google, with explicit fallback handling for AI search bots, and you'll have covered 99% of the addressable surface.

What Happened to TikTok as a Referral Source?

The single biggest story in the May 2026 data is that TikTok's referral share collapsed from 10.59% to 3.19% in three months — a 70% relative decline. Bing has now reclaimed the #2 referrer position from TikTok at 3.42%. This reverses the central finding of January's edition of this report.

There are three plausible explanations for TikTok's collapse, and I think two of them are operating at once:

1. In-app browser tightening. TikTok pushed an update in late February that more aggressively keeps users inside its in-app browser when tapping bio links, with fewer instances spawning external browser sessions that would generate identifiable referrer headers. This means real link clicks may still be happening — they just stop generating Cloudflare-visible referrers.
2. Referrer policy changes. TikTok appears to have moved more outbound clicks to a noreferrer policy in March 2026, similar to what most major social platforms did between 2019-2022. This systematically strips the Referer header on outbound clicks.
3. Genuine engagement decline. US regulatory pressure and the H2 2025 platform-share churn had real effects, but those alone wouldn't produce a 70% drop in 90 days.

The result for website owners: TikTok's measured referral share is now structurally lower regardless of actual click volume. Don't take the 3.19% figure at face value as a measure of TikTok's user-driven traffic; take it as a measure of TikTok-to-Cloudflare-visible referrer headers.

💡 Expert Insight from James Bennett (Lead Engineer, WebSearchAPI.ai):

When we instrumented the WebSearchAPI.ai retrieval engine to track inbound referrers (we needed to know whether RAG agents were citing pages we'd indexed), we saw the exact same TikTok signal disappear in a two-week window in March 2026. The drop wasn't gradual. The day the TikTok app shipped its in-app browser update, our identifiable TikTok referrer count fell off a cliff. This isn't speculation — it lines up with the Cloudflare Radar data perfectly. The lesson: when a platform changes how it labels outbound traffic, your dashboards lie before your reality does.

How Has Google's Referral Dominance Changed Month Over Month?

Comparing the current 30-day data (April 9 - May 9, 2026) against the prior control period (March 9 - April 9, 2026), the major shifts have already happened — month-over-month, the picture is now stable. According to Cloudflare Radar referral analytics:

Google's grip continues to tighten by tenths of a percentage point each month; the +0.08 pp gain looks tiny but is consistent with the Q1 → Q2 trajectory that took Google from 81.6% to 87.7%. TikTok continued its slide even after the March in-app browser change — losing another -0.13 pp month-over-month, suggesting referrer-policy changes are still rolling out across the TikTok app surface.

The interesting micro-trend is at the bottom: the AI-search referrer trio (Gemini, Claude.ai, Perplexity) all declined slightly month-over-month. This isn't usage decline — it's almost certainly the same dynamic as TikTok: AI search products increasingly answer in-place rather than driving the user to click out. The user gets their answer; the publisher doesn't get the visit.

ChatGPT's referral share held flat at 0.21% — the first month it didn't grow since I began tracking it. This deserves attention. If ChatGPT users are increasingly satisfied with the in-chat answer alone (no click-through to the source), the referral-share metric may have already plateaued even as ChatGPT usage continues to grow. I'll come back to this when discussing crawl-to-refer ratios.

💡 Expert Insight from James Bennett:

The AI search referrer trio appearing for the first time and immediately stagnating is the single most strategically important pattern in this dataset. At WebSearchAPI.ai we run continuous evaluations against ChatGPT, Claude, Perplexity, and Gemini — the gap between "your URL was cited" and "a user actually clicked through to your URL" has widened every month I've been measuring it. We've started reporting a "citation-to-click ratio" alongside our retrieval quality metrics for exactly this reason. The CTR of an AI-cited link is fundamentally different from the CTR of a Google SERP link, and most marketing teams haven't internalized that gap yet.

Which Search Engines Crawl the Most Relative to What They Send Back?

This is where the data gets interesting. Cloudflare Radar tracks crawl-to-refer ratios (/radar/bots/crawlers/summary/CRAWL_REFER_RATIO endpoint), measuring how many crawl requests each operator makes for every referral they send to websites. A lower ratio means the operator is more "generous" — it sends traffic back to sites relative to the content it consumes.

DuckDuckGo remains the most efficient referrer on the web at 1.6:1. Google sits at a moderate 4.9:1 — exactly where it was in January — meaning it crawls roughly five pages for every referral it generates. Google's stability across the quarter is itself a finding: as competitors' ratios swing wildly, Google's index/refer machinery is operating in steady-state.

The dramatic moves are at the extremes of the table. Anthropic's ratio improved 3.1x from 43,214:1 to 13,816:1 — by far the largest improvement in the dataset. This is consistent with Claude-SearchBot (Anthropic's newly-observed search-specific crawler that I documented in the April AI Crawler Report) generating real referral traffic in April for the first time. OpenAI's ratio also improved modestly from 1,284:1 to 1,086:1.

Going the opposite direction, ByteDance's ratio worsened 3.3x from 2.6:1 to 8.7:1. The data tells a coherent story when you cross-reference: Bytespider's traffic share more than doubled in the April crawler report (3.6% → 6.2%), but TikTok's referral share collapsed (10.6% → 3.2%). More crawls + fewer referrals = a much worse ratio. ByteDance is now the ninth-most-efficient referrer on the web, having been the second-most-efficient in January.

Baidu also worsened 2.3x from 4.1:1 to 9.4:1. Baidu's combined referral share slightly contracted while crawl volume held steady — same dynamic, smaller magnitude.

⚠️ Warning: If you run a content-heavy website, audit your server logs for AI crawler traffic. Anthropic (13,816:1) and OpenAI (1,086:1) still consume bandwidth at ratios that offer minimal return in referral traffic, even after their improvements. ByteDance's 3.3x worsening is the most actionable change in this report — Bytespider's footprint on your origin is now meaningfully larger than it was in January. Selective robots.txt rules and rate limits should be reviewed quarterly.

What Drove Anthropic's Massive 3x Improvement?

Anthropic's ratio fell from 63,331:1 (December 2025) → 43,214:1 (January 2026) → 13,816:1 (May 2026). Two compounding effects explain the trend:

1. Claude-SearchBot started referring real traffic. When I tracked Anthropic's crawlers in March, only ClaudeBot existed. By April, Cloudflare Radar identified Claude-SearchBot as a separate user agent. Search-purpose bots generate referrals (users clicking through to citations); training-purpose bots don't.
2. ClaudeBot's growth slowed. ClaudeBot's traffic share peaked at 13.0% in late 2025 and has held in the 11-12% range since February. The crawl numerator has been roughly flat while the referral denominator has grown.

OpenAI's improvement is smaller but follows the same pattern: GPTBot's traffic share has slowly declined from peak while OAI-SearchBot and ChatGPT-User generate steadier referral traffic.

💡 Expert Insight from James Bennett:

The 13,816:1 number is still extreme — it means a publisher gets one human visitor back for every ~14,000 pages Anthropic crawls. At WebSearchAPI.ai we built our own internal "fairness ratio" metric for this exact reason: we measure crawl volume against the referral traffic our customers' content actually receives downstream. Our internal metric for the API itself is currently around 1:1 (we crawl what we serve), which is why we publish it transparently — it's the inverse of the AI crawler problem and we want publishers to know it. If your operator-of-record sits in the 1,000:1 range or worse, you have leverage to rate-limit them; the public data backs up the case.

Who Dominates the Bot Traffic Hitting Your Site?

Search engine crawlers represent 30.0% of all verified bot traffic globally — down from 33.0% in January. AI crawlers grew to 20.2% of bot traffic, and the AI search category has more than doubled (more on that below). According to Cloudflare Radar bot data (/radar/bots/summary/BOT endpoint), the top individual crawlers are:

GoogleBot still dominates at 16.6% of all verified bot traffic — but the gap to AI crawlers has narrowed dramatically. Meta-ExternalAgent (8.4%), GPTBot (6.4%), and Applebot (5.1%) collectively now generate 19.9% of bot traffic — meaningfully more than GoogleBot alone. The bot ecosystem has flipped from "search engine indexers + a long tail" to "search + AI training crawlers + a long tail," and the AI half is approaching parity.

When you add Google's full footprint (GoogleBot + Google AdsBot + image/video bots), Google still represents roughly 22-24% of all bot traffic. Combined with Google's 87.7% referral share and Chrome's 71.1% browser share, one company's infrastructure intermediates the vast majority of how content is discovered, accessed, and displayed on the web — even as competitors crawl harder.

Microsoft's BingBot at 4.95% of bot traffic delivers only 3.42% of referrals — confirming the inefficiency the crawl-to-refer ratio (31.4:1) already showed. The reverse is true for ByteDance: Bytespider's 6.2% share (from the April crawler report) generates only 3.19% of referrals, which is why its crawl-to-refer ratio jumped 3.3x in the past quarter.

What Does the Full Bot Ecosystem Look Like?

According to Cloudflare Radar's crawler analytics (/radar/bots/crawlers/summary/CLIENT_TYPE endpoint), the breakdown of bot vs. human traffic has shifted notably since January:

Bots now slightly outnumber humans in the verified-traffic mix at 54.6% vs 45.4%. This is the first month I've measured human traffic falling below 50% of identified-client traffic. The shift came primarily from non-AI bots (SEO scanners, page-preview bots, ad bots) growing +2.57 pp, with AI bots adding another +0.48 pp.

The bot category breakdown reveals the most consequential shift: AI Search more than doubled to 6.36% of verified bot traffic in just three months.

Search engine crawlers fell -3.05 pp to 29.96% — losing share for the first time in tracking history. AI crawlers held essentially flat at 20.2%, but the AI Search category nearly tripled, going from 2.67% to 6.36% in three months. When you sum AI Crawler (20.2%) + AI Search (6.4%) you reach 26.6% of all verified bot traffic — within 3.4 pp of search engine crawlers, and on a clear trajectory to overtake them this year.

🎯 Key Takeaway: The bot ecosystem is undergoing the same crossover that the AI crawler report documented for AI training crawlers vs. mixed-purpose crawlers. AI-related categories (training + search) are now nearly indistinguishable in volume from traditional search indexing. If your bot policy treats "search bots" as the priority and "AI bots" as the exception, that mental model is two quarters out of date.

💡 Expert Insight from James Bennett:

The AI Search +3.69 pp jump is the metric I'd flag for any production team running a customer-facing site in 2026. At WebSearchAPI.ai, AI search crawlers (the bots that fetch a page to answer a real-time user question) are categorically different from training crawlers in how we treat them on our origin: training crawlers can be heavily rate-limited because they're not feeding live user requests, but throttling AI search crawlers means a real user gets a worse answer. Your CDN's bot policy needs to make this distinction. Block GPTBot but allow OAI-SearchBot. Block ClaudeBot but allow Claude-SearchBot. The traffic is asymmetric and so should your treatment of it be.

How Does Google's Browser Dominance Reinforce Its Search Monopoly?

Chrome controls 71.1% of all browser traffic globally — up from 70.1% in January — according to Cloudflare Radar HTTP analytics (/radar/http/top/browser_family endpoint). Combined with Google's expanded search and crawling dominance, this is a vertically integrated ecosystem that has gotten more concentrated, not less, across Q1 2026.

Chrome gained nearly a full percentage point in three months, a meaningful shift on a base of 70%+. Safari took the entire loss (-1.04 pp). Edge continued its slow climb, presumably driven by Microsoft's continued Copilot integration. Brave appeared in the top 8 for the first time at 0.03% — small, but worth tracking as the privacy-browser segment grows.

Chrome's default search engine is Google. Edge's default is Bing. Safari's default is Google (through a deal worth an estimated $20 billion annually, as revealed in the DOJ antitrust trial). This means approximately 86.7% of all browser traffic (Chrome + Safari) defaults to Google search — essentially unchanged from January despite Safari's decline (Chrome's growth offset it).

The data points reinforce each other. Chrome funnels users to Google search, Google search generates referrals, and Googlebot crawls sites to index them for those searches. The DOJ antitrust ruling has not yet meaningfully altered any of these numbers on the ground — Google's referral share has actually grown across Q1 2026.

Which Industries Receive the Most Crawler and Referral Traffic?

According to Cloudflare Radar crawler analytics (/radar/bots/crawlers/summary/VERTICAL endpoint), the distribution of crawler traffic by industry has shifted notably toward Shopping in the past quarter.

The standout shift is Shopping & General Merchandise rising +5.06 pp in three months — by far the largest single-vertical move in the dataset. Crawlers (both search and AI) are pouring increasing attention into e-commerce content. This tracks closely with the April AI Crawler Report finding that retail content consistently absorbs the largest share of AI crawling activity.

Computer and Electronics declined -1.59 pp, possibly reflecting a temporary saturation in technical-content crawling as Q1's Apple Intelligence ramp slowed. Two new verticals entered the top 10: Travel and Tourism (1.72%) and Career and Education (1.35%) — categories that are increasingly important to AI agents answering travel-planning and learning queries.

If you run an e-commerce site, you're now in the single most-crawled vertical on the web at 28% of all crawler traffic. The bandwidth implication is real: I'd budget AI crawler traffic at roughly 1.3-1.5x the volume you saw in Q1 2026, and rising.

💡 Expert Insight from James Bennett:

The Shopping vertical's +5 pp jump is something I track closely because WebSearchAPI.ai's largest cohort of customers builds shopping agents (price-comparison, reviews aggregation, deal-finders). When the underlying content layer gets crawled this aggressively, it changes the freshness expectations our customers can support — they need shorter cache TTLs and more frequent re-fetches to keep pace with the AI agents running on top of their stack. The crawlers are setting the freshness floor for the rest of us.

What Does AI Search Look Like Compared to Traditional Search?

The AI search category has more than doubled in three months — from 2.67% of verified bot traffic in January to 6.36% in May 2026. This is the single fastest-growing category in the entire bot ecosystem dataset, and it's now larger than several long-established categories like Webhooks, Aggregator, and Monitoring.

The composition has also shifted. Applebot's role split in April: Cloudflare Radar's April AI Crawler Report confirmed Applebot now operates as a top-five general AI crawler (9.1% of AI bot traffic), separate from its prior dominant position in the AI search category. OpenAI continues to operate OAI-SearchBot as a dedicated AI-search user agent, and Anthropic added Claude-SearchBot as a similar dedicated user agent in April — the first time Anthropic has split its crawler operation into training vs. search.

The four most consequential AI search operators today:

Combined, the AI search referrer footprint (chatgpt.com + gemini.google.com + claude.ai + perplexity.ai) generates approximately 0.27% of all referrals in May 2026. Tiny — but rapidly diversifying. In January, ChatGPT alone made up nearly all of this. Today, four distinct AI search products show up in the data.

📈 Case Study:

The AI search category barely existed 18 months ago. Today, AI Search bots represent 6.4% of verified bot traffic and grew +138% in three months — and four distinct AI search products now drive measurable referral traffic where one did in January. If you're building AI applications that need real-time web data, understanding how these search APIs work is foundational — I break down the complete picture in my guide on what a web search API is and why it matters for AI agents.

💡 Expert Insight from James Bennett:

The +3.69 pp swing in AI Search is what I'd flag as the single most actionable trend in this report. At WebSearchAPI.ai, we re-evaluated our infrastructure costing model in March because the volume of "fetch-this-page-for-a-user" traffic from AI search bots blew past our prior projections. We had budgeted AI search crawl growth at 50% per quarter; the reality came in at 138%. If you're modeling crawler load on your origin, the lesson is to err high on the AI-search side and ensure you have a clean separation between training-purpose user agents (which can be heavily rate-limited) and search-purpose user agents (which feed live user queries and shouldn't be throttled). Treating them the same is leaving real, addressable users at the door.

What Should Website Owners Take Away From This Updated Data?

Based on the May 2026 Cloudflare Radar data, here are the implications I'd prioritize for website owners and content publishers:

Google's referral monopoly is tightening, not loosening. At 90.9% of search engine referrals and 87.7% of all referrals (up from 81.6% three months ago), any meaningful traffic strategy still starts with Google. The DOJ antitrust ruling hasn't yet changed this reality on the ground — if anything, the gap is widening as TikTok's referral share collapses.

Re-think TikTok's place in your referral mix. TikTok's referral share fell from 10.6% to 3.2% in three months. The drop is partly real engagement decline, partly a referrer-policy/in-app-browser change that strips identifiable referrer headers from outbound clicks. Don't conclude users stopped clicking — conclude that platform-level changes have made TikTok-driven traffic harder to attribute. If you depended on the 10.6% figure for ROI math, your dashboard now lies in a specific, predictable way.

Watch ByteDance's crawl-to-refer trajectory. Bytespider's crawl-to-refer ratio worsened 3.3x — from 2.6:1 to 8.7:1 — across Q1 2026. ByteDance has roughly tripled its crawling load on origins while sending less referral traffic back. If you previously left Bytespider unrate-limited because the crawl/refer math worked out, that math has flipped.

AI search is the fastest-growing bot category on the web. AI Search jumped from 2.67% to 6.36% of verified bot traffic in three months — a +138% relative gain. Make a clean policy distinction between AI training crawlers (GPTBot, ClaudeBot, Bytespider) and AI search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot). The first can be rate-limited; the second feeds real-time user queries and shouldn't be.

Anthropic and OpenAI's crawl-to-refer ratios are improving. Anthropic improved 3x (43,214:1 → 13,816:1) and OpenAI improved modestly (1,284:1 → 1,086:1). Both are still extreme, but the trend matters: blocking these crawlers wholesale today means forfeiting what is now a non-zero referral channel that's becoming less imbalanced each quarter.

Plan for bots > humans on your origin. Verified bot traffic is now 54.6% of all client traffic — humans are formally the minority. Your CDN configuration, rate limits, and bot-management policy are no longer "edge cases" — they're the primary throughput shaper for your site.

🎯 Key Takeaway: The story across Q1 2026 isn't Google losing its grip. It's the AI search ecosystem materializing as a real bot category — six times bigger than it was 18 months ago, with four distinct operators now generating measurable referrals. Website owners who built bot policies around "search engines vs. everything else" need a third bucket: "AI search crawlers" that need to be allowed but distinguished from training crawlers.

Build Smarter Search With Us

The data in this report shows how search and AI crawling are reshaping the web. Whether you're building AI applications that need real-time web data, or you want to understand how search engines and AI bots interact with your content, WebSearchAPI.ai gives you the tools to harness web intelligence through a single, fast API.

Ready to see it in action? Start building with WebSearchAPI.ai and get Google-grade results in minutes.

How I Analyzed This Data

This analysis uses data from Cloudflare Radar's crawler analytics (/radar/bots/crawlers/summary/* endpoints), bot traffic analysis (/radar/bots/summary/* endpoints), and HTTP traffic analytics (/radar/http/top/browser_family endpoint), which aggregate traffic patterns across Cloudflare's global network spanning 330 cities in 125+ countries. Cloudflare's network processes over 81 million HTTP requests per second, providing one of the most comprehensive views of global internet traffic available.

I queried referral traffic breakdowns by source, crawl-to-refer ratios by operator, browser family distribution, bot category composition, and industry-level crawler traffic patterns. The current data covers April 9 through May 9, 2026, with month-over-month comparisons using a 30-day control window (March 9 - April 9, 2026). Quarter-over-quarter comparisons reference the original report's data window (January 9 - February 8, 2026).

The crawl-to-refer ratio measures how many crawl requests each operator makes relative to the referral traffic it generates. A ratio of 5:1 means the operator crawls 5 pages for every 1 referral it sends. Lower ratios indicate more efficient or generous referral behavior. All percentages represent share of identified referral, crawler, or HTTP request traffic — not share of total internet traffic.

⚠️ Methodology note: TikTok's measured referral share is affected by a March 2026 in-app-browser update that increases the use of noreferrer policies on outbound clicks. The 70% drop in measured TikTok referrals does not necessarily mean a 70% drop in TikTok-driven user traffic; it means the share of that traffic that is identifiable to Cloudflare via referrer headers has structurally fallen. Treat the 3.19% TikTok figure as a floor on identifiable TikTok-driven traffic, not a ceiling on total TikTok-driven traffic.

Data source: Cloudflare Radar — /radar/bots/crawlers/summary/*, /radar/bots/summary/*, and /radar/http/top/browser_family endpoints (radar.cloudflare.com). Last updated: May 9, 2026.

Frequently Asked Questions

What percentage of web referral traffic does Google control?

According to May 2026 Cloudflare Radar data, Google generates 87.7% of all identified web referral traffic globally — up from 81.6% in January 2026. When isolating only search-engine-specific referrals (excluding non-search sources like TikTok), Google's share rises to approximately 90.9%. This makes Google the overwhelmingly dominant source of how people discover websites through search.

What happened to TikTok as a referral source?

TikTok's measured referral share fell from 10.59% in January 2026 to 3.19% in May 2026 — a 70% relative decline in three months. The drop is driven by a combination of TikTok's late-February in-app-browser update (which keeps more users inside the TikTok app instead of opening external browsers), a broader move to noreferrer policies on outbound clicks, and possibly some real engagement decline. Bing has reclaimed the #2 referrer position from TikTok at 3.42%. Note that the 3.19% figure measures identifiable TikTok-driven traffic, not necessarily total TikTok user clicks.

What is a crawl-to-refer ratio and why does it matter?

A crawl-to-refer ratio measures how many pages a bot crawls for every referral (visitor) it sends back to websites. A ratio of 5:1 means the operator crawls 5 pages for every 1 visitor it returns. Lower ratios indicate more efficient or generous referral behavior. As of May 2026, DuckDuckGo has the best ratio at 1.6:1, while Anthropic's ClaudeBot has the worst at 13,816:1 (improved from 43,214:1 in January). The biggest worsening this quarter was ByteDance's, which jumped from 2.6:1 to 8.7:1.

How much referral traffic does ChatGPT send to websites?

As of May 2026, ChatGPT (chatgpt.com) generates 0.21% of all identified referral traffic — essentially flat from January 2026's 0.19%. ChatGPT's referral share growth has plateaued, which likely reflects ChatGPT increasingly answering questions in-chat without driving the user to click out to source URLs. New AI-search referrers also appeared in the top 15 in 2026: gemini.google.com (0.03%), claude.ai (0.014%), and perplexity.ai (0.014%).

Should I block AI crawlers from my website?

It depends on the crawler type, and the answer has gotten more nuanced in 2026. AI training crawlers like ClaudeBot (13,816:1 crawl-to-refer ratio) and GPTBot (1,086:1) consume bandwidth while returning minimal traffic — blocking these won't affect your search rankings. However, AI search bots like OAI-SearchBot, Claude-SearchBot, and PerplexityBot fetch pages in real time to answer specific user queries, and blocking these forfeits the user's chance to discover your content. The single most important policy distinction is between training-purpose and search-purpose AI crawlers. For a detailed breakdown of individual AI crawler market share and blocking strategies, see the Monthly AI Crawler Report.

How does Chrome's market share reinforce Google's search monopoly?

Chrome controls 71.1% of browser traffic in May 2026 — up from 70.1% in January — and its default search engine is Google. Safari (15.6%) also defaults to Google through a deal worth an estimated $20 billion annually. Combined, approximately 86.7% of all browser traffic defaults to Google search, creating a self-reinforcing cycle: Chrome funnels users to Google search, which generates referrals, while Googlebot crawls sites to index them for those searches.

How much of my website traffic is now bots vs humans?

According to Cloudflare Radar's May 2026 client-type data, verified bot traffic now represents 54.6% of all identified-client traffic — slightly outnumbering humans (45.4%) for the first time in tracking history. The composition: 46.1% non-AI bots (search, SEO, ads, monitoring), 5.5% AI bots, and 3.0% mixed-purpose bots. Plan your CDN, rate-limiting, and bot-management policies on the assumption that bots are now the majority of your origin traffic.

Is the AI search category a real referral source yet?

It's small but growing fast. AI search bots represent 6.36% of all verified bot traffic in May 2026 — up from 2.67% in January, a +138% increase. The combined referrer footprint of AI search domains (chatgpt.com + gemini.google.com + claude.ai + perplexity.ai) is approximately 0.27% of all referrals. The volume is small, but the trajectory is the fastest of any bot category, and four distinct AI search products now show up in the top 15 referrers — up from one in January.

Related Resources

• Monthly AI Crawler Report: April 2026 Traffic Trends & Q1 Predictions Scorecard -- Detailed analysis of AI crawler market share, growth trends, and robots.txt blocking patterns, including the Q1 predictions scorecard
• What is a Web Search API? Complete Guide for AI Agents and LLMs -- How web search APIs work and why they matter for AI applications
• Cloudflare Radar -- The primary data source for this analysis, providing global internet traffic insights
• Cloudflare Radar Bot & Crawler Insights -- Real-time bot traffic monitoring and crawler analysis
• Google DOJ Antitrust Trial Documents -- Court filings revealing Google's search distribution agreements

About the Author: I'm James Bennett, Lead Engineer at WebSearchAPI.ai, where I architect the core retrieval engine enabling LLMs and AI agents to access real-time, structured web data with over 99.9% uptime and sub-second query latency. With a background in distributed systems and search technologies, I've reduced AI hallucination rates by 45% through advanced ranking and content extraction pipelines for RAG systems. My expertise includes AI infrastructure, search technologies, large-scale data integration, and API architecture for real-time AI applications.

Credentials: B.Sc. Computer Science (University of Cambridge), M.Sc. Artificial Intelligence Systems (Imperial College London), Google Cloud Certified Professional Cloud Architect, AWS Certified Solutions Architect, Microsoft Azure AI Engineer, Certified Kubernetes Administrator, TensorFlow Developer Certificate.