This is a part of a bigger opinion piece. Read the full text on the IPXO blog.
When Did You Last Visit Google?
I cannot remember the last time I typed a query into google.com and waited for a page of blue links.
That is not a trivial observation. For most of the past twenty-five years, the search engine was the front door to the internet. You had a question, you went to Google, and you navigated through results to find an answer. The entire architecture of the modern web was built around that behavior: pages optimized to rank in search results, advertising sold against search intent, publishing businesses built on the traffic that flowed through those results pages.
That behavior is changing – not gradually, and not at the margins. The query still happens. But increasingly, it goes to an AI tool that synthesizes an answer directly, without requiring the user to navigate anywhere. The front door has moved. And almost everything built behind the old front door is now in a more complicated position than the people running those businesses have fully reckoned with.
The Attention Interception Model
Advertisers did not pay for content. They never did, not really. They paid for human attention that happened to be in the vicinity of the content. The banner ad, the pre-roll video, the sponsored search result: all are mechanisms for intercepting a human being at the moment they navigate toward information and inserting a commercial message into that journey.
Google built a business worth trillions of dollars on that insight. Publishers created content to attract human navigators. Platforms aggregated those navigators at scale. Advertisers paid to reach them at the moment of highest intent. The open web was built on the assumption that this model would continue to function indefinitely.
It worked as long as one assumption held: that humans would keep navigating.
AI does not navigate. It arrives. When a user queries an AI tool, the answer comes to them. There is no page visit, no scroll through results, no moment of a human eye moving past an advertisement on the way to information. The interception point the entire attention economy was built around simply does not exist in that interaction.
The numbers confirm what behavior already suggests. As of 2025, 60% of Google searches end without a single click to any website. For news-related queries, the figure reaches 69%. When users do receive AI-generated answers with cited sources, only 1% click through to those sources. The interception model is not under pressure. For most queries, it is already broken.
Publisher Damage Is Already Showing Up in Hard Numbers
Google’s global search market share fell below 90% for the first time in a decade in late 2024, dropping from a peak of 92.9% in 2023 to 89.54% by July 2025. The publisher damage is not theoretical. Forbes experienced a 50% year-on-year decline in traffic in July 2025. CNN’s monthly visits fell from approximately 440 million to around 315 million in the same period. Business Insider, The Huffington Post, and NBC News all saw declines of 40% or more. The median publisher recorded a 10% year-on-year decline in the first half of 2025.
These are not struggling niche sites. They are among the most established digital publishers in the world. The shift is not approaching – it is already extracting its cost.
Google Was Always a Bot. The Difference Is That It Paid Its Way.
Here is something the industry rarely says plainly: Google is a bot. Googlebot is one of the largest automated crawlers on the internet, systematically extracting content from every publicly accessible page it can reach. The difference between Google and the AI crawlers now consuming the web is not one of kind. It is one of exchange.
The bargain Google maintained with publishers was imperfect, but it was a bargain. Google crawled your content and, in return, sent human visitors back to your site. Those visitors funded the next piece of content. The crawler and the publisher were uneasily engaged in a value exchange.
Cloudflare’s June 2025 data quantifies precisely where that exchange broke down. Google’s crawler accessed websites 14 times for every referral it sent back – not generous, but a functioning cycle. OpenAI’s crawler scraped websites 1,700 times per referral. Anthropic’s crawler reached 73,000 crawls per referral. The exchange did not erode. It collapsed entirely. AI crawlers are consuming the web’s content at a scale that dwarfs anything Google built, while returning a fraction of the traffic Google’s much-criticized model provided.
Google was not a neutral actor in the open web ecosystem. But it was a participant. AI crawlers, as currently structured, are not participants. They are extractors.
Cloudflare’s Pay-Per-Crawl: The First Infrastructure for a New Bargain
The most significant practical development came on July 1, 2025, when Cloudflare – which handles traffic for approximately 20% of the web – launched Pay-Per-Crawl and set blocking AI crawlers as the default for all new domains. The underlying logic was straightforward: if AI crawlers are going to consume content at an industrial scale, publishers should have the tools to charge for that access rather than simply accept or block it.
Using the HTTP 402 status code, publishers can now require AI crawlers to present payment before content is served. Cloudflare verifies crawler identity using cryptographic methods to prevent spoofing and distributes payments to publishers. Major publishers – including Condé Nast, TIME, The Associated Press, The Atlantic, and Fortune – signed on immediately. On an average day, Cloudflare customers already send over one billion HTTP 402 responses to crawlers. The market is signaling that the free extraction model is not being passively accepted.
Everyone Should Be a Winner. Here Is How That Works.
Framing this as a conflict between AI companies and content creators treats a structural problem as a zero-sum dispute. The reality is that the current model is harmful to everyone in the long run – including the AI companies extracting content from it.
AI systems that train on freely scraped content are undermining the economic conditions that produce the content they depend on. As publishers lose advertising revenue and cut editorial investment, the quality and volume of new human-generated content declines. As that declines, the models degrade. As the models degrade, the AI products built on them lose competitive differentiation. The extraction model is self-defeating at scale.
A model where everyone wins looks roughly like this. AI crawlers operate with declared identity and purpose, so publishers can make informed decisions about access. Content acquisition becomes a commercial transaction – not an extraction – with micropayment pricing that lets small publishers participate alongside large ones. And users who have shifted to AI-native information retrieval can trust that the content those tools synthesize is produced by creators with a sustainable economic reason to keep creating it.
The analogy is not hard to find. The fishing industry learned, after generations of overfishing, that sustainable yield requires investing in the conditions that produce fish, not simply maximizing extraction until the stock collapses. The content economy is facing the same choice now, compressed into a much shorter time window.
Cloudflare’s Pay-Per-Crawl is the first infrastructure step toward a legitimate AI content economy. It will not be the last. The question is whether the AI industry takes it seriously – or treats the current window of free extraction as something to be extended as long as possible. The answer will determine whether the open web survives the AI era as a commercially viable ecosystem, or whether it hollows out into something that looks like a web but no longer functions like one.
