As AI disrupts the web economy, can publishers and brands take back control?

New Delhi: While we are enjoying the technological leaps AI is making every day, including the quick summaries of content from our favourite websites and getting tasks done in the blink of an eye, a disturbance is reshaping the forces that underpin the open web.

Dave Bouskill and Debra Corbeil, creators of the renowned travel blog The Planet D, reported a 90% drop in traffic during Gemini’s initial rollout. The couple had to lay off staff and ultimately shut down their website. “I feel betrayed,” Bouskill said. “We built something for 16 years, and it was gone in six months.”

This is just one anecdote but no matter how you look at the data, the growth of Artificial Intelligence- driven search and AI agents is upending the web’s business model as we know it, and, in the long term, that spells doom for the thriving ecosystem of content development and free distribution on the internet.

Today’s web economy can be described as 

Search engine crawls websites >> User searches for information>> Search Engine sends user to publisher and brand website>> Website shows their content and earns revenue through advertising

This is rapidly changing.

• AI-driven search is causing a substantial average traffic decline for publishers of 20–60%. 
• AI crawler activity is significantly outpacing human visits, with crawl-to-visit ratios soaring from 2:1 for search crawlers to 250:1 and higher for AI crawlers. Matthew Prince, Cloudflare CEO
• For the first time, bot traffic, largely AI bot-driven, exceeded human traffic in 2024, constituting over 51% of total web traffic.

This means that over time, the new web economy model will evolve into: 

AI engine crawls content >> Users deploy AI agents for information>> AI Driven Results (AIDR) present publisher and brand content to users >> No human visits to websites >> Zero revenue for open web content creators 

This is not just a publisher's problem. As users shift to AI agent-managed interfaces, brand reputations and messaging are also at the mercy of summarised results shown by AI agents.

This, in turn, creates a doom cycle for LLMs and AI agents. Less content on the open internet means less raw material for training and even less new content to generate useful information for their users, thus killing the value of their outputs. LLMs and AI agents need to ensure that content development on the open and free internet continues to thrive.

Now is the time to prepare for the imminent end of the old web economy and usher in a new web economy by building for content monetisation and distribution in an AI-driven world.

Publishers can take control of their monetisation and brands their identity when integrating with AI agents. It will require publishers and brands to take control of how their content is accessed and used in a three-step process:

1. Create scarcity of content by managing and controlling access to bot traffic using robots.txt and web application firewall (WAF) management
2. Develop LLM-friendly content discovery by providing content access rules and content meta data in LLM-friendly mechanisms: Meta data, or LLMs.txt
3. Monetise/ distribute content via cost per crawl (CPCr) APIs and/or direct LLM integration via APIs. 

IAB Tech Lab released a proposed standards framework for Publisher and Brand LLM integration that aims to guide the industry on this new journey and execute the above-mentioned three steps at scale.

These new tools would enable content owners to provide more relevant content to AI agents and also understand how and where their content is being used. The API proposals also allow for dynamically pricing content based on demand, enabling monetisation in this new Agentic Web Economy.

As the digital advertising ecosystem evolves, we need to evolve with it. We’re eager to hear more from the industry and standardise the new web economy model. The time is now. To investigate and to lean into new technology and new monetisation models. We can’t do this alone.