AI search is hurting publisher traffic - badly
And secret scraping is skyrocketing
Analysis shows a massive fall in traffic to publishers’ websites since the advent of AI search.
A new report has exposed the stark reality behind AI search engine promises, revealing a plunge in referral traffic for publishers and a surge in covert website scraping by AI giants like OpenAI and Perplexity.
The data, compiled by content licensing platform TollBit and shared with Forbes, presents a grim picture for news websites and blogs reliant on online traffic.
Contrary to claims that AI-powered search engines would drive readership, TollBit's analysis of 160 websites, including national and local news outlets, tech blogs, and shopping sites, shows a 96% reduction in referral traffic compared to traditional Google search.
This drop coincides with a doubling of website scraping by AI developers in recent months.
The report, which examined data from the fourth quarter of 2024, found that AI companies such as OpenAI, Perplexity and Meta averaged 2 million scraping incidents, with each page being accessed approximately seven times.
"We are seeing an influx of bots that are hammering these sites every time a user asks a question," TollBit CEO Toshit Panigrahi told Forbes.
"The amount of demand for publisher content is nontrivial."
TollBit gathered data from its platform's users, offering a direct window into scraping and traffic patterns.
While OpenAI declined to comment and Meta did not respond to requests, a Perplexity spokesperson, while not addressing the specific claims, asserted that the company respects "robots.txt" directives, which dictate website crawling permissions.
The findings corroborate earlier predictions by research firm Gartner, which forecasted a 25% decline in traditional search engine traffic by 2026 due to AI chatbots.
Edtech company Chegg, which recently sued Google for alleged copyright infringement and traffic theft through AI summaries, has already experienced a 49% year-over-year traffic drop in January, a sharp acceleration from the 8% decline seen after Google's AI summary launch.
Chegg CEO Nathan Schultz, contemplating going private or seeking acquisition, decried the breach of the "social contract" between search engines and publishers.
Ian Crosby, Chegg's legal representative, warned of an "AI slurry" if publishers are forced out of business.
However, Google dismissed Chegg's lawsuit as "meritless," claiming its AI search service increases traffic diversity.
Travel booking sites like Kayak and TripAdvisor have also expressed concerns about Google's AI overviews, while news publishers have launched legal battles against OpenAI and Perplexity for alleged intellectual property infringement.
A particular revelation from the TollBit report, which deserves highlighting, is Perplexity's continued referral traffic to sites it has been blocked from accessing, suggesting covert scraping using unidentified web crawlers. In one instance, the AI startup scraped a website 500 times but sent over 10,000 referrals.
Perplexity has faced prior accusations of republishing paywalled content without proper attribution, including from Forbes, CNBC and Bloomberg.
The report highlighted the difficulty publishers face in identifying AI scraping bots, as many developers do not properly disclose their user agents. Even Google's bots, used for multiple purposes, create challenges for publishers attempting to manage access.
Panigrahi said the server costs publishers incur due to the sheer volume of AI bot visits are escalating, a problem exacerbated by the rise of AI research agents that autonomously scrape numerous websites.
There is an urgent need for sustainable economic models for publishers in the age of AI, according to the report.