The rise of generative AI has changed the landscape of bot traffic, with automated users of informational web content now outnumbering humans, according to new research from F5.
The 2025 Advanced Persistent Bots Report analysed 207 billion web and API transactions from November 2023 to September 2024. It examined records from customers with existing bot defenses in place, demonstrating how automated traffic operators behave when confronted with countermeasures.
The report found that 50.04% of page requests for web content1 were from automated sources, compared to 22.3% of search requests on web and 21.5% of ‘add to cart’ transactions. It suggests significant growth in the kind of web scrapers used by LLM providers such as OpenAI, Anthropic and Perplexity, and the persistence of these bots to continue sending requests when blocked.
In total, 21.22 billion of the transactions monitored (10.2%) came from a variety of automated sources, some of them benign, but 10 billion (4.8%) comprising malicious bot traffic.
“When we break down automated traffic by the flow (function) it is targeting, Content is now well ahead of any other area on web platforms,” said David Warburton, Director of Threat Research Team, F5 Labs.
“For years, bot traffic has primarily been targeted at Search flows, as well as all aspects of the user journey from when someone signs up or logs in to use a service, to the point when they add an item to their basket, check out or seek to change their password.
“The huge upsurge in content scraping, undoubtedly associated with the explosion of Generative AI and LLMs, underlines how dynamic bot traffic is and the need for organisations to be constantly on watch for changes in attack patterns.”
Bots ease up – but not for everyone
Patterns and prevalence of bot traffic varied according to industry. The most targeted industries on web were hospitality (44.6% of traffic from bots), healthcare (32.6%) and eCommerce (22.7%).
On mobile, entertainment (23%) was by far the most targeted sector, well ahead of eCommerce (4.5%) and QSR (4.2%).
Several industries still experience high levels of credential stuffing attacks that seek to take control of user accounts. On web, over a third of login attempts for companies in the technology sector were attempted account takeovers (33.5%), ahead of general retail (25.7%) and gaming (19.6%). On mobile, such attacks were most prevalent against entertainment companies (24.7%) and eCommerce providers (23.8%).
The sophistication of attacks also varied by industry. The vast majority of automated traffic targeting healthcare on both web and mobile was classed as ‘basic’. Other industries experienced relatively high levels of more sophisticated traffic considered ‘advanced’ – the top three on web being general retail, bank and airline.
Despite the high levels of bot traffic, the majority of industries tracked experienced a decline in automated activity compared to 2023, suggesting that bot controls in place were having the desired effect.
The outliers were hospitality on web, which increased by 18.3% and QSR on web, up by 11.2%. Although it experienced a much greater share of bot traffic on mobile than any other industry, the entertainment sector recorded a 11.5% decrease from 2023.
“Certain industries are perennial targets for unwanted bot traffic,” said David Warburton. “Hospitality experiences high volumes because aggregators want to scrape hotel room rate and availability data, or malicious actors are trying to steal loyalty points. In turn, eCommerce providers are targeted by resellers and bots trained to exploit voucher and gift card data.
“These data also show how certain industries have adapted over time: widely targeted sectors such as airlines and financial services have built up defenses to frustrate less sophisticated attackers, meaning they must now contend with a higher proportion of traffic from more advanced, highly persistent operators.”
Mitigation: a double-edged sword?
The report also assessed the impact of deterrence on bot traffic, comparing the experiences of customers who were monitoring automated traffic with those who were mitigating it.
On mobile, the trend was clear and expected. Organizations mitigating traffic saw a significantly lower share of automated activity in their Search traffic (0.9% compared to 24.8% for those monitoring), a pattern matched in Login (5% for mitigators compared to 21.7% for those monitoring) and Sign up (2.4% compared to 21.7%).
On web it was a different story. In most workflows automated traffic was higher for organisations that were actively mitigating bots. These customers saw 20.9% of automated traffic in Search versus 14.9% for those simply monitoring, and the equation was the same in Add to Cart (19.2% vs. 18.2%), Checkout (8.6% vs. 7.4%) and Account Recovery (6.6% vs. 4.6%).
“Typically we expect mitigation to lead to a decline in bot traffic, as operators that are blocked move on in search of weaker targets,” said David Warburton. “That was true to some extent in this analysis, but we also saw operators who had been frustrated try even harder to access the information they were seeking.
“While it feels counterintuitive that mitigation should lead to an increase in traffic, it also makes sense in certain contexts. There are now whole business models built around the scraping of data, prices and intellectual property: those operators are not going to give up easily when they are deterred. An increase in traffic means these actors are trying harder and in more ways, not that they are succeeding in breaching defenses. The consistent trend of this research, and all of our experience at F5, is that mitigation works and deterrence makes a difference.”
