LONDON, UK— Millions of websites, including major publishers like Sky News, BuzzFeed, and The Associated Press, can now block artificial intelligence (AI) bots from accessing their content without permission — and potentially charge them for doing so — thanks to a new system launched by internet infrastructure giant Cloudflare.
The move, announced this week, is being hailed as a turning point in the ongoing battle between content creators and AI companies accused of scraping the internet for training data without consent or compensation.
Cloudflare, which powers around 20% of the internet, says its new AI bot-blocking technology is now active on over a million websites.
Eventually, the system will allow sites to demand payment from AI firms for accessing their articles, images, and other content.
“This is a critical step toward creating a fair value exchange on the internet that protects creators, supports quality journalism and holds AI companies accountable,” said Roger Lynch, CEO of Condé Nast, whose titles include The New Yorker, Vogue, and GQ.
‘AI Firms Breaking the Deal’
AI bots — also known as crawlers — comb the internet to collect and index data that powers large language models and generative AI tools.
But critics say these bots often ignore existing rules meant to block them and exploit creative work without driving traffic back to the original sources.
“AI crawlers break the unwritten agreement between publishers and search engines,” Cloudflare CEO Matthew Prince said. “They harvest content to generate answers, but they don’t send users back — which means publishers lose revenue.”
Cloudflare is now developing a “Pay Per Crawl” model that will let websites charge AI firms for data access, a potential game-changer in how the internet’s economics work in the age of AI.
The company also said that AI bot activity on its network has exploded, now exceeding 50 billion requests a day.
Some crawlers have reportedly ignored “robots.txt” and other exclusion tools — prompting Cloudflare to earlier deploy a “labyrinth” of junk pages to trap them.
Growing Legal and Creative Tensions
The fight between creators and AI companies is intensifying across the globe.
In the UK, tensions flared between the government and artists including Sir Elton John over copyright protections for creative works being used to train AI.
Meanwhile, the BBC recently threatened legal action against U.S.-based AI firm Perplexity for allegedly using its content without authorization — demanding both compensation and a halt to further use.
Perplexity accused the BBC of protecting Google’s monopoly, highlighting the complex relationships between publishers, search engines, and AI firms.
Ed Newton-Rex, founder of Fairly Trained — a company that certifies AI firms using licensed data — welcomed Cloudflare’s move but warned it’s not enough.
“This is really only a sticking plaster when what’s required is major surgery,” he told the BBC. “It will only offer protection for people on websites they control… The only real way to protect people’s content from theft by AI companies is through the law.”
Call for Fairer AI Economy
Filmmaker and digital rights advocate Baroness Beeban Kidron praised Cloudflare’s leadership, calling the move a model for ethical tech innovation.
“If we want a vibrant public sphere, we need AI companies to contribute to the communities they benefit from,” she said. “That means paying tax, settling with those whose work they’ve used, and — as Cloudflare has shown — using technology to ensure fairness between digital and human creators.”
The new tools are already active for new Cloudflare customers and for websites that previously signed up to block AI crawlers. More are expected to adopt the system in coming months.
