Brier Dudley is editor of The Seattle Times Save the Free Press Initiative. This column was originally published in The Times on August 1st and is reprinted here with permission. Dudley’s work will appear regularly on the Medill Local News Initiative site.
Local newspapers have an intriguing new ally in their quest to get paid by AI firms hoovering their stories and putting their future at risk.
Matthew Prince, CEO of San Francisco-based Cloudflare, believes his company has a tool that may solve this dilemma for both news publishers and tech firms.
Prince has skin in the game beyond Cloudflare. He and his wife, Tatiana, own The Park Record, an award-winning newspaper in his hometown of Park City, Utah.
But it was pleas for help from bigger publishers, who use Cloudflare’s network, security and content-delivery systems, that prompted the company to address the swarms of AI bots endlessly scraping their websites.
Those bots take content to feed AI models. The AI models in turn are killing traditional search traffic to news publishers, jeopardizing their ability to succeed online.
Cloudflare built a tool called “Pay Per Crawl” that Prince announced last month. It’s intended to give publishers more control over AI crawlers accessing their websites.
Perhaps most importantly, it enables publishers to start charging AI companies. It uses web instructions that inform crawlers they need to pay for content they take.
We’ll see if it works. Government policies and litigation may still be required to strengthen and enforce payment. But I think this could be one of the keys to saving local journalism.
Right now only the largest news organizations have the clout and resources to negotiate payment deals with tech giants.
The 9,000 or so local outlets, including around 5,600 smaller newspapers, need a simple tool like Cloudflare’s to erect a tollbooth for AI crawlers taking their work.
This complements legislation proposed in the U.S., and enacted in other countries, to help news outlets get fairly compensated by tech giants making billions off their content.
A “tollbooth for crawlers” clarifies the situation for policymakers and could provide a framework for negotiating payments.
It also removes complexity and ambiguity that bogged down the Journalism Preservation and Competition Act in Congress and similar policies in Oregon and California. They were all trying to figure out how to ensure publishers get paid online, but until now there weren’t standard ways to proceed.
In a phone interview, Prince said the tool may also give publishers a new way to value their work that’s less focused on search traffic.
The tool should help AI companies as well, Prince said.
He believes AI companies’ graphics processing units and talent will become more commodified but content won’t.
“And so I think that over time, the AI companies that will win will be the ones that have access to unique content that makes their system more valuable,” he said, adding that they should eventually be spending more on content than salaries and processors.
If that proves true, “then we should all be buying local newspapers because they’re going to be pretty valuable,” he added.
How will Cloudflare get paid? Prince said the tool supports its customers, other companies could pay to use Cloudflare tools to monitor transactions and Cloudflare could get paid to help negotiate payments.
“Even if we didn’t make $1 off of it, we’d still do it, because it’s the right thing for the internet,” he said. “And again, we see this as an existential threat to us, that if people stop putting content on the internet, the internet goes away. If the internet goes away, our business goes away.”
Cloudflare isn’t doing this alone. It’s working with an internet engineering standards group that’s fast-tracking a new standard for how AI crawlers interact with content.
Prince expects “responsible crawling” standards to emerge in the next 90 days. Companies should start aligning behind them shortly afterward, though Google may take some persuading.
He predicts this will lead to “significant deals done by AI companies” in 2026, paying on a per-crawl basis.
“I would hope that this becomes a meaningful revenue stream to the world’s content creators over the course of second half of 2026 and going forward from there,” he said.
If that pans out it could help stabilize newspapers’ online businesses and support thousands of newsroom jobs.
Maybe I’m putting too much stock in this because signs of hope are rare on this beat.
But I wonder if the tools and standards are a turning point, similar to the emergence of digital-rights technology that helped save the music industry after rampant piracy in the late 1990s.
That also required public policy and courts to strengthen copyright protections and enforcement. It needed the public to realize it must pay for quality content. And the industry needed new technology to protect its work and get paid for its use online.
It worked, and now 100 million Americans subscribe to digital music services. Traditional formats also survived, with physical media accounting for 11% of recorded music sales last year.
Prince, a former intellectual-property law professor, recast my music-industry analogy.
His “utopian future” envisions more of the fees, that everyone pays the bot system crawling the web, going back to creators of original content.
“If we can get to that, there is a chance we have a music industry-like moment where the content industry actually grows and doesn’t shrink as a result of AI,” he said. “But we have to do that. We have to get these things right, right now. And that’s why this is just one of our top strategic priorities.”