Online Media Brands Hope a New Protocol Will Stop Unwanted AI Crawlers

Online media brands, including Yahoo, Quora and Medium, are taking a new step to prevent AI companies from copying and using their content to train models without their permission.

The publishers, including CNET’s parent company Ziff Davis, see this new tool, called RSL, as another way to ensure large AI developers don’t use their work without payment or compensation — an issue that’s already led to a host of lawsuits. 

RSL, which stands for Really Simple Licensing, is inspired by Really Simple Syndication, a longtime web standard that provides up-to-date and automatic content updates in a computer-readable format. Like RSS, RSL is open, decentralized and can work with pretty much any piece of content online, including web pages, videos and datasets. 

Watch this: The New iPhone Air Changes the Game for Preorders

05:34

Right now, when an AI company’s roving internet robot, known as a crawler, wants to suck up the information on a site, it has to go through robots.txt, which acts as a basic entry or non-entry door. AI companies have found ways around robots.txt or ignored it altogether and have subsequently been sued. The goal for RSL is to be a more robust layer of tech to deal with AI crawlers, which now account for more than half of all internet traffic. (Disclosure: Ziff Davis, CNET’s parent company, in April filed a lawsuit against OpenAI, alleging it infringed Ziff Davis copyrights in training and operating its AI systems.)  

“RSL builds directly on the legacy of RSS, providing the missing licensing layer for the AI-first Internet,” Tim O’Reilly, CEO of O’Reilly Media, said in a press release. “It ensures that the creators and publishers who fuel AI innovation are not just part of the conversation but fairly compensated for the value they create.”

Brands that have signed onto RSL include Reddit, People, Internet Brands, Fastly, wikiHow, O’Reilly, Daily Beast, The MIT Press, Miso, Adweek, Ranker, Evolve Media and Raptive.

“If AI is trained on our writers’ work, then it needs to pay for that work,” Medium CEO Tony Stubblebine said in a press release. “Right now, AI runs on stolen content. Adopting this RSL Standard is how we force those AI companies to either pay for what they use, stop using it, or shut down.”

The advent of RSL comes as online web traffic has cratered with changes to Google and the preponderance of AI. Google’s integrated AI-generated answers at the top of Google Search have been criticized by publishers as taking away from potential clicks they would have received otherwise. Google contends that AI Overviews send “higher quality clicks” to sites, people who are more engaged and stay on sites longer. AI chatbots like ChatGPT also help with research and synthesis, meaning people don’t have to jump around various sites to pull together pieces of information in the same way they did before. Overall, publishers are losing up to 25% of traffic due to AI platforms, according to a report from Infactory.

“Widespread adoption of the RSL Standard will protect the integrity of original work and accelerate a mutually beneficial framework for publishers and AI providers,” Ziff Davis CEO Vivek Shah said.

In response, publishers are suing AI companies or inking licensing deals. In other instances, sites are turning to services like Tollbit, which aim to charge AI crawlers every time they ask to examine a site’s contents. Content delivery networks like Cloudflare, which help ensure people have quick access to sites online, are blocking AI crawlers outright. 

RSL co-founder Eckart Walther said the RSL standard and efforts like that by Cloudflare are complementary, with many of the same media companies participating in both. Walther compared the tools like Cloudflare to bouncers that protect a website from unwanted crawlers, while RSL just allows the crawler to understand the rules and the price of admission. “These compensation methods can also work together. For example, a publisher might want to charge for crawling their content, and then also require a royalty payment every time the content is used by an AI model to reply to a question,” Walther said in an email to CNET.