Cloudflare’s latest launch and IAB Tech Lab’s Content Ingest API marks a turning point for fair use in the age of generative AI.
Artificial intelligence is transforming digital media. For publishers, one of the most pressing issues remains unresolved – how to manage automated access to content in a way that is fair, enforceable, and monetizable. Until now, the options have been limited. Publishers could either allow AI crawlers to scrape their content with no compensation or block them entirely. Neither approach offered a sustainable solution for content producers.
That’s changing. Cloudflare’s new pay-per-crawl model introduces a third path, allowing website owners to charge AI crawlers for access. This new capability is built directly into Cloudflare’s infrastructure and supports three permissions:
- Allow: Free access to approved crawlers
- Block: Full restriction
- Charge: Access granted only through a paid model
What makes this noteworthy is the addition of enforceable billing and clear crawler notifications. Even without a billing arrangement in place, AI agents will now receive explicit rules about access and usage. This brings a level of operational clarity that has been missing.
This advancement aligns with ongoing work at the IAB Tech Lab, where the Content Ingest API was introduced earlier this year. The initiative is focused on creating a standard protocol for how AI systems access content, covering transparency, permissions, and compensation. The Pay-Per-Call working group within Tech Lab is actively building scalable models that support monetization without slowing innovation.
Together, Cloudflare’s implementation and Tech Lab’s standards represent meaningful progress for the digital publishing ecosystem. This is not just a technical update. It’s a sign that the industry is serious about addressing the value exchange between AI developers and content producers.
What this means for Canadian publishers: Now is the time to get involved. The frameworks are taking shape, and early participation is essential in shaping how these standards evolve. IAB Canada encourages our members to join IAB Tech Lab’s working groups and contribute to building a future that supports both innovation and sustainable content creation.