Category: Cyber Security | Published: 2025-07-17
AI Crawlers A Problem for Publishers and Creators
In recent years, the rapid growth of AI tools has sparked a battle over ownership, access, and compensation. At the centre of the controversy are _“AI crawlers”_, i.e. automated bots developed by companies like OpenAI, Google, and Anthropic to trawl the internet, copying data from websites to train large language models (LLMs) or power AI assistants.
For creators and publishers, the issue is that this content is often scraped without permission or compensation. Unlike traditional web crawlers used by search engines, which drive traffic back to the original source and support advertising revenue, AI bots typically use the content to generate summaries, answers or outputs directly, without crediting or linking to the sites they pulled from. This bypasses publishers entirely, cutting them out of the value chain.