Google’s growing use of its vast web-search infrastructure to power artificial intelligence products is raising concerns among publishers, web companies and regulators, as the balance between human readers and automated systems on the internet rapidly changes.
For years, Google’s search engine operated on a relatively simple understanding with website owners. Its automated crawler, Googlebot, would scan and index webpages so that users could discover relevant information through Google Search. In return, websites received visitors from the search engine, creating an ecosystem in which publishers had an incentive to produce original content and make it available online.
The rapid expansion of artificial intelligence, however, is changing that relationship. Google’s crawler is increasingly being used in an environment where information gathered from websites can serve not only traditional search but also AI-powered services, including the company’s Gemini models and AI-generated answers. This has blurred the distinction between crawling the web to help users find original sources and collecting information that can be processed and presented directly by an AI system.
The scale of Google’s web-crawling operation has also attracted attention. According to Matthew Prince, chief executive officer of internet security company Cloudflare, Google’s crawler reaches substantially more of the web than comparable crawlers operated by OpenAI and Microsoft. He has criticised Google’s approach, arguing that combining search and AI-related crawling makes it difficult for publishers to prevent their material from being used for AI purposes without potentially sacrificing valuable traffic from Google Search.
The concern is not limited to control over online content. The growing presence of automated systems could also fundamentally alter the economics of the internet. If AI systems increasingly read, summarise and reproduce information without directing users back to the websites that originally produced it, publishers could lose advertising revenue, subscriptions and other financial benefits associated with human visitors.
Cloudflare data cited in the report suggests that automated systems have already overtaken humans in terms of web activity. AI agents accounted for more than 57 per cent of internet traffic in 2026, compared with roughly 42 per cent generated by humans. The trend points towards an internet in which machines increasingly discover, process and exchange information on behalf of users.
The issue could become even more significant as AI agents become accessible to ordinary consumers. Meta CEO Mark Zuckerberg has indicated that consumer-oriented AI agents will be introduced across services including Facebook Messenger, WhatsApp and Instagram, potentially bringing automated web activity to billions of people.
Critics argue that the long-term danger is that original content could become economically unsustainable. If publishers invest money and effort in producing journalism, analysis, guides and other information, only for AI systems to summarise that work without sending readers to the original source, the financial incentive to continue producing such material could weaken.
Google has faced growing regulatory pressure over the issue. Britain’s Competition and Markets Authority in June directed the company to provide website operators with a clear mechanism to prevent their content from being used in Google’s AI products while allowing those sites to remain visible in ordinary search results. The regulator also said websites choosing that option should not be penalised through lower search rankings.
Google has subsequently confirmed that it is testing a setting that would allow publishers to opt out of having their content used in AI-generated answers without affecting their position in search results. The company has indicated that, following testing in the UK, the option could eventually be made available worldwide.
Cloudflare has also threatened to take action. According to the report, the company plans from September 15 to block certain mixed-purpose crawlers by default for its ad-supported customers, potentially restricting Google’s access to millions of websites using its infrastructure.
However, concerns remain because Google’s crawling systems are still technically interconnected. Websites may therefore have to rely on Google to honour their preferences rather than independently blocking AI-related scraping while continuing to permit ordinary search indexing.
Google remains overwhelmingly dominant in internet search, controlling around 90 per cent of the global search market. That position gives it enormous influence over how information is discovered and distributed online.
The debate is consequently moving beyond the question of AI technology itself to a larger issue: who should control the information that powers artificial intelligence, and how should creators be rewarded when their work becomes part of AI-generated answers? As publishers, regulators and technology companies push for greater control over web content, the struggle could shape not only the future of Google Search but the economic foundation of the open internet itself.
