AI crawlers

A researcher says OpenAI's automated agents scanned a United Nations statistics website roughly 16,500 times, raising fresh questions about the load and consent issues created by AI web crawlers. The claim highlights growing friction between AI companies collecting web data at scale and the sites whose infrastructure absorbs the traffic.

Automated AI agents flooded and hammered a United Nations website, according to a tech news brief. The incident highlights growing strain on public web infrastructure as AI-driven crawlers and bots increasingly hit sites at scale. The UN has not issued a detailed public statement on the disruption, and reports so far offer limited detail on which site or service was affected or for how long.

Security researcher Rowan Howard-Jones reports that OpenAI's automated agents hit the UN Conference on Trade and Development's statistics site more than 16,000 times between April and June. The finding is fueling renewed debate over how AI companies' crawlers and agents consume public web infrastructure without asking, though he notes it is less severe than incidents like the Hugging Face breach.

Cloudflare argues that the internet now serves a second audience beyond human readers: automated systems such as AI crawlers and agents that consume web content at scale. The piece discusses what this shift means for publishers and the open web, as machine traffic grows and content is increasingly read by software rather than people.

Reddit says it must restrict access to its legacy 'Old Reddit' interface because of AI scraping bots overwhelming the site. The company frames the move as necessary to protect the platform from automated traffic, though the change is likely to frustrate longtime users who prefer the simpler, faster classic layout over the modern redesign.

Cloudflare is promoting a 'Pay Per Use' model in which websites and creators get compensated when AI systems crawl or use their content. The idea responds to growing concern that AI companies train and power products on published work without payment or permission. The proposal would give publishers a way to charge for access rather than letting crawlers take content for free.

Website owners are being urged to inspect their server logs, where unfamiliar user agents like GPTBot and ClaudeBot reveal AI crawlers scraping content. The question dividing developers and SEO specialists is whether to block these bots via robots.txt, allow them to index content, or charge for access. The debate touches on copyright, traffic loss, and whether blocking AI training data helps or hurts visibility.