web crawler

A researcher says OpenAI's automated agents scanned a United Nations statistics website roughly 16,500 times, raising fresh questions about the load and consent issues created by AI web crawlers. The claim highlights growing friction between AI companies collecting web data at scale and the sites whose infrastructure absorbs the traffic.

Automated AI agents flooded and hammered a United Nations website, according to a tech news brief. The incident highlights growing strain on public web infrastructure as AI-driven crawlers and bots increasingly hit sites at scale. The UN has not issued a detailed public statement on the disruption, and reports so far offer limited detail on which site or service was affected or for how long.

Security researcher Rowan Howard-Jones reports that OpenAI's automated agents hit the UN Conference on Trade and Development's statistics site more than 16,000 times between April and June. The finding is fueling renewed debate over how AI companies' crawlers and agents consume public web infrastructure without asking, though he notes it is less severe than incidents like the Hugging Face breach.

A Cloudflare Radar dashboard for AS13238, the network associated with Russian search engine Yandex, shows heavy bot traffic over the past seven days, and the figures are being circulated and debated among web infrastructure professionals. Commenters are discussing the scale of automated crawling attributed to Yandex and what it means for sites dealing with unwanted crawler load.

Yandex's crawler traffic has collapsed by roughly 99% following an attack on one of its data centers, according to traffic monitoring data for the Russian search company's network. The drop means Yandex's bots have largely stopped crawling the web, prompting discussion about the outage's cause, scale, and impact on sites that rely on or block its indexing activity.

A debate is under way over whether online aggregators — platforms that collect and compare content, prices or listings — should block AI agents from accessing their services. As AI assistants increasingly browse, scrape and transact on users' behalf, aggregators face a choice between blocking them to protect traffic and revenue, or allowing access. The question touches on broader industry tensions over AI crawlers and website control.

Web developers are being reminded that blocking well-known crawlers and AI scrapers is not enough: some bots send a hyphen or an empty string as their user-agent to evade filtering. The advice circulating among developers is to also block these blank user agents in robots.txt or server rules, since many scraping and AI training bots use them.

Designer and product leader Luke Wroblewski has published a new entry in his Ask LukeW series looking at instant content indexing, the practice of making newly published material searchable almost immediately rather than waiting for traditional crawler-based indexing. The piece is circulating among web and internet commentators interested in how search and discovery are evolving.

A new animation showing a spider moving through a web as a metaphor for how a web crawler indexes pages online is circulating widely. The visual breaks down a technical process into an easy-to-grasp analogy, drawing attention from both tech audiences and general viewers who are sharing and discussing it.

Cloudflare argues that the internet now serves a second audience beyond human readers: automated systems such as AI crawlers and agents that consume web content at scale. The piece discusses what this shift means for publishers and the open web, as machine traffic grows and content is increasingly read by software rather than people.

A blog essay by Chris Siebenmann argues that one possible future for the open web is one in which pervasive gatekeepers control access to websites, whether through platforms, crawlers, or intermediaries. The piece is being shared and discussed on decentralized social networks, where users are debating how much control intermediaries should have over who reaches independent sites.

A discussion among web publishers and Chrome on Linux users is asking whether Meta's Muse crawler or agent can be blocked at the server level without resorting to fingerprinting scripts that would also block legitimate Chrome on Linux visitors. Some users report being caught by bot-detection systems more often, and site owners are weighing how to distinguish AI agents from real people.

The Public Book, a platform for sharing book passages, is now indexed by search engines. Every public passage has its own crawlable page showing the quote and its source, making individual excerpts discoverable through web search. Personal collections remain private, visible only to the people they are shared with, and are excluded from the index. The change was announced by the project's developer as part of an ongoing build-in-public effort, and early reaction from the reading community has been modest but positive.

Yandex's crawler traffic has reportedly collapsed by roughly 99% following an attack on one of the company's data centers, according to bot traffic data published on Cloudflare Radar. The figures are circulating on Hacker News, where users are weighing what the drop means for the Russian search giant's web indexing and for sites that rely on its crawler.

An AI Crawler Index has been launched, cataloguing the web crawlers used by artificial intelligence companies. It aims to help site owners identify which AI agents access their content, a growing concern as publishers debate whether to block or license their material to AI firms. Discussion has focused on how sites can manage automated AI traffic.