robots.txt

A check of 300 top Japanese websites from the Tranco list on 7 October 2026 found that 23 of the 248 pages that loaded normally returned HTTP 403 errors to AI crawler user-agents, even though the sites' robots.txt files permitted those bots. The writeup details which sites are affected, how the blocking patterns differ, how to verify it, and the limits of the method.

A Bloomberg report argues that the web's underlying systems are unprepared for the surge of automated agents now browsing, shopping and acting online. These AI bots often ignore conventions like robots.txt and other rules sites use to govern automated access, straining infrastructure and undermining site owners' control. The piece highlights a widening gap between how the internet was built and how AI-driven traffic actually behaves.