search
web crawler
Trends
- 1Developer's site flooded with AI bot traffic after Hacker News post●I posted to HN, and then my website was flooded with AI bot traffic
Developer Sam Straus says his personal website was overwhelmed by AI crawler traffic shortly after he shared a post on Hacker News. The incident highlights growing frustration among site owners as AI companies aggressively scrape web content, often without permission and at scales that can disrupt small sites. Developers are increasingly comparing notes on blocking AI bots.
- 2Public Book passages become searchable by engines▼The Public Book can now be found by search engines. Every public passage has its own page a crawler can read, with the q
The Public Book, a platform for sharing book passages, is now indexed by search engines. Every public passage has its own crawlable page showing the quote and its source, making individual excerpts discoverable through web search. Personal collections remain private, visible only to the people they are shared with, and are excluded from the index. The change was announced by the project's developer as part of an ongoing build-in-public effort, and early reaction from the reading community has been modest but positive.