Mmastodon TechnologyTechnology first seen 7 d ago, last 7 d ago, peak #3
Web developers warned to block empty user-agent bots
Original: If you want block # crawlers and # ai bots, don't forget to block "-" and "" (empty string) bots too. picrel # webdev #
Web developers are being reminded that blocking well-known crawlers and AI scrapers is not enough: some bots send a hyphen or an empty string as their user-agent to evade filtering. The advice circulating among developers is to also block these blank user agents in robots.txt or server rules, since many scraping and AI training bots use them.
Why now: Growing concern over AI companies scraping websites has developers sharing blocking techniques.
Evidence
- If you want block # crawlers and # ai bots, don't forget to block "-" and "" (empty string) bots too. picrel # webdev # tech · invirtuate · 2
API: https://socialmediatrends-api.osmike.com/v1/trends/2821881