Mmastodon Business Business first seen 8 d ago, last 5 d ago, peak #1

Meta says robots.txt can block its AI training crawler

Original: META also says: "content owners can use robots.txt flags to indicate their preferences. The Meta-ExternalAgent crawler (

Meta has confirmed that its Meta-ExternalAgent crawler, used for AI training and indexing, will honor robots.txt directives, meaning site owners who object must manually add a User-agent: meta-externalagent Disallow: / rule to their robots.txt file. Critics are unimpressed, arguing the opt-out burden falls on publishers rather than requiring consent upfront, reigniting debate over AI firms scraping web content without permission.

Why now: Publishers and webmasters are reacting to Meta placing the responsibility for blocking AI scraping on content owners.

Meta Meta-ExternalAgent robots.txt

Open on mastodon →

Evidence

META also says: "content owners can use robots.txt flags to indicate their preferences. The Meta-ExternalAgent crawler (for AI training and indexing) will honor it" 🙄 So, on the robots.txt file we need to add User-agent: meta-externalagent Disallow: / to block it. 🔗 Source:… · arteesetica · 9

API: https://socialmediatrends-api.osmike.com/v1/trends/2777965