web scraping

Scrapling, an open-source Python web scraping framework by developer D4Vinci, is climbing GitHub's trending list. The project markets itself as handling everything from a single request to a full-scale crawl, and promotes a Discord community and daily scraping tips on social media. Developers are discussing it as a newer alternative for data extraction tasks.

A new essay argues that efforts to wall off the open web—such as paywalls, logins and bot-blocking measures—will not stop malicious automated traffic, and may instead harm legitimate users, researchers and smaller sites. The piece contends that bad actors will always find ways around restrictions, so locking down access punishes good-faith visitors without solving the underlying problem. It is prompting debate about how the web should handle crawlers and abuse.

The New York Times and a group of other publications have filed lawsuits against the company behind the Brave web browser, accusing it of unlawfully scraping their content to train AI systems and of helping users bypass paywalls. The case adds to a growing wave of legal action by media companies against tech firms over the unauthorised use of copyrighted journalism in artificial intelligence products.

The New York Times has filed a lawsuit against Brave Software, the company behind the Brave web browser, accusing it of scraping the newspaper's content to power AI features and of circumventing its paywall. The case adds to a growing list of legal battles between publishers and tech companies over the use of copyrighted material to train or run AI tools.

The tech magazine iX reports on a proposed update to the robots.txt standard that would let website operators block AI training crawlers with a single entry, instead of listing each bot separately. The idea has sparked discussion among developers and privacy advocates, who see it as a simpler way for publishers to defend their content against unwanted use by AI companies.