An update on residential proxies and the scraper situation(lwn.net)
211 points by chmaynard 16 hours ago | 217 comments
tl;dr: AI scraper traffic increasingly comes through "residential proxy" networks—millions of compromised or opt-in devices (often via shady free VPNs, SDKs, or malware-infected streaming boxes) that make each request look like a unique human visitor, defeating traditional IP-based blocking. LWN has resisted deploying tools like Anubis to avoid burdening real users, instead using undisclosed optimizations and defenses, while noting that proof-of-work is a weak deterrent when attackers command millions of hijacked machines. Recent law-enforcement takedowns of networks like IPIDEA and NetNut have offered only temporary relief in an ongoing arms race threatening the open web.
HN Discussion:
  • Agrees proof-of-work like Anubis is inadequate against scrapers with massive resources
  • ~Solution should be a better common crawl to reduce scraping incentives rather than blocking
  • Anti-scraping tools like Anubis harm legitimate users and the open web more than they help
  • Residential proxies are essentially normalized botnets that should be treated as crimes
  • Curiosity about detecting compromised devices contributing to proxy networks