Breaking the open web won't stop the bad bots(techpolicy.press)
269 points by raybb 3 days ago | 115 comments
tl;dr: Proposed federal and New York bills targeting "stealth" web crawlers would require bots to disclose their identity and purpose, with penalties for anonymous scraping that could "burden" websites. The authors argue this approach misses the mark: unmasking crawlers doesn't reduce server load or compensate publishers, while it would chill legitimate anonymous scraping used by journalists, researchers, archivists (like the Internet Archive), and accessibility advocates. They suggest policymakers instead support technical standards for managing bot traffic and tie liability to actual harmful conduct rather than anonymity itself.
HN Discussion:
  • ↑Agrees identity disclosure is irrelevant; actual behavior and scale cause harm, not anonymity
  • •Questions the viability of an open web without paywalls or ID-gating, exploring alternatives like micropayments
  • ↑Current bot-protection solutions already harm legitimate users with faulty fingerprinting
  • ↓Suspicious of the article as AI-industry lobbying to kill reasonable regulation
  • ~More targeted legislation differentiating scale of abuse would be better than blanket rules