Humans missed 1 in 3 threats approving AI agent commands across 40k game runs(scalex.dev)
330 points by Wirbelwind 15 days ago | 244 comments
tl;dr: Data from 40,000 runs of a browser game simulating human approval of AI agent commands showed players missed 33% of threats, with credential exfiltration attacks missed three times more often than obviously destructive commands like `rm -rf /`. The most-missed attacks hid malicious payloads behind familiar-looking scripts like `npm run analyze` (64.7% approval rate), even when the suspicious contents were visible in the log. The author argues human-in-the-loop is an unreliable security boundary due to alert fatigue and missing context, and that sandboxing and isolation should be prioritized over approval prompts.
HN Discussion:
  • Study methodology is flawed due to misleading prompts and artificial time pressure, making conclusions meaningless
  • Sandboxing and isolation should be prioritized because human approval prompts inherently fail due to reflexive clicking
  • Approval prompts were never a real security mechanism, just legal cover for vendors
  • Careful manual review and sandboxing of AI code is essential and professional practice
  • ~Creative interventions like forced attention rituals could improve approval reliability