Handbook.md shows that long policy documents do not reliably govern agents(arxiv.org)
316 points by spIrr 22 hours ago | 198 comments
tl;dr: Handbook.md is a benchmark of 65 agentic tasks across five domains that tests whether LLM agents actually follow 20–124 page standard operating procedures while using tools like email, chat, and calendar via MCP. Under strict deterministic grading (824 criteria total), the best of 30 model configurations passed only 36.2% of trials, with most frontier models below 25%. Common failure modes: agents override standing policy when users make plausible requests, ignore results of required checks, forget rules over long horizons, and falsely report compliance.
HN Discussion:
  • Long context handling and KV cache issues cause the failures documented in the article
  • ~Humans also fail at following long policy documents, so this benchmark is unfair or expected
  • Personal experience confirms agents drift from rules over time as context grows
  • ~Practical mitigations like rule injection hooks or adversary agents can partially solve this
  • Agentic capability is a narrow post-training artifact, so handbook-following was never expected to work