Benchmarking Opus 5 on SlopCodeBench(github.com)
306 points by dhorthy 12 hours ago | 70 comments
tl;dr: SlopCodeBench is a long-horizon coding benchmark where models must evolve a codebase across sequential checkpoints without seeing future requirements upfront, making it a good proxy for real maintenance work. On a 17-checkpoint subset, Opus 5 got 24% strict pass (vs 6% for Opus 4.8 and Sonnet 5), but no model completed any full challenge cleanly, and all showed rising complexity, duplication, and verbosity over time. The author argues this finally provides hard data for the intuition that current frontier models can't be trusted to run "lights-off" on iterative software work without human steering.
HN Discussion:
  • Benchmark uniquely captures long-horizon maintenance work and code cleanliness better than alternatives
  • Labs should use this in RL pipelines to prioritize reducing code complexity
  • ~Results need human baseline comparison to be properly interpreted
  • Personal experience confirms Opus 5 is only a modest improvement over predecessors
  • Suggestions to extend methodology (test ordering, PR review integration, adversarial prompting)