SlopCodeBench 10/2026 – Opus 5.5, Astra wins overall, xhigh considered harmful(github.com)
304 points by dhorthy 10 days ago | 120 comments
tl;dr: On the full 36-problem SlopCodeBench, GPT-6 Astra XHigh led strict pass rate at 16.3%, narrowly beating GPT-6.1 SOL Medium (14.8%) and Opus 5.5 Medium (14.3%), while Opus 5.5 Medium topped the looser "core" rate at 74%. Higher reasoning effort (XHigh) cost 2-4x more but didn't reliably improve strict scores—Opus 5.5 actually did worse on XHigh than Medium, though its XHigh output was ~5x cleaner by slop metrics. Models still routinely break regression tests as requirements evolve, with only 0-3 of 36 final checkpoints passing strictly.
HN Discussion:
  • ↑Astra produces unreadable code-golf style output despite good writing quality
  • ↑Higher reasoning effort has diminishing returns; medium is often optimal
  • ↑LLM slop stems from overly defensive programming and reluctance to simplify
  • ↓Benchmark omits important configurations like high, max, and default Sol Low
  • •Questions about reproducibility and practical ways to run the benchmark