Livenerf: Has Opus 5.5 been nerfed yet?(github.com)
911 points by bryan0 14 days ago | 388 comments
tl;dr: Livenerf is an append-only benchmark that tests whether Claude Opus 5.5 degrades after launch, using a frozen panel of 78 "sometimes right" questions from GPQA Diamond, MMLU-Pro, and competition math run daily through a pinned Claude Code CLI. It uses pre-registered statistics (paired deltas with clustered SEs), a control arm running Opus 5, and a decision rule requiring a 3+ point change clearing 99% intervals in two consecutive 10-day windows. Validation shows it can detect lower reasoning effort but can't distinguish a same-family model swap in one window's samples.
HN Discussion:
  • ~Similar benchmarks already exist and nerfs are often honeymoon-effect perception rather than reality
  • ↑Nerfs are definitely real due to constant infrastructure changes causing regressions
  • ↑Personal experience strongly confirms models get nerfed after launch hype dies down
  • ↑Appreciates the tracking effort to settle recurring debates with actual data
  • ↓Nerfing claims aren't real; users just hit model complexity ceilings over time