| Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases(withspecific.com) | |
| 248 points by theanonymousone 15 hours ago | 137 comments | |
tl;dr: Real-SWE benchmarks frontier AI coding agents on tasks pulled from real, private enterprise codebases—work that's never appeared in training data and carries actual business consequences like billing, tax, and migrations. Top performer Fable 5.1 (via Claude Code) resolved just 38.8% of tasks, with 6 of 10 sampled tasks scoring under 15%; the most common failure mode across models was missing requirements, followed by unverified assumptions. Per-rollout costs ranged from $2.50 to $6.96, suggesting current models remain far from meeting enterprise engineering standards. | |
HN Discussion:
| |