Choosing an AI model: one prompt, 11 models, different results(netlify.com)
218 points by toddmorey 8 days ago | 95 comments
tl;dr: Netlify partnered with OpenRouter to expand model choice in its Agent Runners (coding agents) and AI Gateway, adding open models like Kimi K3, GLM 5.2, and DeepSeek V4 alongside Claude, GPT, and Gemini. In a test generating a simple coffee shop site, credit costs varied wildly—from 2.4 credits (DeepSeek V4 Flash) to over 1,000 (Claude Opus)—with Opus producing the most polished designs but often overspending, while mid-tier models like GPT 5.6 Terra offered strong value. Follow-up posts will evaluate models on more complex app scenarios involving databases, auth, and AI integration.
HN Discussion:
  • Single-shot simple prompts don't reflect real development work, making the benchmark unhelpful
  • Sample size of one is statistically worthless given model output variance
  • ~Outputs all look depressingly similar and generic AI-styled
  • Generic benchmarks are obsolete; custom evals with LLM judges are trivial to build
  • Appreciates the apples-to-apples comparison and wants more, especially for open-weight models