Kimi K3, and what we can still learn from the pelican benchmark

	Kimi K3, and what we can still learn from the pelican benchmark(simonwillison.net)
	349 points by droidjj 20 hours ago \| 184 comments
	tl;dr: Moonshot AI released Kimi K3, a 2.8 trillion parameter model priced at $3/$15 per million input/output tokens—matching Claude Sonnet pricing and making it the most expensive Chinese model to date, with open weights promised by July 27. Simon Willison's pelican-on-a-bicycle SVG benchmark no longer correlates well with model quality (GLM-5.2 outperforms frontier models on it), but he still finds value in it as a "hello world" prompt for testing new models, gauging reasoning costs, and verifying basic spatial reasoning capabilities.
	HN Discussion: •Technical curiosity about hidden system prompts and token counting anomalies in Kimi K3 ~The pelican benchmark is limited and misses what matters most like agentic tool calling ↑The benchmark's value lies in comparing cost/speed/quality tradeoffs rather than ranking models ↓Existing benchmarks including pelican fail to measure what truly matters in real code work ↓Pelican benchmark is likely contaminated by training data and single-run variance makes comparisons unreliable