Not really sure how much to correct for this though. These models still perform pretty well on closed benchmarks (though these have their own issues, likely more errors).
Not really sure how much to correct for this though. These models still perform pretty well on closed benchmarks (though these have their own issues, likely more errors).
Though it's probably better than Opus 4.8 from ... 5 months ago.
Though it's probably better than Opus 4.8 from ... 5 months ago.
Idk if this is because companies hill-climb more with these, or just because they are less able to generalize.
Models that once looked very similar to each other no longer look so similar. maxxbench.unlap.ai/research/why...
Idk if this is because companies hill-climb more with these, or just because they are less able to generalize.
Study: r40.org/papers/kimi_... (arxiv version shortly)
Data: huggingface.co/datasets/R40...
Source: @swyx on X
Original Post:
https://x.com/swyx/status/2118400768957202439
Source: @swyx on X
Original Post:
https://x.com/swyx/status/2118400768957202439
Introduction: "Benchmaxxing" in artificial intelligence occurs when large language models are overly optimized to ace famous public benchmarks while failing catastrophically…
Introduction: "Benchmaxxing" in artificial intelligence occurs when large language models are overly optimized to ace famous public benchmarks while failing catastrophically…
>mysteriously, Chinese models and Grok fall way behind
>Anthropic, OpenAI, and Google all still do about the same
The only thing surprising about this is that Facebook held up so well tbh, I had assumed they were benchmaxxing the shit out of Muse
>mysteriously, Chinese models and Grok fall way behind
>Anthropic, OpenAI, and Google all still do about the same
The only thing surprising about this is that Facebook held up so well tbh, I had assumed they were benchmaxxing the shit out of Muse
Coding index looks like a more an old fair comparison, Terra isn't inflated here.
This still feels like a disingenuous move..
Coding index looks like a more an old fair comparison, Terra isn't inflated here.
This still feels like a disingenuous move..
Looking at you Kimi K3.1 Pro, Gemini 4 and Grok 4.7
Looking at you Kimi K3.1 Pro, Gemini 4 and Grok 4.7
🔗
🔗