#Benchmaxxing
if Eric Schmidt was still at Google they’d be benchmaxxing
November 12, 2026 at 3:55 PM
Benchmaxxing, explained:
November 16, 2026 at 8:28 PM
to think that's all thanks to benchmaxxing on simon willison
November 10, 2026 at 11:37 PM
The good kind of benchmaxxing 💚
November 10, 2026 at 3:24 PM
I was referring to benchmaxxing papers generally involving polygenic scores measured in explained variance. Usually very marginal gains, occasionally significant in a particular set of trait categories, which can be handpicked in the biobank era of too many measured traits.
November 10, 2026 at 9:28 AM
My own experience is that glm 5.3 still edges it, especially on tricky or complex tasks where DS tends to spin and make worse judgment calls. The recent benchmaxxing tests seem possibly relevant here.
November 10, 2026 at 2:12 PM
Yeah every lab is benchmaxxing and this is probably worse for small models and models from Chinese labs.

Not really sure how much to correct for this though. These models still perform pretty well on closed benchmarks (though these have their own issues, likely more errors).
November 10, 2026 at 4:02 PM
My conclusion from this is that K3.1 falls short of Fable 5. Particularly in light of some results on benchmaxxing (though don't take these at face value, private benchmarks have their own problems).

Though it's probably better than Opus 4.8 from ... 5 months ago.
November 9, 2026 at 10:05 PM
They're all benchmaxxing while opus is building the bench
November 8, 2026 at 4:41 PM
All companies are benchmaxxing. But small models memorize stuff more.

Idk if this is because companies hill-climb more with these, or just because they are less able to generalize.
What happens to model performance on the popular DeepSWE coding benchmark if you make small alterations to the questions, like changing a file name to lowercase?

Models that once looked very similar to each other no longer look so similar. maxxbench.unlap.ai/research/why...
November 9, 2026 at 3:03 PM
Commenters kept coming back to prompt geometry: models recognize the 'benchmark shape' of an issue and output what the test harness wants. The fight was whether labs are deliberately benchmaxxing to pump numbers or if smaller architectures simply lack the capacity to generalize. 2/4
November 8, 2026 at 10:00 AM
finally some good evidence for the disparity between the benchmarks and the "vibe test" - benchmaxxing to the maxx
This study finds that Moonshot AI's Kimi K3.1 Pro is bench maxxed (Terminal Bench tasks).

Study: r40.org/papers/kimi_... (arxiv version shortly)
Data: huggingface.co/datasets/R40...
November 6, 2026 at 9:21 AM
ouuuu im benchmaxxing
November 5, 2026 at 2:05 AM
trying out kuna, the new "best decompiler", and thoroughly not impressed :/ hit a half dozen bugs and missing features for things that i often need to do in ghidra, and for almost no functions are its output much better or even often any different. it seems like its mostly just "benchmaxxing"
November 8, 2026 at 2:20 AM
different but actually test the same underlying logic and reasoning steps). not enough people are reading bench traces like this 👇 and the systematic way to catch this benchmaxxing is basically...

Source: @swyx on X
Original Post:
https://x.com/swyx/status/2118400768957202439
November 5, 2026 at 5:56 PM
Benchmaxxed AI Models: The Illusion of High Performance and How to Rigorously Validate LLMs Beyond Public Benchmarks + Video

Introduction: "Benchmaxxing" in artificial intelligence occurs when large language models are overly optimized to ace famous public benchmarks while failing catastrophically…
Benchmaxxed AI Models: The Illusion of High Performance and How to Rigorously Validate LLMs Beyond Public Benchmarks + Video
Introduction: "Benchmaxxing" in artificial intelligence occurs when large language models are overly optimized to ace famous public benchmarks while failing catastrophically on held-out, real-world evaluation sets. This phenomenon mirrors high-performance vehicles with legendary badges that disappoint under the hood, exposing critical vulnerabilities in standard model evaluation and deployment pipelines. Learning Objectives & Secrets: Objective 1: Master the identification of benchmark contamination and overfitting in open-source and proprietary Large Language Models (LLMs) using statistical anomaly detection.
undercodetesting.com
November 4, 2026 at 11:25 PM
Labs when asked if they're benchmaxxing:
November 2, 2026 at 6:31 PM
>switch to private benchmarks
>mysteriously, Chinese models and Grok fall way behind
>Anthropic, OpenAI, and Google all still do about the same

The only thing surprising about this is that Facebook held up so well tbh, I had assumed they were benchmaxxing the shit out of Muse
October 28, 2026 at 6:36 PM
We should goodhart max to the point that the benchmaxxing conveys aesopian fables to them
Boy I hope I’m not held to any metrics that have become measures at work today
October 28, 2026 at 5:04 PM
Couldn't they transparently publish like 5% random samples? Or is that enough for benchmaxxing?
October 28, 2026 at 1:47 PM
Your regular update of anti-benchmaxxing or: open models are still behind artificialanalysis.ai/articles/art...
October 28, 2026 at 4:08 AM
pretty sure AA heavily dropping open weight models is less about 'benchmaxxing' and moreso changing the curriculum for other tasks, like computer use, robotics and finance.

Coding index looks like a more an old fair comparison, Terra isn't inflated here.

This still feels like a disingenuous move..
October 28, 2026 at 8:23 PM
I hope there could be independent verification of their results, then. I understand the need for closed benchmarks to prevent benchmaxxing, but it was also important to make sure AA wasn't lying about the results.
October 28, 2026 at 2:16 PM
You can see who has been benchmaxxing most after @ArtificialAnlys released their v5 benchmark.

Looking at you Kimi K3.1 Pro, Gemini 4 and Grok 4.7
October 28, 2026 at 2:35 PM
The new Artificial Analysis Intelligence Index just blew the lid off the open‑weight vs closed model gap—Kimi K3.1 Pro and Grok are feeling the heat. See how private test sets are reshaping benchmarks. #ArtificialAnalysisIntelligenceIndex #OpenWeightModels #Benchmaxxing

🔗
October 28, 2026 at 3:42 PM