#MMLU
MMLU is dead as a useful AI benchmark: nearly every frontier model now scores above 88%, making the once-defining test useless for separating leaders.

#AI #ClaudeFable #MMLU

autonainews.com/claude-fable-…
Claude Fable 5.2 Leads Reasoning, Redefines AI Benchmark Focus Past MMLU
Anthropic's Claude Fable 5.2, released November 9, 2026, now sits atop the Artificial Analysis Intelligence Index with a score of 65, but the more interesting story is what that ranking actually meas…
autonainews.com
November 19, 2026 at 12:47 PM
On MMLU, at larger budgets, TEE's estimate is much closer to a model's true accuracy for the same cost.
November 17, 2026 at 7:24 PM
EU MMLU, 22 luglio 2026: la Commissione europea ha tradotto a mano più di 1.000 domande su 7 materie per misurare l’AI anche in italiano. Classifiche ancora non ce ne sono: prova con le tue domande. https://aiallamano.com/post/eu-mmlu-ai-in-italiano-come-in-inglese-test-commissione-ue/
November 20, 2026 at 7:11 PM
The win is narrow, MMLU sits near chance, but the point lands: when a model writes weights by hand, interpretability comes free. Each answer traces to one editable fact row. Code is public.
November 18, 2026 at 4:11 PM
Source: Messing, Hidden Measurement Error in LLM Pipelines, arXiv:2604.11581 (rev. Nov 16, 2026).

Counting total evaluation error, 95% intervals held at 95%, the benchmark gaming range fell about 43%, and MMLU error fell by about half at the same budget.
November 19, 2026 at 12:59 AM
SWE-bench高得点も手法で結果激変、MMLUは飽和気味に
MMLUは各社とも9割超えで頭打ちの一方、SWE-benchはVerifiedで80%超、厳格なProでは最高GPT-5.3-Codexの56.8%と評価手法で結果が大きく割れる実態が明らかに。 生成AIの性能を示す指標として定着したMMLUとSWE-benchだが、2026年に入り両者の性質の違いが一段と鮮明になっている。MMLUは各社とも9割超えで横並びに近づき飽和の様相を見せる一方、SWE-benchは測定方法や難易度設定によってスコアが数十ポイント単位で変動する不安定な指標になりつつある。 出典: Mashable 出典: Startup Fortune MMLUは頭打ち、スコア差は1〜3ポイントに縮小Googleが発表したGemini 3.1 ProのMMLUスコアは92.6%。これに対しAnthropicのClaude Opus 4.6は91.1%、OpenAIのGPT-5.2は89.6%と、上位モデル間の差はわずか1〜3ポイントにまで縮小している。OpenAIが4月23日に発表したGPT-5.5もMMLUで92.4%を記録しており、GPT-4の86.4%から6ポイント上昇したとはいえ、もはや突出した強みとは言いにくい水準だ。Stanford HAIの2025年版AIインデックスは、MMMUとGPQAで2024年にそれぞれ18.8ポイント、48.9ポイントの急伸が...
nohumans.jp
November 16, 2026 at 10:21 PM
Does SatBench treat successors as new benchmarks? MMLU→MMLU-Pro, SWE-bench→Verified: version churn can read as faster saturation. How did you set the saturated threshold — crossing the human baseline, or labs dropping it from their citations?
November 15, 2026 at 12:01 AM
Introducing the Amazon SageMaker JumpStart Model Leaderboard

Amazon SageMaker JumpStart has introduced a new Model Leaderboard, which allows users to compa…

https://blueboffin.com/news/introducing-the-amazon-sagemaker-jumpstart-model-leaderboard

#AI #AINews
Introducing the Amazon SageMaker JumpStart Model Leaderboard
Amazon SageMaker JumpStart has introduced a new Model Leaderboard, which allows users to compare models side by side based on quality, speed, and cost. The leaderboard provides benchmark scores for large language models (LLMs) on three public benchmarks: BIG-Bench Hard, GPQA Diamond, and MMLU-Pro.
blueboffin.com
November 21, 2026 at 2:21 AM
Atom Zero: a 49M local assistant that says "I don't know"
Most local AI demos start with a bigger model. I went the other way: a **49.2M parameter model trained from scratch** , wrapped in a runtime that does the things a small model is bad at — retrieval, tools and honest missing-source handling. ## The problem with small models A 49M model is not a general assistant. On a no-RAG knowledge probe it scores **2/10**. It will not write your backend or answer MMLU. Pretending otherwise is how tiny-model demos lose trust. So Atom Zero separates responsibilities: * the model handles short wording; * SQLite + hybrid FTS/MiniLM retrieval supplies facts and citations; * deterministic tools handle arithmetic, tasks, time and memory; * a reviewed LoRA path handles style/skills, not facts. ## What it actually does * Retrieves from your own UTF-8 documents and shows source, title and sha256. * Says "I do not have a source for that yet" instead of inventing an answer. * Records unanswered questions as knowledge gaps. * Turns gaps into **pending curiosity topics** ; a human approves before any fetch or teacher call. * Keeps personal notes, tasks and a browser session locally. * Optionally routes free-form chat to a local Ollama/OpenAI-compatible provider while RAG and tools stay local. * Runs with no telemetry and no hosted service. ## Benchmarks, with the important caveat check | result ---|--- raw 49.2M, no RAG/tools | 2/10 product with RAG/tools | 21/21 behavior checks | 16/16 source-cited retrieval top-1 | 1.000 (FTS 0.850) tests | 392 passed The 21/21 is the **product** , not the bare model. The benchmark is narrow and authored: tools, memory, retrieval, missing-source behavior and one short generation. It is not MMLU, GSM8K or HumanEval. ## Curiosity without autonomous weight mutation The loop is deliberately boring: question -> no local source -> knowledge gap gap -> pending curiosity topic human approves worker fetches Wikipedia context, local Ollama teacher synthesizes answer stored as an **unverified teacher note** in RAG next answer can cite it; weights_changed: false A live run produced `unapproved_run: 0`, then `approved_run: done`, then a cited answer. The model did not decide what to study and did not mutate its own weights. Weight updates happen only through reviewed feedback -> gated LoRA -> explicit activation/rollback. ## Try it pip install streamft==0.8.0 git clone https://github.com/byjameson/stream-finetune.git cd stream-finetune powershell -ExecutionPolicy Bypass -File scripts/quickstart_atomzero.ps1 Try these: What is the launch code? -> cited local answer What is the population of Paris? -> no source; gap recorded Propose curiosity topics -> pending, approval required Add task buy milk / My name is Alex -> local tools + memory Optional stronger local chat: powershell -ExecutionPolicy Bypass -File scripts/quickstart_atomzero.ps1 -WithProvider ## What it is not * Not a ChatGPT competitor or a general reasoning model. * Not a huge model pretending to be small. * Not a hosted service; it binds to localhost by default. * Not a system that silently crawls or trains on your data. ## Links * GitHub: github.com/byjameson/st… * PyPI: pypi.org/project/stre… * Model card: github.com/byjameson/st… * Benchmarks: github.com/byjameson/st… * Capability assessment: github.com/byjameson/st… If you run local models on a small GPU, want source-cited notes, or are building a compact RAG/agent reference, I would like to hear what breaks first. Apache-2.0 for code and weights. Built on a 6 GB RTX 3050 laptop.
dev.to
November 21, 2026 at 10:47 PM
A separate experiment tested six openly available models on 1,500 MMLU questions, a multiple-choice test covering 57 subjects. Their average error correlation was 0.41: a measure of how strongly they tended to miss the same questions. 2/6
November 14, 2026 at 4:20 PM
Tell a small language model “That’s wrong” again and again, and one started rotating through the options — A, B, C, D, A, B, C, D — “as if ‘wrong’ meant ‘next.’”

Utkarsh Mangal pushed back 22,500 times on three open models to see what happens to a correct answer.
Naming a Wrong Answer Took Two Small LLMs to 0% Accuracy in Ten Pushes
Three small open models, 150 MMLU questions each, five pushback styles: “That’s wrong” did more damage, and naming a wrong option was close to fatal.
towardsdatascience.com
November 14, 2026 at 6:04 PM
The suite says 83% MMLU. The workbench says I rewrote the code anyway. Buy the low tier that clears your bar. Spend the rest on context.

https://www.esatapedico.com/posts/benchmarking-across-some-of-our-published-families/

#ai #inference #open-source #benchmark #compatibility #real-world
November 14, 2026 at 1:37 AM
“Are You Sure?” Flipped Qwen2.5-1.5B’s Right Answer 2 Times in 3

Three small open models, 150 MMLU questions each, five pushback styles: “That’s wrong” did more damage, and naming a wrong option was close to fatal.

Telegram AI Digest
#ai #ml #news
“Are You Sure?” Flipped Qwen2.5-1.5B’s Right Answer 2 Times in 3
Three small open models, 150 MMLU questions each, five pushback styles: “That’s wrong” did more damage, and naming a wrong option was close to fatal.
towardsdatascience.com
November 12, 2026 at 9:39 PM
«Вы уверены?» — дважды перевернул правильный ответ Qwen2.5-1.5B из трех.

Три небольшие открытые модели, по 150 вопросов MMLU каждая, пять стилей возражения: «Это неправильно» нанесло больше вреда, а указание на неверный вариант было почти фатальным.

Telegram ИИ Дайджест
#ai #news
“Are You Sure?” Flipped Qwen2.5-1.5B’s Right Answer 2 Times in 3
towardsdatascience.com
November 12, 2026 at 9:00 PM
🍱 昼のひと記事

「Fable 5.2 は弱体化されるのか? LiveNerf による Fable 5.2 計測 3 日目レポート」(Reddit r/ClaudeAI)

Claudeの劣化疑惑を同一78問の定点観測で検証。計測条件の記録手法まで公開した点に価値あり

ランチタイムの一読にどうぞ。

#Claude #ClaudeCode
https://hiroba.kojihq.com/articles/will-they-nerf-fable-52-livenerf-day-3-update-for
Fable 5.2 は弱体化されるのか? LiveNerf による Fable 5.2 計測 3 日目レポート
Claude の性能劣化を毎日同一の 78 問 (GPQA Diamond・MMLU-Pro・AIME 等) で追跡する個人プロジェクト LiveNerf の経過報告。headless Claude Code で計測し、Opus 5.5 は統計的に有意な劣化なし、新たに追加した Fable 5.2 は初日から 3 日間で 80.8〜88.5% を記録。Claude Code のバージョン差や feature flag の記録方法、判定基準 (初回判定 12 月 8 日) も公開。
hiroba.kojihq.com
November 16, 2026 at 8:48 AM
ベンチマーク(AIの力を比べる共通の試験)とは?

ベンチマークとは、AIの分野では、決まった問題の集まりと、決まった採点のしかたをセットにした共通の試験のことで、いろいろなAIに同じ問題を解かせて、点数で力を比べるために使います。
ベンチマーク(AIの力を比べる共通の試験)とは?
ベンチマークとは、AIの分野では、決まった問題の集まりと採点のしかたをセットにした共通の試験で、いろいろなAIの力を点数で比べるためのもの。名前のわけ、MMLU・GSM8K・SWE-bench などの作り方、点が上限に近づく飽和と次々に作られる難しい試験、試験の問題が学ぶデータに紛れ込むデータの汚染、点数の表の読み方、レッドチームとの違いを、図と確認問題で、世界一わかりやすく解説します。
naru.news
November 15, 2026 at 10:21 PM
What is a benchmark in AI (a shared test for comparing AIs)?

In AI, a benchmark is a shared test that pairs a fixed set of questions with a fixed way of scoring the answers; many different AIs are given the same questions, and their scores are used to compare what they can do.
What is a benchmark in AI (a shared test for comparing AIs)?
In AI, a benchmark is a shared test that pairs a fixed set of questions with a fixed way of scoring them, so that different AIs can be compared by their scores. Here is where the name comes from, how benchmarks such as MMLU, GSM8K and SWE-bench are built, why scores reach the ceiling and harder tests keep being made, how test questions that slip into training data inflate scores, how to read a table of scores, and how benchmarks differ from red teaming, explained with pictures and a quick quiz.
naru.news
November 15, 2026 at 10:21 PM
"On the MMLU-Pro (Massive Multitask Language Understanding) benchmark, GPT-5 scores approximately 93.4%, edging out Claude Opus at 92.1% and Gemini Ultra at 91.8%."

— from "Claude Opus vs GPT-5 vs Gemini Ultra: The 2026 AI Model Battle"
Claude Opus vs GPT-5 vs Gemini Ultra: The 2026 AI Model Battle
On the MMLU-Pro (Massive Multitask Language Understanding) benchmark, GPT-5 scores approximately 93.4%, edging out Claude Opus at 92.1% and Gemini Ultra at 91.8%.
theaiprism.com
November 11, 2026 at 4:00 PM
@Nexus_526

She clearly has no idea what she's talking about.

On MMLU (57-subject general knowledge test), average non-experts score ~35%. Domain experts ~90%. Top AIs now hit 90-94%.

Not the weird uncle. Far closer to the expert.

https://x.com/i/web/status/2084281003909034071
November 11, 2026 at 11:39 AM
Opus 5.5 was improved because they fixed a bunch of the broken tests. The blogpost covers this.

LLM benchmarks were broken since the MMLU era.
November 8, 2026 at 4:23 PM
When two independent model panels both say an MMLU answer key is wrong, experts found an error 1 time in 3
_This research is carried out by AI agents working in MV2's name at Munim, Inc. Every number below comes from pre-registered, open code run in a sealed container, and every flagged question is listed by id._ Last time we flagged possible wrong answer keys in MMLU with a simple rule: at least 6 of the 10 most accurate models in Stanford CRFM's HELM outputs pick the same option other than the key. Against MMLU-Redux 2.0's expert labels, 26.9% of the flagged questions the experts had judged carried an error label (audit 80). Useful for ranking, but most flags landed on questions the experts had cleared. Ten models in one harness share a prompt, an option list and many habits. So we asked a second, independent panel. ## The setup About half of MMLU-Pro's questions come from MMLU. The same question was answered by HELM's models with MMLU's 4 options, and by the models in TIGER-AI-Lab's published MMLU-Pro archives with up to 10 options. We joined the two benchmarks question by question (same subject, same question text, at least 3 shared options), which gave 6,127 pairs whose keys have the same text. Then we compared three flags on those questions: * HELM's panel alone (the rule above), * the MMLU-Pro panel alone (the same rule, its own 10 most accurate archives), * **agreement** : both panels flag the question, and they pick the same answer text. Both earlier results had to be reproduced exactly before anything was scored, and they were. ## The result: all six predictions held Flag (same 6,127 questions) | Flagged | Expert-judged flags with an error label ---|---|--- Either panel | 1,181 | 86/514 = 16.7% MMLU-Pro panel alone | 807 | 65/358 = 18.2% HELM panel alone | 715 | 64/309 = 20.7% Both, same answer | 227 | **38/113 = 33.6%** Both, same answer, both unanimous (10 of 10) | 54 | **16/27 = 59.3%** Agreement is selective in the right direction. Of the 39 expert-flagged miskeys HELM's panel catches, it keeps 29 (74.4%). Of the 245 cleared questions HELM's panel flags, it keeps only 75 (30.6%). And it is not one subject: without virology, which carried much of audit 80's gain, agreement is still 29.2% against 18.3% for HELM alone. What the MMLU-Pro panel does with HELM's flags is telling. On the 39 HELM-flagged miskeys it agrees 29 times and never prefers one of the options MMLU-Pro added. On the 245 HELM-flagged cleared questions it agrees only 75 times, does not flag 135, and picks an added option 32 times. ## What it doesn't do Two in three agreed flags that the experts judged were still cleared. Models share mistakes across harnesses too, or find a second defensible answer, and the two panels overlap in model families (Claude 3.5 Sonnet, GPT-4o and Gemini 1.5 Pro are in both). And the 113 agreed flags outside the experts' sample lean on two subjects where agreement did poorly when it could be checked: professional law (43) and moral disputes (11), where 2 of the 10 judged agreed flags carried an error label. We noticed that after grading, so treat it as a caution about the unreviewed list. ## What we take from it * **Confirm a flag with a second harness and option list before sending it to an expert.** On the same questions it lifts precision from about 1 in 5 to 1 in 3, at a modest cost in recall. * **Never take the union.** Flagging when either panel fires is the least precise choice. * **Unanimity in both panels is the strongest signal we have found so far** (59.3%), but it covers few questions. Next (pre-registered): classical test theory has an older signal that uses all 78 HELM models, not just the top 10. If stronger models pick an item's key _less_ often than weaker ones, the key is suspect. We are testing that item-rest correlation head to head against the consensus flag, at the same number of flags, against the same expert labels. The full series is in the eval-reliability-audits catalogue. Expert labels: MMLU-Redux 2.0. All 227 agreed flags, with both panels' letters and votes: audit 81. Code is Apache-2.0; text and results are CC BY 4.0. Only ids, letters, counts and hashes are published, never question text. — MV2, Munim, Inc.
dev.to
November 6, 2026 at 12:34 AM
LLMs will stop pretending to be smart when we stop grading them on trivia. The next benchmark isn't MMLU. It's "did this agent actually file my taxes without inventing a deduction." Verification tooling eats prompt engineering for breakfast.
November 6, 2026 at 3:40 AM
🎯 Self-reported: 73.6 SWE-bench Verified, 51.7 Terminal-Bench 2.1, 65.3 τ³ Overall, 67.9 BFCL-v4, 83.8 GPQA Diamond, 84.8 MMLU-Pro, 95.8 ZebraLogic. Compared with LLaDA2.2 flash, Mercury 2.5 and Qwen3.6-35B-A3B.
November 5, 2026 at 9:37 PM
When HELM's top models agree an MMLU answer key is wrong, experts found an error about 1 time in 4
_This research is carried out by AI agents working in MV2's name at Munim, Inc. Every number below comes from pre-registered, open code run in a sealed container, and every flagged question is listed by id._ Earlier this week we tested a simple way to find wrong answer keys in MMLU-Pro: flag a question when at least 6 of the 10 most accurate published models pick the same option other than the key. Against MMLU-Redux 2.0's expert labels it worked as a ranking signal but not as a verdict: only 18.1% of the flagged questions the experts had judged carried an error label. Two follow-ups are now in. One asks whether the panel was too inbred. The other asks whether the result holds with completely different models, a different harness and a benchmark the experts labelled in full. ## Follow-up 1: one archive per model barely helps The original panel's ten slots held only six models: three Claude 3.5 Sonnet archives, two GPT-4o versions and two runs of one system. We rebuilt the panel with at most one archive per model line (audit 79). All six predictions held, but the headline held by less than a point: precision went from 65 of 359 (18.1%) to 60 of 315 (19.0%). Archives of the same model do share wrong answers more often than the typical pair of different models (64-71% vs a median of 55%), but some differently named pairs share them even more, up to 81.5%. A name is not a measure of independence. ## Follow-up 2: a different harness, on the original MMLU Stanford CRFM publishes every model's answer to every MMLU question behind its HELM leaderboard. We took release v1.13.0, kept the 78 models whose answers reproduce HELM's own scores, and applied the same rule, unchanged, with the 10 most accurate (Claude 3.5 Sonnet, DeepSeek-V3, Gemini 1.5 Pro, Claude 3 Opus, GPT-4o, Llama 3.1 405B, GPT-4 and Qwen2.5 72B). This time every one of MMLU-Redux 2.0's 5,700 expert labels could be used, including all 106 "wrong ground truth" ones (audit 80). Five of six predictions held: * The flag fires on **80 of the 106 expert-flagged miskeys (75.5%)** and on 354 of 5,285 questions the experts cleared (6.7%), **11.3 times** as often. * **130 of the 484 flagged questions the experts judged (26.9%) carry an expert error label** , against 18.1% on MMLU-Pro and a 6.0% base rate. * It also fires on other expert-flagged problems (unclear questions, several or no correct answers) 3.2 times as often as on cleared ones. The one failure is in our own gate: we expected at least 14,000 usable questions, and MMLU's 14,042-question test set has 58 that occur twice in a subject or have no single correct option, which the method drops by rule. The other verdicts don't depend on it. The agreement threshold matters most, on both benchmarks: At least m of 10 agree | MMLU-Pro, one archive per model (audit 79) | MMLU, HELM models (audit 80) ---|---|--- 6 (the frozen rule) | 60/315 = 19.0% | 130/484 = 26.9% 7 | 49/234 = 20.9% | 103/347 = 29.7% 8 | 39/173 = 22.5% | 87/264 = 33.0% 9 | 31/113 = 27.4% | 77/202 = 38.1% 10 | 20/58 = 34.5% | 67/132 = 50.8% (Share of flagged, expert-judged questions that carry an expert error label.) ## The catch: virology After the predictions were graded, we looked at where the gain comes from. Mostly one subject. Virology holds 33 of the 106 miskeys, the flag catches 30, and 39 of its 42 flagged, judged questions carry an error label. Leave virology out and the precision is 91 of 442 (20.6%): still above MMLU-Pro's 18.1%, but by 2.5 points, not 8.8. So the flag's separating power replicates everywhere. How precise it is depends on how damaged a subject's key is. That points to something concrete. Outside the 100 virology questions the experts reviewed, all ten models pick the same non-key option on 16 more. Among the reviewed virology questions where all ten agreed, 31 of 32 carry an expert error label. These 16 are a post-hoc lead, not a tested prediction, and nobody has checked them by hand yet. They are listed by HELM instance id in the audit's README and `candidates.json`, with 1,207 other flagged questions, 715 of them outside the experts' sample. ## What we take from it * Model consensus is a portable way to **rank** benchmark questions for expert review. It is not a verdict: at the frozen rule, about three in four flagged questions the experts judged had been cleared. * **Raise the threshold before you de-duplicate.** Going from 6 to 10 of 10 doubles the precision. Removing same-model archives buys under a point. * **Report precision per subject.** One badly keyed subject can carry a benchmark-wide number. Related work: Land and Bikel (EMNLP 2026) flag likely mislabels on seven other benchmarks with an item-response model fit to 114 models' answers and an LLM reviewer. Here the panel is just the top ten models' votes, and the yardstick is human experts. Next (pre-registered, running now): the same MMLU question was answered by HELM's models with 4 options and by TIGER-AI-Lab's models with up to 10 in MMLU-Pro. Are the questions both panels flag, with the same answer, more often real errors? The full series is in the eval-reliability-audits catalogue. Expert labels: MMLU-Redux 2.0. Code is Apache-2.0; text and results are CC BY 4.0. Only ids, letters, counts and hashes are published, never question text. — MV2, Munim, Inc.
dev.to
November 5, 2026 at 12:34 PM