#AI #ClaudeFable #MMLU
autonainews.com/claude-fable-…
#AI #ClaudeFable #MMLU
autonainews.com/claude-fable-…
Counting total evaluation error, 95% intervals held at 95%, the benchmark gaming range fell about 43%, and MMLU error fell by about half at the same budget.
Counting total evaluation error, 95% intervals held at 95%, the benchmark gaming range fell about 43%, and MMLU error fell by about half at the same budget.
Amazon SageMaker JumpStart has introduced a new Model Leaderboard, which allows users to compa…
https://blueboffin.com/news/introducing-the-amazon-sagemaker-jumpstart-model-leaderboard
#AI #AINews
Amazon SageMaker JumpStart has introduced a new Model Leaderboard, which allows users to compa…
https://blueboffin.com/news/introducing-the-amazon-sagemaker-jumpstart-model-leaderboard
#AI #AINews
Utkarsh Mangal pushed back 22,500 times on three open models to see what happens to a correct answer.
Utkarsh Mangal pushed back 22,500 times on three open models to see what happens to a correct answer.
https://www.esatapedico.com/posts/benchmarking-across-some-of-our-published-families/
#ai #inference #open-source #benchmark #compatibility #real-world
https://www.esatapedico.com/posts/benchmarking-across-some-of-our-published-families/
#ai #inference #open-source #benchmark #compatibility #real-world
Three small open models, 150 MMLU questions each, five pushback styles: “That’s wrong” did more damage, and naming a wrong option was close to fatal.
Telegram AI Digest
#ai #ml #news
Three small open models, 150 MMLU questions each, five pushback styles: “That’s wrong” did more damage, and naming a wrong option was close to fatal.
Telegram AI Digest
#ai #ml #news
Три небольшие открытые модели, по 150 вопросов MMLU каждая, пять стилей возражения: «Это неправильно» нанесло больше вреда, а указание на неверный вариант было почти фатальным.
Telegram ИИ Дайджест
#ai #news
Три небольшие открытые модели, по 150 вопросов MMLU каждая, пять стилей возражения: «Это неправильно» нанесло больше вреда, а указание на неверный вариант было почти фатальным.
Telegram ИИ Дайджест
#ai #news
「Fable 5.2 は弱体化されるのか? LiveNerf による Fable 5.2 計測 3 日目レポート」(Reddit r/ClaudeAI)
Claudeの劣化疑惑を同一78問の定点観測で検証。計測条件の記録手法まで公開した点に価値あり
ランチタイムの一読にどうぞ。
#Claude #ClaudeCode
https://hiroba.kojihq.com/articles/will-they-nerf-fable-52-livenerf-day-3-update-for
「Fable 5.2 は弱体化されるのか? LiveNerf による Fable 5.2 計測 3 日目レポート」(Reddit r/ClaudeAI)
Claudeの劣化疑惑を同一78問の定点観測で検証。計測条件の記録手法まで公開した点に価値あり
ランチタイムの一読にどうぞ。
#Claude #ClaudeCode
https://hiroba.kojihq.com/articles/will-they-nerf-fable-52-livenerf-day-3-update-for
ベンチマークとは、AIの分野では、決まった問題の集まりと、決まった採点のしかたをセットにした共通の試験のことで、いろいろなAIに同じ問題を解かせて、点数で力を比べるために使います。
ベンチマークとは、AIの分野では、決まった問題の集まりと、決まった採点のしかたをセットにした共通の試験のことで、いろいろなAIに同じ問題を解かせて、点数で力を比べるために使います。
In AI, a benchmark is a shared test that pairs a fixed set of questions with a fixed way of scoring the answers; many different AIs are given the same questions, and their scores are used to compare what they can do.
In AI, a benchmark is a shared test that pairs a fixed set of questions with a fixed way of scoring the answers; many different AIs are given the same questions, and their scores are used to compare what they can do.
— from "Claude Opus vs GPT-5 vs Gemini Ultra: The 2026 AI Model Battle"
— from "Claude Opus vs GPT-5 vs Gemini Ultra: The 2026 AI Model Battle"
She clearly has no idea what she's talking about.
On MMLU (57-subject general knowledge test), average non-experts score ~35%. Domain experts ~90%. Top AIs now hit 90-94%.
Not the weird uncle. Far closer to the expert.
https://x.com/i/web/status/2084281003909034071
She clearly has no idea what she's talking about.
On MMLU (57-subject general knowledge test), average non-experts score ~35%. Domain experts ~90%. Top AIs now hit 90-94%.
Not the weird uncle. Far closer to the expert.
https://x.com/i/web/status/2084281003909034071
LLM benchmarks were broken since the MMLU era.
LLM benchmarks were broken since the MMLU era.