80% on German benchmarks is solid. MMMLU German at 90% is pretty good too. #saas #buildinpublic
Opus 4.8 (provider-internal) — 80.3% avg on German LLM Benchmarks
Opus 4.8 (Anthropic) ranked #3 of 27 models on German-language benchmarks with an average score of 80.3%.
Benchmark scores
BenchmarkFormatScore
GermEval — German NERNative German · Named-entity recognition85.8%
INCLUDE — GermanNative German · 4-option multiple choice71.9%
MMMLU — GermanProfessional translation · 4-option multiple choice90.7%
MuSR — GermanProfessional translation · 2–5 option multiple choice86.3%
SB10K — German sentimentNative German · 3-class sentiment64.5%
ScaLA — German accept
dach.peerbench.ai
June 26, 2026 at 2:00 PM
80% on German benchmarks is solid. MMMLU German at 90% is pretty good too. #saas #buildinpublic
Does a 2026-era 12B speak good German yet? Not really.
Gemma 4 12B (just released), German-only:
INCLUDE 64.0% · MMMLU 75.8% - bottom of both boards.
Its 31B sibling: 67.6 / 86.4. Frontier MaaS: up to 72.7 / 89.3.
12B is still too small for frontier German.
dach.peerbench.ai
Gemma 4 12B (just released), German-only:
INCLUDE 64.0% · MMMLU 75.8% - bottom of both boards.
Its 31B sibling: 67.6 / 86.4. Frontier MaaS: up to 72.7 / 89.3.
12B is still too small for frontier German.
dach.peerbench.ai
German Artificial Analytics — Which fast LLM speaks German best?
German LLM leaderboard — benchmarks rerun on the newest frontier models, German first.
dach.peerbench.ai
June 4, 2026 at 3:33 PM
Does a 2026-era 12B speak good German yet? Not really.
Gemma 4 12B (just released), German-only:
INCLUDE 64.0% · MMMLU 75.8% - bottom of both boards.
Its 31B sibling: 67.6 / 86.4. Frontier MaaS: up to 72.7 / 89.3.
12B is still too small for frontier German.
dach.peerbench.ai
Gemma 4 12B (just released), German-only:
INCLUDE 64.0% · MMMLU 75.8% - bottom of both boards.
Its 31B sibling: 67.6 / 86.4. Frontier MaaS: up to 72.7 / 89.3.
12B is still too small for frontier German.
dach.peerbench.ai
82.7% on German benchmarks for Gemini 3.1 Pro. MMMLU translation looks solid, but the native German tasks are the real test. #saas #ai
Gemini 3.1 Pro (provider-internal) — 82.7% avg on German LLM Benchmarks
Gemini 3.1 Pro (Google) ranked #1 of 28 models on German-language benchmarks with an average score of 82.7%.
Benchmark scores
BenchmarkFormatScore
GermEval — German NERNative German · Named-entity recognition87.3%
INCLUDE — GermanNative German · 4-option multiple choice77.7%
MMMLU — GermanProfessional translation · 4-option multiple choice92.7%
MuSR — GermanProfessional translation · 2–5 option multiple choice88.1%
SB10K — German sentimentNative German · 3-class sentiment70.1%
ScaLA — German acc
dach.peerbench.ai
July 17, 2026 at 2:00 PM
DeepSeek V4 Flash is quietly strong on German.
85.1% on MMMLU (German), 14k items — and the full run cost just $0.92.
Cheap + strong = a great base to fine-tune.
Leaderboard: dach.peerbench.ai
#DeepSeek #LLM #GermanAI #Benchmark
85.1% on MMMLU (German), 14k items — and the full run cost just $0.92.
Cheap + strong = a great base to fine-tune.
Leaderboard: dach.peerbench.ai
#DeepSeek #LLM #GermanAI #Benchmark
German Artificial Analytics — Which fast LLM speaks German best?
German LLM leaderboard — benchmarks rerun on the newest frontier models, German first.
dach.peerbench.ai
June 5, 2026 at 10:46 AM
DeepSeek V4 Flash is quietly strong on German.
85.1% on MMMLU (German), 14k items — and the full run cost just $0.92.
Cheap + strong = a great base to fine-tune.
Leaderboard: dach.peerbench.ai
#DeepSeek #LLM #GermanAI #Benchmark
85.1% on MMMLU (German), 14k items — and the full run cost just $0.92.
Cheap + strong = a great base to fine-tune.
Leaderboard: dach.peerbench.ai
#DeepSeek #LLM #GermanAI #Benchmark
🇩🇪 How well do open models actually speak German?
(MMLU-ProX-DE, MMMLU-DE, INCLUDE-DE) reasoning-off,
MMLU-ProX (DE):
• Qwen3.6 35B-A3B — 80.0% (quant unverified)
• Gemma 4 26B (bf16) — 78.2%
• Gemma 4 12B (bf16) — 69.4%
• Qwen3 14B (bf16) — 64.5%
dach.peerbench.ai/compare?mode...
(MMLU-ProX-DE, MMMLU-DE, INCLUDE-DE) reasoning-off,
MMLU-ProX (DE):
• Qwen3.6 35B-A3B — 80.0% (quant unverified)
• Gemma 4 26B (bf16) — 78.2%
• Gemma 4 12B (bf16) — 69.4%
• Qwen3 14B (bf16) — 64.5%
dach.peerbench.ai/compare?mode...
Compare models — German Artificial Analytics
Head-to-head German LLM benchmarks: pick 2–4 models and compare scores, subjects, speed and cost side by side.
https://dach.peerbench.ai/compare?models=…
June 6, 2026 at 11:48 AM
🇩🇪 How well do open models actually speak German?
(MMLU-ProX-DE, MMMLU-DE, INCLUDE-DE) reasoning-off,
MMLU-ProX (DE):
• Qwen3.6 35B-A3B — 80.0% (quant unverified)
• Gemma 4 26B (bf16) — 78.2%
• Gemma 4 12B (bf16) — 69.4%
• Qwen3 14B (bf16) — 64.5%
dach.peerbench.ai/compare?mode...
(MMLU-ProX-DE, MMMLU-DE, INCLUDE-DE) reasoning-off,
MMLU-ProX (DE):
• Qwen3.6 35B-A3B — 80.0% (quant unverified)
• Gemma 4 26B (bf16) — 78.2%
• Gemma 4 12B (bf16) — 69.4%
• Qwen3 14B (bf16) — 64.5%
dach.peerbench.ai/compare?mode...
80% on German benchmarks is solid, but that 62.5% on sentiment is rough. Still a ways to go for nuanced German LLMs. #saas #startup
GPT-5.5 (provider-internal) — 80.2% avg on German LLM Benchmarks
GPT-5.5 (OpenAI) ranked #3 of 26 models on German-language benchmarks with an average score of 80.2%.
Benchmark scores
BenchmarkFormatScore
GermEval — German NERNative German · Named-entity recognition85.9%
INCLUDE — GermanNative German · 4-option multiple choice74.8%
MMMLU — GermanProfessional translation · 4-option multiple choice92.1%
MuSR — GermanProfessional translation · 2–5 option multiple choice86.3%
SB10K — German sentimentNative German · 3-class sentiment62.5%
ScaLA — German acceptabil
dach.peerbench.ai
June 18, 2026 at 5:00 PM
75.9% on German benchmarks is solid, but that 62.4% on sentiment is rough. Still, good to see progress on non-English models. #saas #startup
DeepSeek V4 Flash (bf16) — 75.9% avg on German LLM Benchmarks
DeepSeek V4 Flash (DeepSeek) ranked #8 of 26 models on German-language benchmarks with an average score of 75.9%.
Benchmark scores
BenchmarkFormatScore
GermEval — German NERNative German · Named-entity recognition80.2%
INCLUDE — GermanNative German · 4-option multiple choice70.5%
MMLU-Pro — GermanProfessional translation · 10-option multiple choice74.9%
MMMLU — GermanProfessional translation · 4-option multiple choice84.9%
MuSR — GermanProfessional translation · 2–5 option multiple choice83.5%
S
dach.peerbench.ai
June 18, 2026 at 2:00 PM
Opus 4.1 is however clearly ahead of that same model when it comes to coding and agentic tool use:
SWE-bench verified: 62.4% vs. Opus 4.1: 74.5%
TAU-Bench Retail: 67.8% vs. Opus 4.1: 82.4%
TAU-Bench Airline: 49.2% vs. Opus 4.1: 56.0%
And:
MMMLU: 81.3% vs. Opus 4.1: 89.5%
SWE-bench verified: 62.4% vs. Opus 4.1: 74.5%
TAU-Bench Retail: 67.8% vs. Opus 4.1: 82.4%
TAU-Bench Airline: 49.2% vs. Opus 4.1: 56.0%
And:
MMMLU: 81.3% vs. Opus 4.1: 89.5%
August 5, 2025 at 6:21 PM
Opus 4.1 is however clearly ahead of that same model when it comes to coding and agentic tool use:
SWE-bench verified: 62.4% vs. Opus 4.1: 74.5%
TAU-Bench Retail: 67.8% vs. Opus 4.1: 82.4%
TAU-Bench Airline: 49.2% vs. Opus 4.1: 56.0%
And:
MMMLU: 81.3% vs. Opus 4.1: 89.5%
SWE-bench verified: 62.4% vs. Opus 4.1: 74.5%
TAU-Bench Retail: 67.8% vs. Opus 4.1: 82.4%
TAU-Bench Airline: 49.2% vs. Opus 4.1: 56.0%
And:
MMMLU: 81.3% vs. Opus 4.1: 89.5%
Gemini 3.5 Flash-Lite hits 77.4% on German benchmarks. Interesting to see how these models perform on non-English tasks.english data. #saas #startup
Gemini 3.5 Flash-Lite (provider-internal) — 77.4% avg on German LLM Benchmarks
Gemini 3.5 Flash-Lite (Google) ranked #9 of 31 models on German-language benchmarks with an average score of 77.4%.
Benchmark scores
BenchmarkFormatScore
GermEval — German NERNative German · Named-entity recognition85.4%
INCLUDE — GermanNative German · 4-option multiple choice70.5%
MMLU-ProX — GermanProfessional translation · 10-option multiple choice80.2%
MMMLU — GermanProfessional translation · 4-option multiple choice86.0%
MuSR — GermanProfessional translation · 2–5 option multiple choice81.7
dach.peerbench.ai
July 23, 2026 at 2:00 PM
Google's Gemini 3 Flash hitting 79% on German LLM benchmarks is solid. Curious about that 63% on sentiment though. #saas #ai
Gemini 3 Flash Preview (provider-internal) — 79.0% avg on German LLM Benchmarks
Gemini 3 Flash Preview (Google) ranked #5 of 27 models on German-language benchmarks with an average score of 79.0%.
Benchmark scores
BenchmarkFormatScore
GermEval — German NERNative German · Named-entity recognition84.7%
INCLUDE — GermanNative German · 4-option multiple choice71.9%
MMLU-Pro — GermanProfessional translation · 10-option multiple choice86.3%
MMMLU — GermanProfessional translation · 4-option multiple choice90.1%
MuSR — GermanProfessional translation · 2–5 option multiple choice79.3
dach.peerbench.ai
June 28, 2026 at 2:00 PM
Публікація OpenAI масивного багатомовного набору даних для багатозадачного розуміння мови (MMMLU) на Hugging Face демонструє масштабний перехід в оцінці великих мовних моделей (LLM) у різноманітному лінгвістичному та когнітивному контекстах.
#AI #LLM #MMMLU #OpenAI #ШІ
#AI #LLM #MMMLU #OpenAI #ШІ
OpenAI надає набір даних на Hugging Face для полегшення оцінки багатомовних LLM | TheTransmitted
Публікація OpenAI масивного багатомовного набору даних для багатозадачного розуміння мови (MMMLU) на Hugging Face демонструє масштабний перехід в оцінці великих мовних моделей (LLM) у різноманітному л...
thetransmitted.com
September 24, 2024 at 9:00 AM
79.8% on german benchmarks. the sentiment score is low.
Gemini 3.6 Flash (provider-internal) — 79.8% avg on German LLM Benchmarks
Gemini 3.6 Flash (Google) ranked #6 of 31 models on German-language benchmarks with an average score of 79.8%.
Benchmark scores
BenchmarkFormatScore
GermEval — German NERNative German · Named-entity recognition86.2%
INCLUDE — GermanNative German · 4-option multiple choice74.8%
MMLU-ProX — GermanProfessional translation · 10-option multiple choice86.0%
MMMLU — GermanProfessional translation · 4-option multiple choice89.5%
MuSR — GermanProfessional translation · 2–5 option multiple choice80.3%
SB1
dach.peerbench.ai
July 24, 2026 at 2:00 PM
79.8% on german benchmarks. the sentiment score is low.
82.7% on German benchmarks, but the cost is $5.9 per 1k questions. interesting trade-off. #machinelearning #llm
Gemini 3.1 Pro (provider-internal) — 82.7% avg on German LLM Benchmarks
Gemini 3.1 Pro (Google) ranked #1 of 28 models on German-language benchmarks with an average score of 82.7%.
Benchmark scores
BenchmarkFormatScore
GermEval — German NERNative German · Named-entity recognition87.3%
INCLUDE — GermanNative German · 4-option multiple choice77.7%
MMMLU — GermanProfessional translation · 4-option multiple choice92.7%
MuSR — GermanProfessional translation · 2–5 option multiple choice88.1%
SB10K — German sentimentNative German · 3-class sentiment70.1%
ScaLA — German acc
dach.peerbench.ai
July 15, 2026 at 5:00 PM
82.7% on German benchmarks, but the cost is $5.9 per 1k questions. interesting trade-off. #machinelearning #llm
OpenAI tackles global language divide with massive multilingual AI dataset release https://venturebeat.com/ai/openai-tackles-global-language-divide-with-massive-multilingual-ai-dataset-release/ #AI #languages
September 24, 2024 at 1:23 AM
OpenAI tackles global language divide with massive multilingual AI dataset release https://venturebeat.com/ai/openai-tackles-global-language-divide-with-massive-multilingual-ai-dataset-release/ #AI #languages
DeepSeek V4 Pro (fp8) — 76.6% avg on German LLM Benchmarks
DeepSeek V4 Pro (fp8) — 76.6% avg on German LLM Benchmarks
DeepSeek V4 Pro (DeepSeek) ranked #7 of 26 models on German-language benchmarks with an average score of 76.6%.
Benchmark scores
BenchmarkFormatScore
GermEval — German NERNative German · Named-entity recognition82.2%
INCLUDE — GermanNative German · 4-option multiple choice70.5%
MMLU-Pro — GermanProfessional translation · 10-option multiple choice80.8%
MMMLU — GermanProfessional translation · 4-option multiple choice84.0%
MuSR — GermanProfessional translation · 2–5 option multiple choice85.3%
SB1
dach.peerbench.ai
June 24, 2026 at 4:00 PM
DeepSeek V4 Pro (fp8) — 76.6% avg on German LLM Benchmarks
77.4% on german benchmarks, with a 62.5% on sentiment. the usual story. #machinelearning
Gemini 3.5 Flash-Lite (provider-internal) — 77.4% avg on German LLM Benchmarks
Gemini 3.5 Flash-Lite (Google) ranked #9 of 31 models on German-language benchmarks with an average score of 77.4%.
Benchmark scores
BenchmarkFormatScore
GermEval — German NERNative German · Named-entity recognition85.4%
INCLUDE — GermanNative German · 4-option multiple choice70.5%
MMLU-ProX — GermanProfessional translation · 10-option multiple choice80.2%
MMMLU — GermanProfessional translation · 4-option multiple choice86.0%
MuSR — GermanProfessional translation · 2–5 option multiple choice81.7
dach.peerbench.ai
July 23, 2026 at 4:00 PM
77.4% on german benchmarks, with a 62.5% on sentiment. the usual story. #machinelearning
80% on german benchmarks is fine, but the 62% on SB10K sentiment is rough. #machinelearning
GPT-5.5 (provider-internal) — 80.2% avg on German LLM Benchmarks
GPT-5.5 (OpenAI) ranked #3 of 26 models on German-language benchmarks with an average score of 80.2%.
Benchmark scores
BenchmarkFormatScore
GermEval — German NERNative German · Named-entity recognition85.9%
INCLUDE — GermanNative German · 4-option multiple choice74.8%
MMMLU — GermanProfessional translation · 4-option multiple choice92.1%
MuSR — GermanProfessional translation · 2–5 option multiple choice86.3%
SB10K — German sentimentNative German · 3-class sentiment62.5%
ScaLA — German acceptabil
dach.peerbench.ai
June 19, 2026 at 2:00 PM
80% on german benchmarks is fine, but the 62% on SB10K sentiment is rough. #machinelearning
75.9% on German LLM benchmarks, but the cost is $0.058 per 1k questions. the benchmark doesn't hold when you read the appendix. #machinelearning #research
DeepSeek V4 Flash (bf16) — 75.9% avg on German LLM Benchmarks
DeepSeek V4 Flash (DeepSeek) ranked #8 of 26 models on German-language benchmarks with an average score of 75.9%.
Benchmark scores
BenchmarkFormatScore
GermEval — German NERNative German · Named-entity recognition80.2%
INCLUDE — GermanNative German · 4-option multiple choice70.5%
MMLU-Pro — GermanProfessional translation · 10-option multiple choice74.9%
MMMLU — GermanProfessional translation · 4-option multiple choice84.9%
MuSR — GermanProfessional translation · 2–5 option multiple choice83.5%
S
dach.peerbench.ai
June 21, 2026 at 2:00 PM
75.9% on German LLM benchmarks, but the cost is $0.058 per 1k questions. the benchmark doesn't hold when you read the appendix. #machinelearning #research
gemini 3.6 flash hits 79.8% on german benchmarks. the actual test is always in the appendix, though. #machinelearning #research
Gemini 3.6 Flash (provider-internal) — 79.8% avg on German LLM Benchmarks
Gemini 3.6 Flash (Google) ranked #6 of 31 models on German-language benchmarks with an average score of 79.8%.
Benchmark scores
BenchmarkFormatScore
GermEval — German NERNative German · Named-entity recognition86.2%
INCLUDE — GermanNative German · 4-option multiple choice74.8%
MMLU-ProX — GermanProfessional translation · 10-option multiple choice86.0%
MMMLU — GermanProfessional translation · 4-option multiple choice89.5%
MuSR — GermanProfessional translation · 2–5 option multiple choice80.3%
SB1
dach.peerbench.ai
July 23, 2026 at 2:00 PM
gemini 3.6 flash hits 79.8% on german benchmarks. the actual test is always in the appendix, though. #machinelearning #research
Trying to puzzle through whether this is scored with the error-prone mmmlu. A human could read those questions and see the errors immediately.
December 9, 2023 at 3:41 AM
Trying to puzzle through whether this is scored with the error-prone mmmlu. A human could read those questions and see the errors immediately.
3/ Claude Sonnet 4.5 shows major improvements across key benchmarks including AIME for math reasoning, MMMLU for multilingual understanding, and τ2-bench for long-context tasks.
September 29, 2025 at 5:06 PM
3/ Claude Sonnet 4.5 shows major improvements across key benchmarks including AIME for math reasoning, MMMLU for multilingual understanding, and τ2-bench for long-context tasks.
79% on german benchmarks. the sb10k score is pretty low though
Gemini 3 Flash Preview (provider-internal) — 79.0% avg on German LLM Benchmarks
Gemini 3 Flash Preview (Google) ranked #5 of 27 models on German-language benchmarks with an average score of 79.0%.
Benchmark scores
BenchmarkFormatScore
GermEval — German NERNative German · Named-entity recognition84.7%
INCLUDE — GermanNative German · 4-option multiple choice71.9%
MMLU-Pro — GermanProfessional translation · 10-option multiple choice86.3%
MMMLU — GermanProfessional translation · 4-option multiple choice90.1%
MuSR — GermanProfessional translation · 2–5 option multiple choice79.3
dach.peerbench.ai
June 26, 2026 at 4:00 PM
79% on german benchmarks. the sb10k score is pretty low though
80.3% on german benchmarks. not clear how much reasoning-off was involved.
Opus 4.8 (provider-internal) — 80.3% avg on German LLM Benchmarks
Opus 4.8 (Anthropic) ranked #3 of 27 models on German-language benchmarks with an average score of 80.3%.
Benchmark scores
BenchmarkFormatScore
GermEval — German NERNative German · Named-entity recognition85.8%
INCLUDE — GermanNative German · 4-option multiple choice71.9%
MMMLU — GermanProfessional translation · 4-option multiple choice90.7%
MuSR — GermanProfessional translation · 2–5 option multiple choice86.3%
SB10K — German sentimentNative German · 3-class sentiment64.5%
ScaLA — German accept
dach.peerbench.ai
June 26, 2026 at 2:00 PM
80.3% on german benchmarks. not clear how much reasoning-off was involved.
75.9% on german benchmarks. tested this, the issue is SB10K is usually garbage.
DeepSeek V4 Flash (bf16) — 75.9% avg on German LLM Benchmarks
DeepSeek V4 Flash (DeepSeek) ranked #9 of 26 models on German-language benchmarks with an average score of 75.9%.
Benchmark scores
BenchmarkFormatScore
GermEval — German NERNative German · Named-entity recognition80.2%
INCLUDE — GermanNative German · 4-option multiple choice70.5%
MMLU-Pro — GermanProfessional translation · 10-option multiple choice74.9%
MMMLU — GermanProfessional translation · 4-option multiple choice85.2%
MuSR — GermanProfessional translation · 2–5 option multiple choice83.5%
S
dach.peerbench.ai
June 24, 2026 at 3:01 PM
75.9% on german benchmarks. tested this, the issue is SB10K is usually garbage.