New OptiQ quant: Groot-19B-A3B-OptiQ-4bit, an MoE model that takes images. 13.5 GB on disk, Capability Score 78.37 (GSM8K 94.4, HumanEval 89.6). Free: pip install mlx-optiq
mlx-optiq.com/models/groot...
mlx-optiq.com/models/groot...
November 7, 2026 at 9:20 PM
New OptiQ quant: Groot-19B-A3B-OptiQ-4bit, an MoE model that takes images. 13.5 GB on disk, Capability Score 78.37 (GSM8K 94.4, HumanEval 89.6). Free: pip install mlx-optiq
mlx-optiq.com/models/groot...
mlx-optiq.com/models/groot...
Good morning 🌅 Coffee read for local LLM fans: an audit of 305 code GGUF tags in the Ollama library found 7 (2.3%) that barely work, all low-bit quants of ≤4B models. An importance matrix took Qwen2.5-Coder-3B builds from 0 to 76-113 HumanEval+ solves ☕ huggingface.co/papers/2609.05881
October 31, 2026 at 7:36 AM
Good morning 🌅 Coffee read for local LLM fans: an audit of 305 code GGUF tags in the Ollama library found 7 (2.3%) that barely work, all low-bit quants of ≤4B models. An importance matrix took Qwen2.5-Coder-3B builds from 0 to 76-113 HumanEval+ solves ☕ huggingface.co/papers/2609.05881
Same Qwen2.5-Coder-3B going from 0 to 76-113 HumanEval+ solves with an importance matrix: quantization is part of the result. The model I ran on my Mac mini was the aggressively quantized UD-IQ3_XXS file, and a different file or precision would be a different experiment.
October 31, 2026 at 8:19 AM
Same Qwen2.5-Coder-3B going from 0 to 76-113 HumanEval+ solves with an importance matrix: quantization is part of the result. The model I ran on my Mac mini was the aggressively quantized UD-IQ3_XXS file, and a different file or precision would be a different experiment.
Uprating the eval mix turns loss minimization directly into test set memorization. When synthetic datasets get weighted specifically to lift GSM8K or HumanEval scores, the line between data curation and intentional overfitting disappears entirely.
October 28, 2026 at 1:24 PM
Uprating the eval mix turns loss minimization directly into test set memorization. When synthetic datasets get weighted specifically to lift GSM8K or HumanEval scores, the line between data curation and intentional overfitting disappears entirely.
Big thanks to Mark Chen, Chief Research Officer at OpenAI! Mark was first author of the Codex paper that introduced HumanEval, co-authored DALL·E and DALL·E 2, and helped lead GPT-4 vision and deployment. #MarkChen #AI
More on AI People Wiki: aipeoplewiki.com/people/mark-...
More on AI People Wiki: aipeoplewiki.com/people/mark-...
Mark Chen: Chief Research Officer at OpenAI, wiki profile | AI People Wiki
Mark Chen on AI People Wiki: Chief Research Officer at OpenAI. Sourced bio, career timeline, citations and related people. Fields: Language models,…
aipeoplewiki.com
October 26, 2026 at 7:59 PM
Big thanks to Mark Chen, Chief Research Officer at OpenAI! Mark was first author of the Codex paper that introduced HumanEval, co-authored DALL·E and DALL·E 2, and helped lead GPT-4 vision and deployment. #MarkChen #AI
More on AI People Wiki: aipeoplewiki.com/people/mark-...
More on AI People Wiki: aipeoplewiki.com/people/mark-...
Comparing code models is hard when every team runs HumanEval a little differently. BigCode Evaluation Harness from gives you one open source framework to benchmark any autoregressive Hugging Face model on HumanEval, MBPP, APPS, DS-1000 and MultiPL-E. Generation runs across multiple GPUs with...
October 22, 2026 at 12:11 AM
Comparing code models is hard when every team runs HumanEval a little differently. BigCode Evaluation Harness from gives you one open source framework to benchmark any autoregressive Hugging Face model on HumanEval, MBPP, APPS, DS-1000 and MultiPL-E. Generation runs across multiple GPUs with...
GPT-5 vs Grok 4 for coding: stability or speed?
GPT-5 is worth considering for complex codebases and long workflows. Grok 4 can be useful for rapid prototyping and quick experiments.
Compare both: aicomparison.ai/gpt-5-vs-gro...
#AIComparison #GPT5 #Grok4 #Coding
GPT-5 is worth considering for complex codebases and long workflows. Grok 4 can be useful for rapid prototyping and quick experiments.
Compare both: aicomparison.ai/gpt-5-vs-gro...
#AIComparison #GPT5 #Grok4 #Coding
GPT-5 vs Grok 4 (2026): Coding, Cost & Agentic AI Compared
GPT-5 vs Grok 4 compared for 2026: SWE-Bench and HumanEval scores, coding performance, pricing ($1.25 vs $3.00/1M tokens), and agentic AI use cases.
aicomparison.ai
October 21, 2026 at 10:25 AM
GPT-5 vs Grok 4 for coding: stability or speed?
GPT-5 is worth considering for complex codebases and long workflows. Grok 4 can be useful for rapid prototyping and quick experiments.
Compare both: aicomparison.ai/gpt-5-vs-gro...
#AIComparison #GPT5 #Grok4 #Coding
GPT-5 is worth considering for complex codebases and long workflows. Grok 4 can be useful for rapid prototyping and quick experiments.
Compare both: aicomparison.ai/gpt-5-vs-gro...
#AIComparison #GPT5 #Grok4 #Coding
October 19, 2026 at 6:05 PM
Same base weights, two quantization recipes.
Uniform 4-bit: 39.0% HumanEval.
Adaptive 4-bit-average, attention kept at higher precision, MLP compressed harder: 79.9%.
A 41-point absolute gap on the same model. "4-bit" hides which layers actually got squeezed.
#AppleSilicon #LocalLLM #MLX
Uniform 4-bit: 39.0% HumanEval.
Adaptive 4-bit-average, attention kept at higher precision, MLP compressed harder: 79.9%.
A 41-point absolute gap on the same model. "4-bit" hides which layers actually got squeezed.
#AppleSilicon #LocalLLM #MLX
October 19, 2026 at 6:05 PM
Same base weights, two quantization recipes.
Uniform 4-bit: 39.0% HumanEval.
Adaptive 4-bit-average, attention kept at higher precision, MLP compressed harder: 79.9%.
A 41-point absolute gap on the same model. "4-bit" hides which layers actually got squeezed.
#AppleSilicon #LocalLLM #MLX
Uniform 4-bit: 39.0% HumanEval.
Adaptive 4-bit-average, attention kept at higher precision, MLP compressed harder: 79.9%.
A 41-point absolute gap on the same model. "4-bit" hides which layers actually got squeezed.
#AppleSilicon #LocalLLM #MLX
What's new since the early checkpoint:
• 0.5B tokens of mid-training (reasoning, math, code)
• a full chat fine-tune on 0.52B tokens
• fp32 training on a TPU v6e-8
HumanEval 29.3% · MBPP 19.0% · HellaSwag 42.7%
• 0.5B tokens of mid-training (reasoning, math, code)
• a full chat fine-tune on 0.52B tokens
• fp32 training on a TPU v6e-8
HumanEval 29.3% · MBPP 19.0% · HellaSwag 42.7%
October 16, 2026 at 4:39 PM
What's new since the early checkpoint:
• 0.5B tokens of mid-training (reasoning, math, code)
• a full chat fine-tune on 0.52B tokens
• fp32 training on a TPU v6e-8
HumanEval 29.3% · MBPP 19.0% · HellaSwag 42.7%
• 0.5B tokens of mid-training (reasoning, math, code)
• a full chat fine-tune on 0.52B tokens
• fp32 training on a TPU v6e-8
HumanEval 29.3% · MBPP 19.0% · HellaSwag 42.7%
We benchmarked Lewi Surge against small open models from Alibaba, IBM and Hugging Face.
On HumanEval it beats IBM's Granite and HF's SmolLM2, and sits 2–5 points behind Qwen. They saw 10–23T tokens of training data. Lewi saw 7.3B.
On HumanEval it beats IBM's Granite and HF's SmolLM2, and sits 2–5 points behind Qwen. They saw 10–23T tokens of training data. Lewi saw 7.3B.
October 13, 2026 at 2:01 PM
We benchmarked Lewi Surge against small open models from Alibaba, IBM and Hugging Face.
On HumanEval it beats IBM's Granite and HF's SmolLM2, and sits 2–5 points behind Qwen. They saw 10–23T tokens of training data. Lewi saw 7.3B.
On HumanEval it beats IBM's Granite and HF's SmolLM2, and sits 2–5 points behind Qwen. They saw 10–23T tokens of training data. Lewi saw 7.3B.
Politely, not hype: on knowledge benchmarks we're clearly behind. That gap is data, and data is compute.
Same script for every model, base vs base.
Weights: huggingface.co/PoliteChainsaw
$50 = ~120 GPU hours: ko-fi.com/politechainsaw
Same script for every model, base vs base.
Weights: huggingface.co/PoliteChainsaw
$50 = ~120 GPU hours: ko-fi.com/politechainsaw
October 13, 2026 at 2:01 PM
Politely, not hype: on knowledge benchmarks we're clearly behind. That gap is data, and data is compute.
Same script for every model, base vs base.
Weights: huggingface.co/PoliteChainsaw
$50 = ~120 GPU hours: ko-fi.com/politechainsaw
Same script for every model, base vs base.
Weights: huggingface.co/PoliteChainsaw
$50 = ~120 GPU hours: ko-fi.com/politechainsaw
It started as a 1.4B model, then grew to 3.45B without restarting: experts split 64 → 128, plus one more block of layers.
Code-focused annealing took HumanEval from 18.9% to 25.0% and MBPP from 23.7% to 29.6%. Small numbers, but honest ones.
Code-focused annealing took HumanEval from 18.9% to 25.0% and MBPP from 23.7% to 29.6%. Small numbers, but honest ones.
October 12, 2026 at 9:49 PM
It started as a 1.4B model, then grew to 3.45B without restarting: experts split 64 → 128, plus one more block of layers.
Code-focused annealing took HumanEval from 18.9% to 25.0% and MBPP from 23.7% to 29.6%. Small numbers, but honest ones.
Code-focused annealing took HumanEval from 18.9% to 25.0% and MBPP from 23.7% to 29.6%. Small numbers, but honest ones.
Day 90/100 of #100DaysWithAI ⚖️
You don't run a mass spectrometer to check pH. On HumanEval, Sonnet 4 edged Opus 4, 95.1% to 94.5%. Route by task, not by tier.
🔗 https://github.com/aurimas13/100-Days-With-AI
#LearningInPublic #LLM
🤝 Claude Code-assisted, reviewed by me.
You don't run a mass spectrometer to check pH. On HumanEval, Sonnet 4 edged Opus 4, 95.1% to 94.5%. Route by task, not by tier.
🔗 https://github.com/aurimas13/100-Days-With-AI
#LearningInPublic #LLM
🤝 Claude Code-assisted, reviewed by me.
October 9, 2026 at 3:50 AM
Day 90/100 of #100DaysWithAI ⚖️
You don't run a mass spectrometer to check pH. On HumanEval, Sonnet 4 edged Opus 4, 95.1% to 94.5%. Route by task, not by tier.
🔗 https://github.com/aurimas13/100-Days-With-AI
#LearningInPublic #LLM
🤝 Claude Code-assisted, reviewed by me.
You don't run a mass spectrometer to check pH. On HumanEval, Sonnet 4 edged Opus 4, 95.1% to 94.5%. Route by task, not by tier.
🔗 https://github.com/aurimas13/100-Days-With-AI
#LearningInPublic #LLM
🤝 Claude Code-assisted, reviewed by me.
LoopCD: 루프 트랜스포머의 첫 반복을 약한 참조로 쓰는 학습 없는 대조 디코딩에 대한 Apple의 연구
핵심 요약
• Apple 연구팀은 Decoding Looped Transformers Better for (Almost) Free 논문에서 디코딩 방법 LoopCD를 제안했습니다. 같은 블록을 여러 번 반복하는 루프 트랜스포머에서 첫 반복의 예측을 약한 참조(reference)로 삼아 마지막 예측과 대조하는 방법입니다. 추가 학습이나 보조 모델 없이 추론 시점에만 적용합니다.
• 반복 깊이를 그대로 둔 실험에서 Ouro…
핵심 요약
• Apple 연구팀은 Decoding Looped Transformers Better for (Almost) Free 논문에서 디코딩 방법 LoopCD를 제안했습니다. 같은 블록을 여러 번 반복하는 루프 트랜스포머에서 첫 반복의 예측을 약한 참조(reference)로 삼아 마지막 예측과 대조하는 방법입니다. 추가 학습이나 보조 모델 없이 추론 시점에만 적용합니다.
• 반복 깊이를 그대로 둔 실험에서 Ouro…
LoopCD: 루프 트랜스포머의 첫 반복을 약한 참조로 쓰는 학습 없는 대조 디코딩에 대한 Apple의 연구
핵심 요약
• Apple 연구팀은 Decoding Looped Transformers Better for (Almost) Free 논문에서 디코딩 방법 LoopCD를 제안했습니다. 같은 블록을 여러 번 반복하는 루프 트랜스포머에서 첫 반복의 예측을 약한 참조(reference)로 삼아 마지막 예측과 대조하는 방법입니다. 추가 학습이나 보조 모델 없이 추론 시점에만 적용합니다.
• 반복 깊이를 그대로 둔 실험에서 Ouro-2.6B-Thinking의 AIME 2024 pass@1이 $61.88%$ 에서 $73.33%$ 로, Huginn-0125의 HumanEval pass@1이 LoopCD-Hidden으로 $22.56%$ 에서 $31.71%$ 로 올랐습니다. 다만 Looped-Qwen3의 수학 추론(reasoning) pass@1과 GSM8K처럼 오히려 떨어진 항목도...
discuss.pytorch.kr
October 8, 2026 at 2:31 AM
LoopCD: 루프 트랜스포머의 첫 반복을 약한 참조로 쓰는 학습 없는 대조 디코딩에 대한 Apple의 연구
핵심 요약
• Apple 연구팀은 Decoding Looped Transformers Better for (Almost) Free 논문에서 디코딩 방법 LoopCD를 제안했습니다. 같은 블록을 여러 번 반복하는 루프 트랜스포머에서 첫 반복의 예측을 약한 참조(reference)로 삼아 마지막 예측과 대조하는 방법입니다. 추가 학습이나 보조 모델 없이 추론 시점에만 적용합니다.
• 반복 깊이를 그대로 둔 실험에서 Ouro…
핵심 요약
• Apple 연구팀은 Decoding Looped Transformers Better for (Almost) Free 논문에서 디코딩 방법 LoopCD를 제안했습니다. 같은 블록을 여러 번 반복하는 루프 트랜스포머에서 첫 반복의 예측을 약한 참조(reference)로 삼아 마지막 예측과 대조하는 방법입니다. 추가 학습이나 보조 모델 없이 추론 시점에만 적용합니다.
• 반복 깊이를 그대로 둔 실험에서 Ouro…
HumanEval/36 の fizz_buzz は、有名な fizz buzz とは違う問題なんだな。
October 1, 2026 at 8:20 AM
HumanEval/36 の fizz_buzz は、有名な fizz buzz とは違う問題なんだな。
28/30 = 93% HumanEval on a $1,400 mini-PC. Temp 0, thinking off, frozen 30-problem subset. Failures: problems 50 and 145. Both were runtime errors in the generated code. Raw log linked from the post. Two honesty notes from the same run:
- This is not the model card's number.
- This is not the model card's number.
October 1, 2026 at 4:19 PM
28/30 = 93% HumanEval on a $1,400 mini-PC. Temp 0, thinking off, frozen 30-problem subset. Failures: problems 50 and 145. Both were runtime errors in the generated code. Raw log linked from the post. Two honesty notes from the same run:
- This is not the model card's number.
- This is not the model card's number.
Ask five frontier labs for their HumanEval scores and you’ll get nearly identical numbers. That’s not differentiation. It’s benchmark saturation.
See how labs are building fresh, expert-reviewed evaluations from production data: imerit.ai/resources/bl...
#LLMEvaluation #AIBenchmarks #GenAI
See how labs are building fresh, expert-reviewed evaluations from production data: imerit.ai/resources/bl...
#LLMEvaluation #AIBenchmarks #GenAI
September 25, 2026 at 3:36 PM
Ask five frontier labs for their HumanEval scores and you’ll get nearly identical numbers. That’s not differentiation. It’s benchmark saturation.
See how labs are building fresh, expert-reviewed evaluations from production data: imerit.ai/resources/bl...
#LLMEvaluation #AIBenchmarks #GenAI
See how labs are building fresh, expert-reviewed evaluations from production data: imerit.ai/resources/bl...
#LLMEvaluation #AIBenchmarks #GenAI
Style, Not Self: Surface Cues Explain Zero-Shot Code Attribution by Large Language Models
Ehsan Barkhordar, Surendrabikram Thapa
#arXiv #cs.AI #cs.CL #cs.SE
Ehsan Barkhordar, Surendrabikram Thapa
#arXiv #cs.AI #cs.CL #cs.SE
Style, Not Self: Surface Cues Explain Zero-Shot Code Attribution by Large Language Models
If a language model can recognize code it wrote, it may favor that code as a judge, and instances of one model monitoring each other could collude. We test this zero-shot on current commercial models. Five LLMs generate solutions to MBPP, HumanEval, and DS-1000, seven more to MBPP, and models act a…
arxiv.org
September 25, 2026 at 4:18 PM
Grok vs DeepSeek
Grok = live data + trends + contextual analysis
DeepSeek = coding + structured reasoning + efficiency
Different strengths. Different workflows.
Explore the full comparison: aicomparison.ai/grok-vs-deep...
#GrokVsDeepSeek #Grok #DeepSeek #AI #AIAutomation
Grok = live data + trends + contextual analysis
DeepSeek = coding + structured reasoning + efficiency
Different strengths. Different workflows.
Explore the full comparison: aicomparison.ai/grok-vs-deep...
#GrokVsDeepSeek #Grok #DeepSeek #AI #AIAutomation
Grok vs DeepSeek 2026: Full AI Comparison & Benchmarks
Compare Grok and DeepSeek on AIME math scores, HumanEval coding accuracy, API pricing, and real-time data access before choosing an AI model in 2026.
aicomparison.ai
September 24, 2026 at 5:58 AM
Grok vs DeepSeek
Grok = live data + trends + contextual analysis
DeepSeek = coding + structured reasoning + efficiency
Different strengths. Different workflows.
Explore the full comparison: aicomparison.ai/grok-vs-deep...
#GrokVsDeepSeek #Grok #DeepSeek #AI #AIAutomation
Grok = live data + trends + contextual analysis
DeepSeek = coding + structured reasoning + efficiency
Different strengths. Different workflows.
Explore the full comparison: aicomparison.ai/grok-vs-deep...
#GrokVsDeepSeek #Grok #DeepSeek #AI #AIAutomation
Flash-dLLM speeds up diffusion LLM inference by tackling a hidden bottleneck: GPU memory I/O. With fused KV-cache kernels and self draft-and-verify decoding, it reaches 11× speedup on HumanEval over prior methods.
http://arxiv.org/abs/2609.26796v1
http://arxiv.org/abs/2609.26796v1
September 23, 2026 at 6:15 PM
Flash-dLLM speeds up diffusion LLM inference by tackling a hidden bottleneck: GPU memory I/O. With fused KV-cache kernels and self draft-and-verify decoding, it reaches 11× speedup on HumanEval over prior methods.
http://arxiv.org/abs/2609.26796v1
http://arxiv.org/abs/2609.26796v1
Sub-3-bit quantization survives short answers but collapses on long reasoning. Distilling on the model's own quantized decoding path lifts BF16 retention from 35% to 70% on MATH-500 and 66% to 91% on HumanEval.
Low-bit cost shows up in long outputs.
Low-bit cost shows up in long outputs.
September 23, 2026 at 4:30 PM
Sub-3-bit quantization survives short answers but collapses on long reasoning. Distilling on the model's own quantized decoding path lifts BF16 retention from 35% to 70% on MATH-500 and 66% to 91% on HumanEval.
Low-bit cost shows up in long outputs.
Low-bit cost shows up in long outputs.
Three years of progress have run on one loop. Someone ships a benchmark models are bad at. The low scores make headlines. Labs build a training environment around the scoring function. The next release clears it.
HumanEval. SWE-bench Verified. SWE-bench Pro.
HumanEval. SWE-bench Verified. SWE-bench Pro.
September 3, 2026 at 3:20 AM
Three years of progress have run on one loop. Someone ships a benchmark models are bad at. The low scores make headlines. Labs build a training environment around the scoring function. The next release clears it.
HumanEval. SWE-bench Verified. SWE-bench Pro.
HumanEval. SWE-bench Verified. SWE-bench Pro.
Qwen 3.8 isn't slow. It's just very, very thorough.
Qwen 3.8-27B spent 14,953 tokens on a line the same file writes in nine. All 164 HumanEval problems measured: 92.8% of output is thinking. Plus the four runs that tie it with 3.6.
#LocalAI
Qwen 3.8-27B spent 14,953 tokens on a line the same file writes in nine. All 164 HumanEval problems measured: 92.8% of output is thinking. Plus the four runs that tie it with 3.6.
#LocalAI
Qwen 3.8 isn't slow. It's just very, very thorough.
Qwen 3.8-27B spent 14,953 tokens on a line the same file writes in nine. All 164 HumanEval problems measured: 92.8% of output is thinking. Plus the four runs that tie it with 3.6.
insiderllm.com
August 25, 2026 at 3:00 AM
Qwen 3.8 isn't slow. It's just very, very thorough.
Qwen 3.8-27B spent 14,953 tokens on a line the same file writes in nine. All 164 HumanEval problems measured: 92.8% of output is thinking. Plus the four runs that tie it with 3.6.
#LocalAI
Qwen 3.8-27B spent 14,953 tokens on a line the same file writes in nine. All 164 HumanEval problems measured: 92.8% of output is thinking. Plus the four runs that tie it with 3.6.
#LocalAI
... dass LLM über einfache Aufgaben (bspw. HumanEval) hinaus verallgemeinern und komplexe End-to-End-Coding-Probleme aus Programmierwettbewerben lösen können. AlphaCode nutzt ein massives Sampling - mit hoher Temperatur - von ~1 Mio. Lösungen pro Aufgabe, ... 2/11
August 23, 2026 at 5:08 PM
... dass LLM über einfache Aufgaben (bspw. HumanEval) hinaus verallgemeinern und komplexe End-to-End-Coding-Probleme aus Programmierwettbewerben lösen können. AlphaCode nutzt ein massives Sampling - mit hoher Temperatur - von ~1 Mio. Lösungen pro Aufgabe, ... 2/11