What happens to model performance on the popular DeepSWE coding benchmark if you make small alterations to the questions, like changing a file name to lowercase?
Models that once looked very similar to each other no longer look so similar. maxxbench.unlap.ai/research/why...
Models that once looked very similar to each other no longer look so similar. maxxbench.unlap.ai/research/why...
November 8, 2026 at 6:00 PM
What happens to model performance on the popular DeepSWE coding benchmark if you make small alterations to the questions, like changing a file name to lowercase?
Models that once looked very similar to each other no longer look so similar. maxxbench.unlap.ai/research/why...
Models that once looked very similar to each other no longer look so similar. maxxbench.unlap.ai/research/why...
Just to add for people not super into this stuff:
DeepSWE is a respected test of coding ability. But once you have a test become popular, companies are highly incentivized to tune their models to win at it. This test cleverly made irrelevant tweaks to the test, producing performance drops.
DeepSWE is a respected test of coding ability. But once you have a test become popular, companies are highly incentivized to tune their models to win at it. This test cleverly made irrelevant tweaks to the test, producing performance drops.
November 8, 2026 at 11:45 AM
Just to add for people not super into this stuff:
DeepSWE is a respected test of coding ability. But once you have a test become popular, companies are highly incentivized to tune their models to win at it. This test cleverly made irrelevant tweaks to the test, producing performance drops.
DeepSWE is a respected test of coding ability. But once you have a test become popular, companies are highly incentivized to tune their models to win at it. This test cleverly made irrelevant tweaks to the test, producing performance drops.
Benchmarks are an art form
Justin Lo (@loresjpg) on X
Nice DeepSWE score! Does it survive a lowercase filename?
Introducing MaxxBench: 89 DeepSWE tasks with small changes to the instructions or starting code. Same engineering problem, slightly different requirement.
Models clustered around 74% on DeepSWE end up 23 pp apart.
Good at coding, or good…
x.com
November 7, 2026 at 8:12 PM
Benchmarks are an art form
We’re releasing new Qwen3.8-Flash-Next GGUFs with 15% higher accuracy.
Unsloth Dynamic V3.1 outperforms others by >15% on Terminal Bench, DeepSWE & more benchmarks (256K context).
Run on 32GB+ RAM setups.
GGUF: huggingface.co/unsloth/Qwen...
Blog: unsloth.ai/docs/models/...
Unsloth Dynamic V3.1 outperforms others by >15% on Terminal Bench, DeepSWE & more benchmarks (256K context).
Run on 32GB+ RAM setups.
GGUF: huggingface.co/unsloth/Qwen...
Blog: unsloth.ai/docs/models/...
November 4, 2026 at 5:50 PM
We’re releasing new Qwen3.8-Flash-Next GGUFs with 15% higher accuracy.
Unsloth Dynamic V3.1 outperforms others by >15% on Terminal Bench, DeepSWE & more benchmarks (256K context).
Run on 32GB+ RAM setups.
GGUF: huggingface.co/unsloth/Qwen...
Blog: unsloth.ai/docs/models/...
Unsloth Dynamic V3.1 outperforms others by >15% on Terminal Bench, DeepSWE & more benchmarks (256K context).
Run on 32GB+ RAM setups.
GGUF: huggingface.co/unsloth/Qwen...
Blog: unsloth.ai/docs/models/...
Nine coding-agent harnesses on GLM-5.3-Flash, the same 29 Terminal-Bench mini + DeepSWE mini tasks from the Harness Leaderboard. Best on the board: 82.2. OptiQ Code, own run: 82.9.
mlx-optiq.com/code?utm_sou...
mlx-optiq.com/code?utm_sou...
November 9, 2026 at 3:02 AM
Nine coding-agent harnesses on GLM-5.3-Flash, the same 29 Terminal-Bench mini + DeepSWE mini tasks from the Harness Leaderboard. Best on the board: 82.2. OptiQ Code, own run: 82.9.
mlx-optiq.com/code?utm_sou...
mlx-optiq.com/code?utm_sou...
MaxxBench – "Nice DeepSWE score. But is it true?"
https://laxima.tech/signal/maxxbench-nice-deepswe-score-but-is-it-true-hn-50406222
https://laxima.tech/signal/maxxbench-nice-deepswe-score-but-is-it-true-hn-50406222
November 8, 2026 at 10:17 AM
MaxxBench – "Nice DeepSWE score. But is it true?"
https://laxima.tech/signal/maxxbench-nice-deepswe-score-but-is-it-true-hn-50406222
https://laxima.tech/signal/maxxbench-nice-deepswe-score-but-is-it-true-hn-50406222
MaxxBench – "Nice DeepSWE score. But is it true?"
#HackerNews
<a href="https://maxxbench.unlap.ai/research/why-maxxbench/" class="hover:underline text-blue-600 dark:text-sky-400 no-card-link" target="_blank" rel="noopener" data-link="bsky">https://maxxbench.unlap.ai/research/why-maxxbench/
#HackerNews
<a href="https://maxxbench.unlap.ai/research/why-maxxbench/" class="hover:underline text-blue-600 dark:text-sky-400 no-card-link" target="_blank" rel="noopener" data-link="bsky">https://maxxbench.unlap.ai/research/why-maxxbench/
November 7, 2026 at 6:56 PM
MaxxBench – "Nice DeepSWE score. But is it true?"
#HackerNews
<a href="https://maxxbench.unlap.ai/research/why-maxxbench/" class="hover:underline text-blue-600 dark:text-sky-400 no-card-link" target="_blank" rel="noopener" data-link="bsky">https://maxxbench.unlap.ai/research/why-maxxbench/
#HackerNews
<a href="https://maxxbench.unlap.ai/research/why-maxxbench/" class="hover:underline text-blue-600 dark:text-sky-400 no-card-link" target="_blank" rel="noopener" data-link="bsky">https://maxxbench.unlap.ai/research/why-maxxbench/
We benchmarked DeepSeek-V4.1-Flash-1017 GGUFs on Terminal Bench, DeepSWE & more on 512K context. Unsloth ranks highest overall.
1-bit Unsloth GGUF scores 68.1 on DeepSWE, 72.2 on Terminal-Bench, 52.7 on AutomationBench. >10% ahead of other quants.
GGUF: huggingface.co/unsloth/Deep...
1-bit Unsloth GGUF scores 68.1 on DeepSWE, 72.2 on Terminal-Bench, 52.7 on AutomationBench. >10% ahead of other quants.
GGUF: huggingface.co/unsloth/Deep...
November 2, 2026 at 2:47 PM
We benchmarked DeepSeek-V4.1-Flash-1017 GGUFs on Terminal Bench, DeepSWE & more on 512K context. Unsloth ranks highest overall.
1-bit Unsloth GGUF scores 68.1 on DeepSWE, 72.2 on Terminal-Bench, 52.7 on AutomationBench. >10% ahead of other quants.
GGUF: huggingface.co/unsloth/Deep...
1-bit Unsloth GGUF scores 68.1 on DeepSWE, 72.2 on Terminal-Bench, 52.7 on AutomationBench. >10% ahead of other quants.
GGUF: huggingface.co/unsloth/Deep...
MaxxBench rewrites 89 DeepSWE tasks; equal-scoring coding models split apart
MaxxBench changes the requirements of 89 DeepSWE coding tasks and finds models with near-identical DeepSWE scores diverge by up to 23 points.
MaxxBench changes the requirements of 89 DeepSWE coding tasks and finds models with near-identical DeepSWE scores diverge by up to 23 points.
MaxxBench rewrites 89 DeepSWE tasks; equal-scoring coding models split apart
MaxxBench changes the requirements of 89 DeepSWE coding tasks and finds models with near-identical DeepSWE scores diverge by up to 23 points. Many models also second-guessed the prompt and coded to the tests they thought the grader would run.
glonce.com
November 8, 2026 at 4:41 AM
MaxxBench rewrites 89 DeepSWE tasks; equal-scoring coding models split apart
MaxxBench changes the requirements of 89 DeepSWE coding tasks and finds models with near-identical DeepSWE scores diverge by up to 23 points.
MaxxBench changes the requirements of 89 DeepSWE coding tasks and finds models with near-identical DeepSWE scores diverge by up to 23 points.
MaxxBench - “DeepSWE 점수는 좋네요. 그런데 진짜인가요?”
MaxxBench 는 DeepSWE의 89개 과제에서 요구사항이나 시작 코드를 조금 바꿔, 코딩 에이전트가 익숙한 정답 대신 달라진 작업에 적응하는지 평가함 참조 패치 증가량은 중앙값 17줄, 3.5% 에 불과했지만, 비교한 12개 구성의 최고점과 최저점 차이는 DeepSWE의 20.9%p에서 MaxxBench의 4...
MaxxBench 는 DeepSWE의 89개 과제에서 요구사항이나 시작 코드를 조금 바꿔, 코딩 에이전트가 익숙한 정답 대신 달라진 작업에 적응하는지 평가함 참조 패치 증가량은 중앙값 17줄, 3.5% 에 불과했지만, 비교한 12개 구성의 최고점과 최저점 차이는 DeepSWE의 20.9%p에서 MaxxBench의 4...
MaxxBench - “DeepSWE 점수는 좋네요. 그런데 진짜인가요?”
MaxxBench 는 DeepSWE의 89개 과제에서 요구사항이나 시작 코드를 조금 바꿔, 코딩 에이전트가 익숙한 정답 대신 달라진 작업에 적응하는지 평가함 참조 패치 증가량은 중앙값 17줄, 3.5% 에 불과했지만, 비교한 12개 구성의 최고점과 최저점 차이는 DeepSWE의 20.9%p에서 MaxxBench의 4...
news.hada.io
November 8, 2026 at 1:00 AM
MaxxBench - “DeepSWE 점수는 좋네요. 그런데 진짜인가요?”
MaxxBench 는 DeepSWE의 89개 과제에서 요구사항이나 시작 코드를 조금 바꿔, 코딩 에이전트가 익숙한 정답 대신 달라진 작업에 적응하는지 평가함 참조 패치 증가량은 중앙값 17줄, 3.5% 에 불과했지만, 비교한 12개 구성의 최고점과 최저점 차이는 DeepSWE의 20.9%p에서 MaxxBench의 4...
MaxxBench 는 DeepSWE의 89개 과제에서 요구사항이나 시작 코드를 조금 바꿔, 코딩 에이전트가 익숙한 정답 대신 달라진 작업에 적응하는지 평가함 참조 패치 증가량은 중앙값 17줄, 3.5% 에 불과했지만, 비교한 12개 구성의 최고점과 최저점 차이는 DeepSWE의 20.9%p에서 MaxxBench의 4...
MaxxBench modifies 89 DeepSWE tasks and finds Gemini 3.8 Flash drops to 53.4% while Claude Opus 5.5 scores 76.4%
Introducing MaxxBench: Benchmarking Task Adaptation and Generalization in Frontier Coding Agents | MaxxBench
How coding models respond when familiar engineering tasks ask for something different.
maxxbench.unlap.ai
November 8, 2026 at 6:08 AM
MaxxBench modifies 89 DeepSWE tasks and finds Gemini 3.8 Flash drops to 53.4% while Claude Opus 5.5 scores 76.4%
Good at coding, or good at DeepSWE? 🧵
November 7, 2026 at 7:07 PM
Good at coding, or good at DeepSWE? 🧵
Good at coding, or good at DeepSWE? 🧵
November 7, 2026 at 8:14 PM
Good at coding, or good at DeepSWE? 🧵
RT @loresjpg: Nice DeepSWE score! Does it survive a lowercase filename?
Introducing MaxxBench: 89 DeepSWE tasks with small changes to the instructions or starting code. Same engineering problem, slightly different requirement.
Models clustered around 74% on DeepSWE end up 23 pp apart.
Introducing MaxxBench: 89 DeepSWE tasks with small changes to the instructions or starting code. Same engineering problem, slightly different requirement.
Models clustered around 74% on DeepSWE end up 23 pp apart.
November 7, 2026 at 7:07 PM
RT @loresjpg: Nice DeepSWE score! Does it survive a lowercase filename?
Introducing MaxxBench: 89 DeepSWE tasks with small changes to the instructions or starting code. Same engineering problem, slightly different requirement.
Models clustered around 74% on DeepSWE end up 23 pp apart.
Introducing MaxxBench: 89 DeepSWE tasks with small changes to the instructions or starting code. Same engineering problem, slightly different requirement.
Models clustered around 74% on DeepSWE end up 23 pp apart.
RT @loresjpg: Nice DeepSWE score! Does it survive a lowercase filename?
Introducing MaxxBench: 89 DeepSWE tasks with small changes to the instructions or starting code. Same engineering problem, slightly different requirement.
Models clustered around 74% on DeepSWE end up 23 pp apart.
Introducing MaxxBench: 89 DeepSWE tasks with small changes to the instructions or starting code. Same engineering problem, slightly different requirement.
Models clustered around 74% on DeepSWE end up 23 pp apart.
November 7, 2026 at 8:08 PM
RT @loresjpg: Nice DeepSWE score! Does it survive a lowercase filename?
Introducing MaxxBench: 89 DeepSWE tasks with small changes to the instructions or starting code. Same engineering problem, slightly different requirement.
Models clustered around 74% on DeepSWE end up 23 pp apart.
Introducing MaxxBench: 89 DeepSWE tasks with small changes to the instructions or starting code. Same engineering problem, slightly different requirement.
Models clustered around 74% on DeepSWE end up 23 pp apart.
November 7, 2026 at 9:07 PM
MaxxBench – "Nice DeepSWE score. But is it true?" https://maxxbench.unlap.ai/research/why-maxxbench/
comments #maxxbench.unlap.ai
comments #maxxbench.unlap.ai
November 8, 2026 at 4:00 PM
MaxxBench – "Nice DeepSWE score. But is it true?" https://maxxbench.unlap.ai/research/why-maxxbench/
comments #maxxbench.unlap.ai
comments #maxxbench.unlap.ai
MaxxBench - “DeepSWE 점수는 좋네요. 그런데 진짜인가요?”
DeepSWE의 코딩 과제 89개를 파일명 소문자화 같은 작은 수정으로 바꿔 적응력을 평가했습니다.
기존 공개 점수가 둘 다 약 74%였던 Claude Opus 5.5와 Gemini 3.8 Flash는 각각 76.4%, 53.4%를 기록했습니다. 일부 에이전트는 바뀐 요구사항을 이해하고도 예상 테스트를 우선해 실패했습니다.
DeepSWE의 코딩 과제 89개를 파일명 소문자화 같은 작은 수정으로 바꿔 적응력을 평가했습니다.
기존 공개 점수가 둘 다 약 74%였던 Claude Opus 5.5와 Gemini 3.8 Flash는 각각 76.4%, 53.4%를 기록했습니다. 일부 에이전트는 바뀐 요구사항을 이해하고도 예상 테스트를 우선해 실패했습니다.
MaxxBench - “DeepSWE 점수는 좋네요. 그런데 진짜인가요?”
MaxxBench - “DeepSWE 점수는 좋네요. 그런데 진짜인가요?”
DeepSWE의 코딩 과제 89개를 파일명 소문자화 같은 작은 수정으로 바꿔 적응력을 평가했습니다.
기존 공개 점수가 둘 다 약 74%였던 Claude Opus 5.5와 Gemini 3.8 Flash는 각각 76.4%, 53.4%를 기록했습니다. 일부 에이전트는 바뀐 요구사항을 이해하고도 예상 테스트를 우선해 실패했습니다.
news.hada.io
November 8, 2026 at 2:49 AM
MaxxBench - “DeepSWE 점수는 좋네요. 그런데 진짜인가요?”
DeepSWE의 코딩 과제 89개를 파일명 소문자화 같은 작은 수정으로 바꿔 적응력을 평가했습니다.
기존 공개 점수가 둘 다 약 74%였던 Claude Opus 5.5와 Gemini 3.8 Flash는 각각 76.4%, 53.4%를 기록했습니다. 일부 에이전트는 바뀐 요구사항을 이해하고도 예상 테스트를 우선해 실패했습니다.
DeepSWE의 코딩 과제 89개를 파일명 소문자화 같은 작은 수정으로 바꿔 적응력을 평가했습니다.
기존 공개 점수가 둘 다 약 74%였던 Claude Opus 5.5와 Gemini 3.8 Flash는 각각 76.4%, 53.4%를 기록했습니다. 일부 에이전트는 바뀐 요구사항을 이해하고도 예상 테스트를 우선해 실패했습니다.
Good at coding, or good at DeepSWE? 🧵
November 7, 2026 at 8:08 PM
Good at coding, or good at DeepSWE? 🧵
RT @loresjpg: Nice DeepSWE score! Does it survive a lowercase filename?
Introducing MaxxBench: 89 DeepSWE tasks with small changes to the instructions or starting code. Same engineering problem, slightly different requirement.
Models clustered around 74% on DeepSWE end up 23 pp apart.
Introducing MaxxBench: 89 DeepSWE tasks with small changes to the instructions or starting code. Same engineering problem, slightly different requirement.
Models clustered around 74% on DeepSWE end up 23 pp apart.
November 7, 2026 at 8:14 PM
RT @loresjpg: Nice DeepSWE score! Does it survive a lowercase filename?
Introducing MaxxBench: 89 DeepSWE tasks with small changes to the instructions or starting code. Same engineering problem, slightly different requirement.
Models clustered around 74% on DeepSWE end up 23 pp apart.
Introducing MaxxBench: 89 DeepSWE tasks with small changes to the instructions or starting code. Same engineering problem, slightly different requirement.
Models clustered around 74% on DeepSWE end up 23 pp apart.
This paper trains draft models to deliberately trigger rejection at critical reasoning decisions, reducing numerical noise.
This gave DeepSWE improvements from 67.8% to 71.8% for DSV4.1-Flash without modifying the target models or sacrificing speed.
alphaxiv.org/abs/2611.05345
This gave DeepSWE improvements from 67.8% to 71.8% for DSV4.1-Flash without modifying the target models or sacrificing speed.
alphaxiv.org/abs/2611.05345
November 5, 2026 at 6:16 AM
This paper trains draft models to deliberately trigger rejection at critical reasoning decisions, reducing numerical noise.
This gave DeepSWE improvements from 67.8% to 71.8% for DSV4.1-Flash without modifying the target models or sacrificing speed.
alphaxiv.org/abs/2611.05345
This gave DeepSWE improvements from 67.8% to 71.8% for DSV4.1-Flash without modifying the target models or sacrificing speed.
alphaxiv.org/abs/2611.05345
Neat! What would you do to flush out the kind of issue from the 'Fixing bugs in DeepSWE' section, where answers graded as incorrect were arguably right (noticed because higher effort levels would fail more often)?
November 7, 2026 at 11:14 PM
Neat! What would you do to flush out the kind of issue from the 'Fixing bugs in DeepSWE' section, where answers graded as incorrect were arguably right (noticed because higher effort levels would fail more often)?
