최소 수정만으로 해결되는 버그에서도 함수 전체 재작성, 보조 로직 추가, 시그니처 변경까지 일어나며 거대한 diff가 생기기 쉬움기존 구조를 유지하는 brown-field 작업에서는 테스트 통과만으로 충분하지 않고, 얼마나 적게 바꿨는지도 함께 봐야 검토 가능성과 변경 안전성이 유지됨프로그램적으로 손상시킨 400개 BigCodeBench 문제를 바탕으로 토

최소 수정만으로 해결되는 버그에서도 함수 전체 재작성, 보조 로직 추가, 시그니처 변경까지 일어나며 거대한 diff가 생기기 쉬움기존 구조를 유지하는 brown-field 작업에서는 테스트 통과만으로 충분하지 않고, 얼마나 적게 바꿨는지도 함께 봐야 검토 가능성과 변경 안전성이 유지됨프로그램적으로 손상시킨 400개 BigCodeBench 문제를 바탕으로 토

When LLM coding agents enter repetitive failure loops, the conventional response is to detect the loop and notify the agent — a behavioral advisory of the form "you are looping, try something different." We demonstrate experimentally, over a sequence of controlled phased experiments on BigCodeBench and HumanEval using two LLM families (gpt-4o-mini and Claude Haiku 4.5), that this approach fails on a specific class of failures we term blind-spot failures, where the model emits the same defensive assumption across all regenerations within a loop. Our central finding is that on the canonical task BigCodeBench/6, three non-causal interventions (no advisory, behavioral advisory, and pure data-dump advisory) collectively fail in 0 out of 13 controlled reruns, while a causal advisory adding a single interpretive sentence to an otherwise identical data dump rescues in 7 out of 7 reruns at attempt 4 with exactly one intervention each. The data shown to the model is identical between the failing and succeeding conditions; the only difference is one sentence beginning with "Consequence:" that explains what the data implies for test execution. We additionally show that an independent failure-risk predictor, consuming only structural observation features and never told that an intervention occurred, detects the rescue as a single-step collapse in predicted risk from 0.91 to 0.58, providing orthogonal validation that the rescue is structurally visible from outside the content channel. We demonstrate a library-based observer architecture that achieves 28/30 (93.3%) on gpt-4o-mini and 30/30 (100%) on Claude Haiku 4.5 over BigCodeBench-30, with the same five-pattern library covering the disjoint blind-spot sets of both models without per-model configuration. Finally, we show via cross-model comparison that what appear to be intervention-resistant failures in smaller models can be capability floors that disappear entirely in larger ones, and we argue that single-model evaluation systematically conflates these two phenomena. We propose a four-dimensional model of observer bandwidth — depth, coverage, brittleness, and verifiability — that predicts both successful rescues and a regression mode where insufficiently-covered observers actively make outcomes worse than no intervention. The work is enabled by CAUM, our previously published structural observability framework for LLM agents [1], used here for both loop detection and orthogonal validation.

When LLM coding agents enter repetitive failure loops, the conventional response is to detect the loop and notify the agent — a behavioral advisory of the form "you are looping, try something different." We demonstrate experimentally, over a sequence of controlled phased experiments on BigCodeBench and HumanEval using two LLM families (gpt-4o-mini and Claude Haiku 4.5), that this approach fails on a specific class of failures we term blind-spot failures, where the model emits the same defensive assumption across all regenerations within a loop. Our central finding is that on the canonical task BigCodeBench/6, three non-causal interventions (no advisory, behavioral advisory, and pure data-dump advisory) collectively fail in 0 out of 13 controlled reruns, while a causal advisory adding a single interpretive sentence to an otherwise identical data dump rescues in 7 out of 7 reruns at attempt 4 with exactly one intervention each. The data shown to the model is identical between the failing and succeeding conditions; the only difference is one sentence beginning with "Consequence:" that explains what the data implies for test execution. We additionally show that an independent failure-risk predictor, consuming only structural observation features and never told that an intervention occurred, detects the rescue as a single-step collapse in predicted risk from 0.91 to 0.58, providing orthogonal validation that the rescue is structurally visible from outside the content channel. We demonstrate a library-based observer architecture that achieves 28/30 (93.3%) on gpt-4o-mini and 30/30 (100%) on Claude Haiku 4.5 over BigCodeBench-30, with the same five-pattern library covering the disjoint blind-spot sets of both models without per-model configuration. Finally, we show via cross-model comparison that what appear to be intervention-resistant failures in smaller models can be capability floors that disappear entirely in larger ones, and we argue that single-model evaluation systematically conflates these two phenomena. We propose a four-dimensional model of observer bandwidth — depth, coverage, brittleness, and verifiability — that predicts both successful rescues and a regression mode where insufficiently-covered observers actively make outcomes worse than no intervention. The work is enabled by CAUM, our previously published structural observability framework for LLM agents [1], used here for both loop detection and orthogonal validation.

🧠 I modelli open source avanzano: IQuest-Coder-V1, un nuovo modello cinese, raggiunge risultati di primo piano su benchmark come SWE-Bench Verified (81,4%), BigCodeBench (49,9%) e LiveCodeBench v6 (81...

최소 수정만으로 해결되는 버그에서도 함수 전체 재작성, 보조 로직 추가, 시그니처 변경까지 일어나며 거대한 diff가 생기기 쉬움기존 구조를 유지하는 brown-field 작업에서는 테스트 통과만으로 충분하지 않고, 얼마나 적게 바꿨는지도 함께 봐야 검토 가능성과 변경 안전성이 유지됨프로그램적으로 손상시킨 400개 BigCodeBench 문제를 바탕으로 토

Rankings of the best LLMs on code completion benchmarks - HumanEval, LiveCodeBench, BigCodeBench, MBPP, and competitive programming - with methodology notes on contamination. Updated April 2026.

Rankings of the best LLMs on code completion benchmarks - HumanEval, LiveCodeBench, BigCodeBench, MBPP, and competitive programming - with methodology notes on contamination. Updated April 2026.

BigCodeBench: 多様な関数呼び出しと複雑な指示を用いたコード生成のベンチマーキング https://arxiv.org/html/2406.15877v3#S7

We’re on a journey to advance and democratize artificial intelligence through open source and open science.