#RealSwe
📰 AI scoort torenhoog op benchmarks, maar valt om op je eigen bedrijfscode

Een nieuwe benchmark laat AI coderen…

👉 Lees verder: www.hetlaatsteainieuws.nl/achtergrond/...
#AI #ArtificieleIntelligentie #AInieuws #AIInnovatie #AiBenchmarks #CodingAgents #RealSwe
October 15, 2026 at 8:13 PM
Coding agents are ranked on curated GitHub issues, but 87% of real user prompts are casually written. RealSWE rephrases 381 task families to match: resolution rates fall 6.4 points and rankings shift. https://arxiv.org/abs/2608.27831
RealSWE: A Compositional Evaluation of Coding Agents under Realistic User Requests
Coding agents are now commonly evaluated on the SWE-bench family of benchmarks, whose tasks are built from curated GitHub issues: long, structured, and information-rich. Real user requests, however, are typically far shorter and less structured. To characterize this gap, we define a six-category information taxonomy and four dimensions of linguistic style, and apply them to real user prompts from SWE-chat and problem statements from SWE-bench Verified and Pro. We find that requests carrying only a problem statement, alone or with limited additional context, account for 88% of real prompts but just 7% of benchmark problems. Furthermore, 87% of real prompts are casually written whereas 94% of benchmark problems are formal. Guided by these observations, we introduce RealSWE, 381 multi-variant task families derived from SWE-bench Verified and Pro. Variants within each family share the same underlying task and gold patch while differing only in information composition and linguistic style. Evaluating seven contemporary LLMs with RealSWE, we find that i) realistic inputs reduce resolution rates by 6.4 pp on average and can change model rankings. Controlled analysis further shows that ii) including Desired Behavior and Motivation significantly affects performance, whereas Environment Information and Reproduction Steps merely add tokens without measurable benefit; iii) linguistic style has only small, model-dependent effects. These findings provide actionable guidance for users and agents: explicitly stating the desired behavior and motivation, which most real prompts omit, substantially improves the LLM's software engineering performance.
arxiv.org
September 2, 2026 at 12:14 PM
Gyuhyeong Kim, Hyojung Gwon, Jeonghyeon Kim, Kyuhong Shim, Sunjae Lee: RealSWE: A Compositional Evaluation of Coding Agents under Realistic User Requests https://arxiv.org/abs/2608.27831 https://arxiv.org/pdf/2608.27831 https://arxiv.org/html/2608.27831
September 1, 2026 at 6:39 AM
Why are we building coding agents that fail the moment a user changes a requirement mid-stream? RealSWE exposes this fragility, showing that compositional coding tasks drop model performance sharply once instructions deviate from rigid, clean-room prompts. #SoftwareEngineering #AIgents
Import AI 471: Why Hugging Face worries me; space mining; FIve Eyes on AI
Welcome to Import AI, a newsletter about AI research. Import AI runs on arXiv, cappuccinos, and feedback from readers. If you’d like to support this, please subscribe. Subscribe now Import AI reade…
jack-clark.net
August 31, 2026 at 2:38 PM
Gyuhyeong Kim, Hyojung Gwon, Jeonghyeon Kim, Kyuhong Shim, Sunjae Lee: RealSWE: A Compositional Evaluation of Coding Agents under Realistic User Requests https://arxiv.org/abs/2608.27831 https://arxiv.org/pdf/2608.27831 https://arxiv.org/html/2608.27831
August 31, 2026 at 6:39 AM