#ExpertCrowd
Toptal's ExpertCrowd built a benchmark of 100 unpublished, expert-authored change requests. Claude Opus 5.5 scored 67.3 percent and GPT-6 Astra 63.7 percent. On 80 of 100 tasks, no model earned full credit on all five runs.
https://arxiv.org/abs/2611.13246
November 15, 2026 at 3:30 PM
ExpertCrowd-SWE: Measuring Frontier Coding Agents on Expert-Authored Engineering Tasks in Unfamiliar Codebases

Josip Vrdoljak et al.

#arXiv #cs.AI #cs.SE
ExpertCrowd-SWE: Measuring Frontier Coding Agents on Expert-Authored Engineering Tasks in Unfamiliar Codebases
ExpertCrowd-SWE (EC-SWE) is a benchmark of 100 expert-authored, multi-file software engineering tasks for evaluating autonomous coding agents in large, unfamiliar codebases. Each task is a change request written by a practising engineer against a pinned commit of a real repository, paired with a se…
arxiv.org
November 11, 2026 at 6:35 PM
ExpertCrowd-SWE: Measuring Frontier Coding Agents on Expert-Authored Engineering Tasks in Unfamiliar Codebases
Read more: https://arxiv.org/html/2611.13246v1
November 11, 2026 at 5:42 PM
Vrdoljak, Singh, Steunebrink, Yarotsky, Lorenzo, Nicoletti, Pigott: ExpertCrowd-SWE: Measuring Frontier Coding Agents on Expert-Authored Engineering Tasks in Unfamiliar Codebases https://arxiv.org/abs/2611.13246 https://arxiv.org/pdf/2611.13246 https://arxiv.org/html/2611.13246
November 11, 2026 at 6:39 AM
ExpertCrowd and HUD Unite to Set New Standards in AI Training Evaluation with SWE Benchmark Release#USA#San_Francisco#HUD#ExpertCrowd#SWE
ExpertCrowd and HUD Unite to Set New Standards in AI Training Evaluation with SWE Benchmark Release
ExpertCrowd and HUD have partnered to develop ExpertCrowd-SWE, an innovative benchmark for evaluating AI performance in real-world scenarios, enhancing AI training.
third-news.com
November 6, 2026 at 3:19 PM
Toptal Launches ExpertCrowd to Train AI with Vetted Domain Experts, Releases Frontier SWE Benchmark

​ SAN FRANCISCO, Oct. 14, 2026 /PRNewswire/ -- Toptal, the world's largest talent network of highly skilled domain experts, today announced the launch of ExpertCrowd, a platform for training and…
Toptal Launches ExpertCrowd to Train AI with Vetted Domain Experts, Releases Frontier SWE Benchmark
​ SAN FRANCISCO, Oct. 14, 2026 /PRNewswire/ -- Toptal, the world's largest talent network of highly skilled domain experts, today announced the launch of ExpertCrowd, a platform for training and improving AI models and systems through RLHF, RL data generation, data labeling, and model evaluation. ExpertCrowd combines its advanced platform for AI model development and testing with Toptal's global network of specialized talent to help frontier AI labs, innovative enterprises, and fast-growing startups build more accurate and capable AI models and systems.
ohsem.me
October 13, 2026 at 4:43 PM