#AgentEval
你的 Agent 评测分数很漂亮?先查一件事:

prompt 里的 few-shot 示例,是不是也在测试集里?

示例和测试题重合,模型等于在「开卷考」自己出的题,分数虚高,上线就打回原形。

示例和测试集分开放,并定期用新收集的真实案例换一批。

#AIAgent #AgentEval
November 20, 2026 at 5:30 PM
Agent 的测试很难稳定:真 API 慢、要花钱、结果每次不一样,CI 时好时坏。

借鉴老派 HTTP 测试的 VCR 思路:第一次跑时录下所有工具调用的入参和返回,之后在测试里直接回放。

确定、免费、秒级👇🧵

#AIAgent #AgentEval #DevTips
October 30, 2026 at 5:30 PM
新 Agent 上线前,先跑一段「影子模式」:

它照常接收真实请求、给出决策,但不执行,只记录。拿它的决策和人工实际做法逐条对照。

分歧多的地方,就是你该补的 eval 用例。比任何离线测试都接近真实分布。

#AIAgent #AgentEval #DevTips
October 28, 2026 at 5:30 PM
Agent 一次跑通,不代表可靠。

同一个任务跑 10 次,数数成功几次。单次 demo 里的「90%」,常常是 10 次里 6 次的运气。

评估 Agent 要看方差,不只看均值。上线前,把关键任务各跑 N 遍。

#AIAgent #AgentEval #DevTips
October 26, 2026 at 5:30 PM
The paper presents AgentPerturb, a framework that systematically perturbs LLM agents along two dimensions and tests frontier models on APEX-Agents and Toolathlon, finding consistent performance degradation and that agents rarely…

#AIAgents #LLMRobustness #AgentEval
https://arxiv.org/abs/2610.21624
October 17, 2026 at 10:01 PM
AgentEval offers a contract-bound framework for assessing stateful agents, separating actions from evidence and surfacing binding faults that hash-only checks miss. Relevant to AI infrastructure teams building…

#AIInfrastructure #AgenticAI #SoftwareVerification
https://arxiv.org/abs/2610.21177
October 17, 2026 at 12:01 PM
AgentEval: Evidence-Bounded Assessment of Stateful Agents

Ruiqi Gao (Alibaba Cloud, Alibaba Group)

#arXiv #cs.AI
AgentEval: Evidence-Bounded Assessment of Stateful Agents
Task completion alone does not determine whether a stateful agent satisfied its obligations, and a recorded judgment does not guarantee that its supporting evidence remains valid. We present AgentEval, a system that carries a contract-bound assessment record through tool execution, state and event …
arxiv.org
October 17, 2026 at 2:14 PM
AGENTEVAL presents a system to evaluate stateful AI agents by merging task execution with evidence capture, ensuring reliable assessments as conditions shift. It separates satisfied goals from unresolved obligations and evidence, enhancing interactive AI evaluations. https://arxiv.org/abs/2610.21177
AgentEval: Evidence-Bounded Assessment of Stateful Agents
ArXiv link for AgentEval: Evidence-Bounded Assessment of Stateful Agents
arxiv.org
October 17, 2026 at 4:40 AM
AGENTEVAL presents an evidence-bounded assessment system for stateful agents, ensuring evaluations beyond task completion. It connects execution and evidence capture while revealing hidden faults, enhancing reliability in agent evaluations. https://arxiv.org/abs/2610.21177
AgentEval: Evidence-Bounded Assessment of Stateful Agents
ArXiv link for AgentEval: Evidence-Bounded Assessment of Stateful Agents
arxiv.org
October 16, 2026 at 7:30 PM
AGENTEVAL refines stateful agent evaluation by unifying task contracts, evidence capture, and assessment reports. It differentiates agent performance into satisfied, violated, or unresolved obligations, revealing evidence gaps beyond simple task completion. https://arxiv.org/abs/2610.21177
AgentEval: Evidence-Bounded Assessment of Stateful Agents
ArXiv link for AgentEval: Evidence-Bounded Assessment of Stateful Agents
arxiv.org
October 16, 2026 at 7:01 PM
Presenting AGENTEVAL, a system to boost stateful agent assessments by linking execution records with strong evidence-bounded evaluations. This method distinguishes satisfied obligations from failures and maintains reliable assessments amid changing contexts. https://arxiv.org/abs/2610.21177
AgentEval: Evidence-Bounded Assessment of Stateful Agents
ArXiv link for AgentEval: Evidence-Bounded Assessment of Stateful Agents
arxiv.org
October 16, 2026 at 6:30 PM
AGENTEVAL offers a novel evidence-bounded assessment framework for stateful agents, accurately linking task completion to obligations. This method enhances accountability in AI evaluations by maintaining a detailed record during tool execution and evidence capture. https://arxiv.org/abs/2610.21177
AgentEval: Evidence-Bounded Assessment of Stateful Agents
ArXiv link for AgentEval: Evidence-Bounded Assessment of Stateful Agents
arxiv.org
October 16, 2026 at 7:20 PM
AGENTEVAL introduces a framework for assessing stateful agents, ensuring task completions are backed by evidence and clear obligations. By linking execution history and report reuse, it elevates standards for agent evaluation, addressing real-world complexities. https://arxiv.org/abs/2610.21177
AgentEval: Evidence-Bounded Assessment of Stateful Agents
ArXiv link for AgentEval: Evidence-Bounded Assessment of Stateful Agents
arxiv.org
October 16, 2026 at 6:50 PM
AGENTEVAL provides a way to assess stateful AI agents by connecting execution, evidence, and assessment records, ensuring accountability beyond task completion. It uncovers hidden failures and promotes inspection and reuse of evaluations across scenarios. https://arxiv.org/abs/2610.21177
AgentEval: Evidence-Bounded Assessment of Stateful Agents
ArXiv link for AgentEval: Evidence-Bounded Assessment of Stateful Agents
arxiv.org
October 16, 2026 at 7:40 PM
AGENTEVAL presents an evidence-bounded assessment system for stateful agents, enhancing evaluation by linking task completion and evidence validity. By distinguishing agent actions from proof, it unearths hidden faults, ensuring reliable assessments across contexts. https://arxiv.org/abs/2610.21177
AgentEval: Evidence-Bounded Assessment of Stateful Agents
ArXiv link for AgentEval: Evidence-Bounded Assessment of Stateful Agents
arxiv.org
October 16, 2026 at 6:40 PM
AGENTEVAL revises agent evaluation using evidence-bounded assessments that link execution, obligations, and evidence integrity to ensure valid outcomes in changing contexts. This method enhances transparency and reliability, promoting accountable AI interactions. https://arxiv.org/abs/2610.21177
AgentEval: Evidence-Bounded Assessment of Stateful Agents
ArXiv link for AgentEval: Evidence-Bounded Assessment of Stateful Agents
arxiv.org
October 16, 2026 at 7:10 PM
AgentEval: Evidence-Bounded Assessment of Stateful Agents

Ruiqi Gao (Alibaba Cloud, Alibaba Group)

#arXiv #cs.AI
AgentEval: Evidence-Bounded Assessment of Stateful Agents
Task completion alone does not determine whether a stateful agent satisfied its obligations, and a recorded judgment does not guarantee that its supporting evidence remains valid. We present AgentEval, a system that carries a contract-bound assessment record through tool execution, state and event …
arxiv.org
October 16, 2026 at 3:26 PM
Ruiqi Gao (Alibaba Cloud, Alibaba Group): AgentEval: Evidence-Bounded Assessment of Stateful Agents https://arxiv.org/abs/2610.21177 https://arxiv.org/pdf/2610.21177 https://arxiv.org/html/2610.21177
October 16, 2026 at 6:39 AM
SkillForgeBench introduces a benchmark that uses agent skills as execution blueprints to balance broad domain coverage with procedural depth. It is described in arXiv:2610.14918v1.

#AgentEval #AIBenchmarks #LLMAgents #PromptEngineering
https://arxiv.org/abs/2610.14918
October 16, 2026 at 2:01 AM
New arXiv work shows 10.7% of passing SWE-agent trajectories are "Lucky Passes" with regressions or blind retries, highlighting that binary pass/fail signals hide messy agent behavior. AgentLens offers…

#SWEAgents #AgentEval #DeveloperTools #AIEngineering
https://arxiv.org/abs/2605.12925
October 7, 2026 at 12:01 AM
AgentEval Launches Open-Source AI Benchmarking Initiative AgentEval is a new initiative to provid...

https://www.artificiallawyer.com/2025/02/26/agenteval-launches-open-source-ai-benchmarking-initiative/

#GenAI #Accuracy

Event Attributes
AgentEval Launches Open-Source AI Benchmarking Initiative
AgentEval is a new initiative to provide the legal market with an open-source collection of genAI benchmarks that can be freely used, and also is building a community to share data, ideas and protocols for evaluating legal AI tools. Heading the project is **Darius Emrani** , who is the CEO of Scorecard, a startup focused on supporting LLM-based product development. He told **Artificial Lawyer** : ‘The idea is to provide a lot of benchmarks and best practices. We want to get people involved and become the trusted source for AI benchmarks.’ This site asked why he’s chosen to focus on legal tech. Emrani said that the idea is to help ‘essential services’, and that includes health and finance, with AI needs. Central to that is assessing accuracy – or developing best practices around accuracy – and so that’s where AgentEval comes in. He added that the agentic aspect of AI was also a key component in terms of sectors understanding accuracy. Emrani also stressed that the goal is to remain open-source and community-driven, and (see community access link) he would be keen to engage with LITIG and other projects around the world focused on legal AI benchmarking. The organisation stated that: ‘Because some benchmarking efforts often rely on proprietary datasets, closed methodologies, and restricted access, it can be difficult for researchers and developers to reproduce results, compare models fairly, and refine systems. ‘At the same time, we’ve seen successful open evaluation frameworks — from NIST and ISO standards to initiatives like MLCommons, LLMSYS Chatbot Arena and LegalBench — showing that collaborative, open-source approaches lead to better benchmarking practices.’ They went on to say that this helps: * **‘Law firms** — Gain a clear, standardized way to compare legal AI solutions and select the best tools for their needs. * **Legal AI vendors** — Understand their performance relative to competitors and improve their models based on objective, industry-standard benchmarks. * **Academics & Policymakers** — Access insights into how AI systems perform in real-world legal applications, ensuring responsible deployment and regulation. * **The broader AI industry** — By making benchmarks and best practices freely available, Agent Eval gives smaller startups, research institutions, and independent developers access to the same high-quality evaluation resources as well-funded private companies.’ And here’s a summary of their goals: **Why we builtAgentEval.org** **Mission** To establish a trusted, open source for sharing AI benchmarks and best practices that drive transparency and continuous improvement in AI evaluation. **Vision** A future where open collaboration and shared data standards accelerate responsible AI innovation, making evaluation methodologies accessible and verifiable for everyone. **Why Open Benchmarks?** **Transparency & Trust** Open-sourcing our benchmarks and methodologies allows anyone to inspect, validate, and contribute to our evaluation processes. **Community-driven innovation** An open platform invites contributions from a broad community, leading to more robust and diverse evaluation practices. **Industry adoption** Open-source tools and standards are more likely to be adopted by academic institutions, industry players, and public agencies. **Non-profit and collaborative alignment** Emphasizing open source aligns perfectly with our mission to move the industry forward through shared knowledge and collective effort. You can find more info here. — **Why This Matters – The AL View** The need for some clarity on genAI accuracy came to the attention of everyone last summer after the ‘Stanford Debacle’, when a group of researchers claimed to have exposed major failings in well-known genAI legal research tools. Since then things have evolved. Artificial Lawyer, and others, raised the idea of setting up some sort of shared approach to genAI accuracy. AL suggested this could range from a ‘Kite mark’ to a set of protocols to help buyers and sellers approach this issue. LITIG and others have got things in motion, and private companies are getting involved as well. But, the challenge remains: what are we trying to achieve here? What does ‘there’ look like when it comes to gauging AI accuracy? Is it just a single benchmark test, or several tests, owned by one entity? Is it a basket of different benchmarks? Is it open-source and free for all to use? Moreover, as AL has suggested, do we need to **think more in terms of a ‘compass’ approach**? I.e. that because foundation models are moving so fast and that any company scoring X on any specific benchmark made by Y entity will immediately try to refine their outputs to improve them – as happened after the Stanford study – then is there any lasting value relying on just a single test approach to benchmarking? Moreover, one of the most insightful things that came out of the Stanford fallout was the point made by **Jeff Pfeifer** at LexisNexis about the need to focus on ‘answer usefulness’. I.e. to some extent the value of any AI output is in the ‘eye of the beholder’. So, perhaps instead what we need is a general direction to head towards, and an idea of what ‘good’ looks like **based on a basket of different benchmarks**. I.e. for X type of task, we should expect Y level of results, with those outputs put in the context of the task, e.g. case law research is different to summarising testimony, and both are very different to red-lining a basic NDA. But the overall approach is holistic, open-source, and community-driven. Does that mean that we don’t need benchmarks? No. Not at all. Quite the opposite. Rather the point is that legal work is subjective because it’s all based on language, reasoning and interpretation. Measuring AI accuracy is not like measuring the speed of a car going down a road. So, we need multiple pathways and viewpoints to assess accuracy and therefore a more principle-based approach that accommodates this is more flexible and can evolve as the sector evolves as well. To conclude, AL welcomes this open-source, community-based approach, that seeks to build a general consensus on what ‘good’ looks like, along with protocols for how to approach genAI tools. Within that ecosystem of evaluation are benchmarks – but they are a multi-prong approach to assessment – and it’s accepted, in fact expected, that the results for each company will evolve as rapidly as the state of the art is. Hence, we return to the compass approach. Any road, that’s AL’s view. There is much more to come in terms of genAI accuracy benchmarking. Watch this space! **Richard Tromans, Founder, Artificial Lawyer** — ### Share this: * * Tweet * Email * *
www.artificiallawyer.com
February 26, 2025 at 10:18 AM
☕ AM Pick

Containerized evals with frozen environments and swappable agent configs solve the reproducibility problem that breaks most LLM testing.

#LLMTesting #AgentEval #LangChain #PromptEngineering
Eval Engineering Skill: Build Evals From Repo Context and Traces
LangChain's Eval Engineering Skill inspects your agent's repo and traces, proposes evals through user interviews, and outputs runnable Harbor tasks.
blog.langchain.dev
July 23, 2026 at 1:04 PM
🥃 Nightcap

Keeping the optimizer and evaluator decoupled so agents can't game their own metrics is smart. The custom rubric that counts specific behaviors instead of just adaptive scoring is what makes the loop actually tighten.

#AgentEval #LLMOps #DevOps #QualityMetrics
Driving the Agent Quality Flywheel from Your Coding Agent- Google Developers Blog
Google introduces a coding agent skill that automates the agent quality improvement loop using model-based AutoRaters developed with DeepMind. The skill orchestrates a five-stage evaluation flywheel (prepare data, run inference, grade, analyze failures, optimize) where the optimizer and evaluator st
developers.googleblog.com
July 1, 2026 at 1:51 AM