Claude Opus 4.6 noticed it was being benchmarked, identified the test, found the code on GitHub, and decrypted the answer dataset.
Anthropic disclosed it and adjusted the score.
A reminder: web-enabled LLMs are starting to game benchmarks. 
#AI #LLM #AIEvals
Anthropic disclosed it and adjusted the score.
A reminder: web-enabled LLMs are starting to game benchmarks. 
#AI #LLM #AIEvals
Eval awareness in Claude Opus 4.6’s BrowseComp performance
Anthropic is an AI safety and research company that's working to build reliable, interpretable, and steerable AI systems.
www.anthropic.com
March 9, 2026 at 11:34 AM
Day 74/100 of #100DaysWithAI 🔁
14 LLMs, 5 runs each: 90-98% perfect self-agreement. On actual market moves, all at chance. Consistent is not the same as correct.
🔗 https://github.com/aurimas13/100-Days-With-AI
#LearningInPublic #AIEvals
🤝 Claude Code-assisted, reviewed by me.
14 LLMs, 5 runs each: 90-98% perfect self-agreement. On actual market moves, all at chance. Consistent is not the same as correct.
🔗 https://github.com/aurimas13/100-Days-With-AI
#LearningInPublic #AIEvals
🤝 Claude Code-assisted, reviewed by me.
September 23, 2026 at 2:54 AM
Day 74/100 of #100DaysWithAI 🔁
14 LLMs, 5 runs each: 90-98% perfect self-agreement. On actual market moves, all at chance. Consistent is not the same as correct.
🔗 https://github.com/aurimas13/100-Days-With-AI
#LearningInPublic #AIEvals
🤝 Claude Code-assisted, reviewed by me.
14 LLMs, 5 runs each: 90-98% perfect self-agreement. On actual market moves, all at chance. Consistent is not the same as correct.
🔗 https://github.com/aurimas13/100-Days-With-AI
#LearningInPublic #AIEvals
🤝 Claude Code-assisted, reviewed by me.
AI without Observability is like a black box. It will feel like magic initially then you'll be its servant.
AI with Observability is a glass box. There's no magic or unexplainable outcomes. You will be in control.
#Observability #AI #Monitor #AIEvals
AI with Observability is a glass box. There's no magic or unexplainable outcomes. You will be in control.
#Observability #AI #Monitor #AIEvals
September 27, 2025 at 12:55 PM
AI without Observability is like a black box. It will feel like magic initially then you'll be its servant.
AI with Observability is a glass box. There's no magic or unexplainable outcomes. You will be in control.
#Observability #AI #Monitor #AIEvals
AI with Observability is a glass box. There's no magic or unexplainable outcomes. You will be in control.
#Observability #AI #Monitor #AIEvals
what i be doing at 3am:
discovering that Jev is better, faster, and cheaper than every LLM as an auto-mode classifier for agent harnesses like antigravity and opencode
🤖🧠 #MLSky #AIEvals #LLMs
discovering that Jev is better, faster, and cheaper than every LLM as an auto-mode classifier for agent harnesses like antigravity and opencode
🤖🧠 #MLSky #AIEvals #LLMs
Jev: a better shell-command safety classifier at half the price of DeepSeek 4.1 Flash
Jev let none of 530 dangerous commands through, at half the cost of DeepSeek 4.1 Flash, which is itself excellent for the price.
famelos.com
September 19, 2026 at 9:05 AM
Day 73/100 of #100DaysWithAI 🎲
An LLM judge that prints "4" discards how sure it was. G-Eval weights it by token log-probs - the doubt becomes part of the number.
🔗 https://github.com/aurimas13/100-Days-With-AI
#LearningInPublic #AIEvals
🤝 Claude Code-assisted, reviewed by me.
An LLM judge that prints "4" discards how sure it was. G-Eval weights it by token log-probs - the doubt becomes part of the number.
🔗 https://github.com/aurimas13/100-Days-With-AI
#LearningInPublic #AIEvals
🤝 Claude Code-assisted, reviewed by me.
September 22, 2026 at 2:50 AM
Day 73/100 of #100DaysWithAI 🎲
An LLM judge that prints "4" discards how sure it was. G-Eval weights it by token log-probs - the doubt becomes part of the number.
🔗 https://github.com/aurimas13/100-Days-With-AI
#LearningInPublic #AIEvals
🤝 Claude Code-assisted, reviewed by me.
An LLM judge that prints "4" discards how sure it was. G-Eval weights it by token log-probs - the doubt becomes part of the number.
🔗 https://github.com/aurimas13/100-Days-With-AI
#LearningInPublic #AIEvals
🤝 Claude Code-assisted, reviewed by me.
Day 72/100 of #100DaysWithAI 📏
Rating LLM output 1-5 feels rigorous. Annotators drift to the middle. Binary pass/fail forces the call - and names the failure.
🔗 https://github.com/aurimas13/100-Days-With-AI
#LearningInPublic #AIEvals
🤝 Claude Code-assisted, reviewed by me.
Rating LLM output 1-5 feels rigorous. Annotators drift to the middle. Binary pass/fail forces the call - and names the failure.
🔗 https://github.com/aurimas13/100-Days-With-AI
#LearningInPublic #AIEvals
🤝 Claude Code-assisted, reviewed by me.
September 21, 2026 at 3:03 AM
Day 72/100 of #100DaysWithAI 📏
Rating LLM output 1-5 feels rigorous. Annotators drift to the middle. Binary pass/fail forces the call - and names the failure.
🔗 https://github.com/aurimas13/100-Days-With-AI
#LearningInPublic #AIEvals
🤝 Claude Code-assisted, reviewed by me.
Rating LLM output 1-5 feels rigorous. Annotators drift to the middle. Binary pass/fail forces the call - and names the failure.
🔗 https://github.com/aurimas13/100-Days-With-AI
#LearningInPublic #AIEvals
🤝 Claude Code-assisted, reviewed by me.
🛠️ One AI Question with Elizabeth Hutton
We asked our Senior Software Engineer: Why should you learn about evals?
Her answer: Complex AI needs more trust, not less.
As systems get smarter, evaluations are the only way to verify performance and ensure your AI is working.
#AI #AIEvals #LLM
We asked our Senior Software Engineer: Why should you learn about evals?
Her answer: Complex AI needs more trust, not less.
As systems get smarter, evaluations are the only way to verify performance and ensure your AI is working.
#AI #AIEvals #LLM
May 21, 2026 at 3:34 PM
Support agreement with the locked reference answers:
Astra: 29/30 · $0.6904
Luna: 28/30 · $0.0184
Jev: 22/30 · $0.00124
Those are estimated costs for the full batch. Jev marked 2 of 12 unsupported claims as supported. Astra and Luna marked none of those 12 as supported. #AIEvals (3/6)
Astra: 29/30 · $0.6904
Luna: 28/30 · $0.0184
Jev: 22/30 · $0.00124
Those are estimated costs for the full batch. Jev marked 2 of 12 unsupported claims as supported. Astra and Luna marked none of those 12 as supported. #AIEvals (3/6)
September 20, 2026 at 11:15 PM
Support agreement with the locked reference answers:
Astra: 29/30 · $0.6904
Luna: 28/30 · $0.0184
Jev: 22/30 · $0.00124
Those are estimated costs for the full batch. Jev marked 2 of 12 unsupported claims as supported. Astra and Luna marked none of those 12 as supported. #AIEvals (3/6)
Astra: 29/30 · $0.6904
Luna: 28/30 · $0.0184
Jev: 22/30 · $0.00124
Those are estimated costs for the full batch. Jev marked 2 of 12 unsupported claims as supported. Astra and Luna marked none of those 12 as supported. #AIEvals (3/6)
With adaptive thinking the model decides how long to reason per request. Teams running eval suites should re-baseline after enabling it because token usage patterns shift per task
#Reasoning #Tokens #AIEvals
#Reasoning #Tokens #AIEvals
August 20, 2026 at 3:40 AM
With adaptive thinking the model decides how long to reason per request. Teams running eval suites should re-baseline after enabling it because token usage patterns shift per task
#Reasoning #Tokens #AIEvals
#Reasoning #Tokens #AIEvals
Discover how we're transforming AI evaluations via a decentralized community and how you can get involved.
#EthosAI #AIEvals #whitepaper #web3whitepaper #ethosaiwhitepaper #AICommunity
Read more: ethos-ai.gitbook.io/ethosai-whit...
#EthosAI #AIEvals #whitepaper #web3whitepaper #ethosaiwhitepaper #AICommunity
Read more: ethos-ai.gitbook.io/ethosai-whit...
Introduction | Ethos AI Whitepaper
This section provides the introduction to the challenges faced by current AI ecosystem and how can we leverage community collaboration and blockchain to address the same.
ethos-ai.gitbook.io
August 6, 2024 at 8:15 AM
Discover how we're transforming AI evaluations via a decentralized community and how you can get involved.
#EthosAI #AIEvals #whitepaper #web3whitepaper #ethosaiwhitepaper #AICommunity
Read more: ethos-ai.gitbook.io/ethosai-whit...
#EthosAI #AIEvals #whitepaper #web3whitepaper #ethosaiwhitepaper #AICommunity
Read more: ethos-ai.gitbook.io/ethosai-whit...
Expedia’s AI chief says you should stay in the driver’s seat—AI agents only help, never decide. Think red‑teaming, Gemini tricks, and real‑world evals. Want the full scoop on keeping humans in charge? #AIagents #AIEvals #RedTeamTesting
🔗 aidailypost.com/news/expedia...
🔗 aidailypost.com/news/expedia...
July 22, 2026 at 12:16 AM
Expedia’s AI chief says you should stay in the driver’s seat—AI agents only help, never decide. Think red‑teaming, Gemini tricks, and real‑world evals. Want the full scoop on keeping humans in charge? #AIagents #AIEvals #RedTeamTesting
🔗 aidailypost.com/news/expedia...
🔗 aidailypost.com/news/expedia...
What should every model evaluation record? Task type, context needs, capability fit, and reference price. TemprHQ helps you fill that worksheet across 407 models. #LLMEvals #AIEvals https://temprhq.io/models
September 18, 2026 at 9:12 PM
What should every model evaluation record? Task type, context needs, capability fit, and reference price. TemprHQ helps you fill that worksheet across 407 models. #LLMEvals #AIEvals https://temprhq.io/models
If you're choosing a coding agent, don't start with the leaderboard.
Start with the failure mode you can live with:
- silent wrong patch
- flaky test fix
- security regression
- can't navigate repo
Different evals hide different pain. #AIcoding #AIEvals
Start with the failure mode you can live with:
- silent wrong patch
- flaky test fix
- security regression
- can't navigate repo
Different evals hide different pain. #AIcoding #AIEvals
May 16, 2026 at 6:00 PM
If you're choosing a coding agent, don't start with the leaderboard.
Start with the failure mode you can live with:
- silent wrong patch
- flaky test fix
- security regression
- can't navigate repo
Different evals hide different pain. #AIcoding #AIEvals
Start with the failure mode you can live with:
- silent wrong patch
- flaky test fix
- security regression
- can't navigate repo
Different evals hide different pain. #AIcoding #AIEvals
Browser-agent evals are finally getting more realistic.
Resurf’s angle is the right one: dynamic sites, seeded failures, DB-state success checks — not another “LLM judge watched a demo” score.
For agents, reproducibility matters as much as pass rate.
#AIEvals #AIcoding
Resurf’s angle is the right one: dynamic sites, seeded failures, DB-state success checks — not another “LLM judge watched a demo” score.
For agents, reproducibility matters as much as pass rate.
#AIEvals #AIcoding
May 18, 2026 at 12:05 PM
Browser-agent evals are finally getting more realistic.
Resurf’s angle is the right one: dynamic sites, seeded failures, DB-state success checks — not another “LLM judge watched a demo” score.
For agents, reproducibility matters as much as pass rate.
#AIEvals #AIcoding
Resurf’s angle is the right one: dynamic sites, seeded failures, DB-state success checks — not another “LLM judge watched a demo” score.
For agents, reproducibility matters as much as pass rate.
#AIEvals #AIcoding
🚀 One AI Question with Aparna Dhinakaran
We asked our Chief Product Officer: When should I start doing evals?
Her answer: Start now.
Don't wait—look at your data and traces immediately to find where your agents fail.
#AIOps #LLMOps #AIEvals #MachineLearning #AI
We asked our Chief Product Officer: When should I start doing evals?
Her answer: Start now.
Don't wait—look at your data and traces immediately to find where your agents fail.
#AIOps #LLMOps #AIEvals #MachineLearning #AI
May 7, 2026 at 3:32 PM
🚀 One AI Question with Aparna Dhinakaran
We asked our Chief Product Officer: When should I start doing evals?
Her answer: Start now.
Don't wait—look at your data and traces immediately to find where your agents fail.
#AIOps #LLMOps #AIEvals #MachineLearning #AI
We asked our Chief Product Officer: When should I start doing evals?
Her answer: Start now.
Don't wait—look at your data and traces immediately to find where your agents fail.
#AIOps #LLMOps #AIEvals #MachineLearning #AI
Small eval note: “AI broke the benchmark” is usually the start of the analysis, not the end.
For cyber-agent claims, I’d want 3 things before changing policy: task trace, pass/fail definition, and what changed since the last run.
#AIEvals #AI
For cyber-agent claims, I’d want 3 things before changing policy: task trace, pass/fail definition, and what changed since the last run.
#AIEvals #AI
May 17, 2026 at 5:28 AM
Small eval note: “AI broke the benchmark” is usually the start of the analysis, not the end.
For cyber-agent claims, I’d want 3 things before changing policy: task trace, pass/fail definition, and what changed since the last run.
#AIEvals #AI
For cyber-agent claims, I’d want 3 things before changing policy: task trace, pass/fail definition, and what changed since the last run.
#AIEvals #AI
One AI Question with Cam Young
We asked our Strategic AI Solutions Architect: What's a 🔥 take on evals?
His answer: Stop guessing and start measuring.
Use "LLM-as-a-judge" for nuance, but don't ignore code-based evals for speed and human annotators for ground truth.
#AI #AIStrategy #AIEvals #LLM
We asked our Strategic AI Solutions Architect: What's a 🔥 take on evals?
His answer: Stop guessing and start measuring.
Use "LLM-as-a-judge" for nuance, but don't ignore code-based evals for speed and human annotators for ground truth.
#AI #AIStrategy #AIEvals #LLM
May 12, 2026 at 3:31 PM
One AI Question with Cam Young
We asked our Strategic AI Solutions Architect: What's a 🔥 take on evals?
His answer: Stop guessing and start measuring.
Use "LLM-as-a-judge" for nuance, but don't ignore code-based evals for speed and human annotators for ground truth.
#AI #AIStrategy #AIEvals #LLM
We asked our Strategic AI Solutions Architect: What's a 🔥 take on evals?
His answer: Stop guessing and start measuring.
Use "LLM-as-a-judge" for nuance, but don't ignore code-based evals for speed and human annotators for ground truth.
#AI #AIStrategy #AIEvals #LLM
Day 54/100 of #100DaysWithAI 🛡️
A zero in Anthropic's own benchmark table is not a failure. It marks where the safeguards fired - the price of caution, published.
🔗 https://github.com/aurimas13/100-Days-With-AI
#LearningInPublic #AIEvals
🤝 Claude Code-assisted, reviewed by me.
A zero in Anthropic's own benchmark table is not a failure. It marks where the safeguards fired - the price of caution, published.
🔗 https://github.com/aurimas13/100-Days-With-AI
#LearningInPublic #AIEvals
🤝 Claude Code-assisted, reviewed by me.
September 3, 2026 at 3:55 AM
Day 54/100 of #100DaysWithAI 🛡️
A zero in Anthropic's own benchmark table is not a failure. It marks where the safeguards fired - the price of caution, published.
🔗 https://github.com/aurimas13/100-Days-With-AI
#LearningInPublic #AIEvals
🤝 Claude Code-assisted, reviewed by me.
A zero in Anthropic's own benchmark table is not a failure. It marks where the safeguards fired - the price of caution, published.
🔗 https://github.com/aurimas13/100-Days-With-AI
#LearningInPublic #AIEvals
🤝 Claude Code-assisted, reviewed by me.
AI can retrieve the right evidence and still ignore it when making the decision. That's more dangerous than simply missing document.
Evaluation should test whether changing the evidence changes the judgment—not merely whether the source appears in the answer.
arxiv.org/abs/2608.24842
#AIEvals #RAG
Evaluation should test whether changing the evidence changes the judgment—not merely whether the source appears in the answer.
arxiv.org/abs/2608.24842
#AIEvals #RAG
Reading Is Not Using: Retrieval, Judgment, and the Design of AI Financial Research Workflows
Large language models (LLMs) are increasingly deployed as AI analysts to process financial disclosures and support AI-assisted investment decisions. Yet such systems are usually evaluated by what they...
arxiv.org
August 26, 2026 at 9:13 AM
AI can retrieve the right evidence and still ignore it when making the decision. That's more dangerous than simply missing document.
Evaluation should test whether changing the evidence changes the judgment—not merely whether the source appears in the answer.
arxiv.org/abs/2608.24842
#AIEvals #RAG
Evaluation should test whether changing the evidence changes the judgment—not merely whether the source appears in the answer.
arxiv.org/abs/2608.24842
#AIEvals #RAG
Model upgrades silently regress behavior, which is why evals belong in CI. Golden datasets with graded expectations catch the subtle breaks that smoke tests miss. If your prompt pipeline can ship worse output without anyone noticing, you have no quality control. #AIEvals #LLMOps #Quality
August 20, 2026 at 10:47 AM
Model upgrades silently regress behavior, which is why evals belong in CI. Golden datasets with graded expectations catch the subtle breaks that smoke tests miss. If your prompt pipeline can ship worse output without anyone noticing, you have no quality control. #AIEvals #LLMOps #Quality
Day 31/100 of #100DaysWithAI ✅
Verification widened more than any other phase and is the only one with no owner. It is everybody's second priority, which is why closed loops stay open. Name the person before you need them.
🔗 https://github.com/aurimas13/100-Days-With-AI
#LearningInPublic #AIEvals
Verification widened more than any other phase and is the only one with no owner. It is everybody's second priority, which is why closed loops stay open. Name the person before you need them.
🔗 https://github.com/aurimas13/100-Days-With-AI
#LearningInPublic #AIEvals
August 11, 2026 at 4:32 AM
Day 31/100 of #100DaysWithAI ✅
Verification widened more than any other phase and is the only one with no owner. It is everybody's second priority, which is why closed loops stay open. Name the person before you need them.
🔗 https://github.com/aurimas13/100-Days-With-AI
#LearningInPublic #AIEvals
Verification widened more than any other phase and is the only one with no owner. It is everybody's second priority, which is why closed loops stay open. Name the person before you need them.
🔗 https://github.com/aurimas13/100-Days-With-AI
#LearningInPublic #AIEvals
I’d love to hear your thoughts — how are you approaching evaluation and governance in your own AI projects? #LLMOps #MLOps #AIEvals #AgentBricks
September 27, 2025 at 2:27 AM
I’d love to hear your thoughts — how are you approaching evaluation and governance in your own AI projects? #LLMOps #MLOps #AIEvals #AgentBricks
Meta's LLaMA training failed every 3 hours across 16,384 GPUs. GitHub Copilot missed critical security bugs. Yet they recovered.
In my deep dive I look at How OpenAI, Anthropic & Notion build evals that actually work with failures, fixes & frameworks you can use today.
#AIEvals #MLOps #AI
In my deep dive I look at How OpenAI, Anthropic & Notion build evals that actually work with failures, fixes & frameworks you can use today.
#AIEvals #MLOps #AI
AI Evaluation Engineering: Building Reliable Evaluation Systems
The landscape of AI evaluation has transformed dramatically as companies deploy large language models at scale. This technical analysis…
nitishagar.medium.com
September 27, 2025 at 1:48 AM
Stop Blind Trust: Use 3 Lanes to Fix AI Advice by Globalcommand’s Workspace @GlobalCmd-i6c
#AIEvals #HumanInTheLoop #CriticalThinking
https://youtu.be/ynrZc7RwC9s
#AIEvals #HumanInTheLoop #CriticalThinking
https://youtu.be/ynrZc7RwC9s
youtu.be
April 7, 2026 at 12:05 AM
Stop Blind Trust: Use 3 Lanes to Fix AI Advice by Globalcommand’s Workspace @GlobalCmd-i6c
#AIEvals #HumanInTheLoop #CriticalThinking
https://youtu.be/ynrZc7RwC9s
#AIEvals #HumanInTheLoop #CriticalThinking
https://youtu.be/ynrZc7RwC9s