🚀 Excited to share that our provocation paper "Evaluations Using Wikipedia without Data Contamination: From Trusting Articles to Trusting Edit Processes" has been accepted at the NeurIPS workshop Evaluating Evaluations! 🌐📚
#NeurIPS2024 #evaleval #AIEvaluation
#NeurIPS2024 #evaleval #AIEvaluation
November 28, 2024 at 9:41 AM
🚀 Excited to share that our provocation paper "Evaluations Using Wikipedia without Data Contamination: From Trusting Articles to Trusting Edit Processes" has been accepted at the NeurIPS workshop Evaluating Evaluations! 🌐📚
#NeurIPS2024 #evaleval #AIEvaluation
#NeurIPS2024 #evaleval #AIEvaluation
Watch today's Century Report podcast here:
https://www.youtube.com/watch?v=4J1azx95ZQs
#AIEvaluation #Anthropic
https://www.youtube.com/watch?v=4J1azx95ZQs
#AIEvaluation #Anthropic
September 21, 2026 at 7:22 PM
Watch today's Century Report podcast here:
https://www.youtube.com/watch?v=4J1azx95ZQs
#AIEvaluation #Anthropic
https://www.youtube.com/watch?v=4J1azx95ZQs
#AIEvaluation #Anthropic
4/4 Benchmarks have been heavily Python-skewed. Production engineering isn't.
Help us build evaluation that reflects how AI agents actually perform across real-world language ecosystems → swebencharena.com
#SWEBenchArena #AIEvaluation #SoftwareEngineering
Help us build evaluation that reflects how AI agents actually perform across real-world language ecosystems → swebencharena.com
#SWEBenchArena #AIEvaluation #SoftwareEngineering
April 19, 2026 at 2:37 PM
4/4 Benchmarks have been heavily Python-skewed. Production engineering isn't.
Help us build evaluation that reflects how AI agents actually perform across real-world language ecosystems → swebencharena.com
#SWEBenchArena #AIEvaluation #SoftwareEngineering
Help us build evaluation that reflects how AI agents actually perform across real-world language ecosystems → swebencharena.com
#SWEBenchArena #AIEvaluation #SoftwareEngineering
What if you could test an AI system with billions of simulated users before putting it in front of real people?
#AI #ArtificialIntelligence #GenerativeAI #AIAgents #AgenticAI #MachineLearning #LLM #AIEvaluation #Simulation #OpenSource #SoftwareEngineering #TechInnovation
matraix.ai
#AI #ArtificialIntelligence #GenerativeAI #AIAgents #AgenticAI #MachineLearning #LLM #AIEvaluation #Simulation #OpenSource #SoftwareEngineering #TechInnovation
matraix.ai
MatrAIx — Simulate Before Reality
MatrAIx is open-source evaluation infrastructure for simulating diverse users before real-world deployment.
matraix.ai
August 11, 2026 at 2:32 PM
What if you could test an AI system with billions of simulated users before putting it in front of real people?
#AI #ArtificialIntelligence #GenerativeAI #AIAgents #AgenticAI #MachineLearning #LLM #AIEvaluation #Simulation #OpenSource #SoftwareEngineering #TechInnovation
matraix.ai
#AI #ArtificialIntelligence #GenerativeAI #AIAgents #AgenticAI #MachineLearning #LLM #AIEvaluation #Simulation #OpenSource #SoftwareEngineering #TechInnovation
matraix.ai
🎓 How can open-source AI help build more resilient public services?
Open Source AI Fellow, Angus Williams, shares how he's working with the Department for Science, Innovation & Technology to measure how well AI tutors perform in classrooms: bit.ly/4eKzYwa
#AI #AIEducation #AIEvaluation
Open Source AI Fellow, Angus Williams, shares how he's working with the Department for Science, Innovation & Technology to measure how well AI tutors perform in classrooms: bit.ly/4eKzYwa
#AI #AIEducation #AIEvaluation
Angus Williams
In January the start of the “
bit.ly
June 25, 2026 at 2:47 PM
🎓 How can open-source AI help build more resilient public services?
Open Source AI Fellow, Angus Williams, shares how he's working with the Department for Science, Innovation & Technology to measure how well AI tutors perform in classrooms: bit.ly/4eKzYwa
#AI #AIEducation #AIEvaluation
Open Source AI Fellow, Angus Williams, shares how he's working with the Department for Science, Innovation & Technology to measure how well AI tutors perform in classrooms: bit.ly/4eKzYwa
#AI #AIEducation #AIEvaluation
Systems can measure your credentials. They can't measure your trajectory. There's a difference and it matters more than we admit. #aievaluation
What AI Can't Measure About Human Potential
hackernoon.com
May 31, 2026 at 7:14 PM
Systems can measure your credentials. They can't measure your trajectory. There's a difference and it matters more than we admit. #aievaluation
I’ve been testing a prompt-level operator that acts like a soft control layer for #LLMs.
It produces a 7.4× contraction in behavioural manifolds and suppresses adversarial drift in repeated generations.
Methods + metrics👉 zenodo.org/records/1771...
#AI #PromptEngineering #Robustness #AIEvaluation
It produces a 7.4× contraction in behavioural manifolds and suppresses adversarial drift in repeated generations.
Methods + metrics👉 zenodo.org/records/1771...
#AI #PromptEngineering #Robustness #AIEvaluation
December 3, 2025 at 3:51 PM
I’ve been testing a prompt-level operator that acts like a soft control layer for #LLMs.
It produces a 7.4× contraction in behavioural manifolds and suppresses adversarial drift in repeated generations.
Methods + metrics👉 zenodo.org/records/1771...
#AI #PromptEngineering #Robustness #AIEvaluation
It produces a 7.4× contraction in behavioural manifolds and suppresses adversarial drift in repeated generations.
Methods + metrics👉 zenodo.org/records/1771...
#AI #PromptEngineering #Robustness #AIEvaluation
Their CEO says most people still think of them as an open-source project.
When community trust becomes a revenue engine, you don't need to pivot. You just need to monetize what you've already built.
#AIPulse #AI #AIEvaluation #Arena #GenerativeAI
When community trust becomes a revenue engine, you don't need to pivot. You just need to monetize what you've already built.
#AIPulse #AI #AIEvaluation #Arena #GenerativeAI
July 2, 2026 at 5:01 AM
Their CEO says most people still think of them as an open-source project.
When community trust becomes a revenue engine, you don't need to pivot. You just need to monetize what you've already built.
#AIPulse #AI #AIEvaluation #Arena #GenerativeAI
When community trust becomes a revenue engine, you don't need to pivot. You just need to monetize what you've already built.
#AIPulse #AI #AIEvaluation #Arena #GenerativeAI
A 407-model catalog is most useful when one workflow spans short edits, long documents, and repo tasks. Compare cost, context, and capabilities for each step instead of picking one default. #AIEvaluation #LLMComparison https://temprhq.io/models
September 22, 2026 at 5:42 PM
A 407-model catalog is most useful when one workflow spans short edits, long documents, and repo tasks. Compare cost, context, and capabilities for each step instead of picking one default. #AIEvaluation #LLMComparison https://temprhq.io/models
Love seeing this deep dive from @diginomica.com 🔍
✅ Yes to better LLM results
✅ Yes to eval tools that actually do something
✅ Yes to RAG + agentic metrics that go beyond “vibes”
Thanks for spotlighting the work we’re doing at Galileo 🙌
#AIagents #LLMops #AIevaluation
✅ Yes to better LLM results
✅ Yes to eval tools that actually do something
✅ Yes to RAG + agentic metrics that go beyond “vibes”
Thanks for spotlighting the work we’re doing at Galileo 🙌
#AIagents #LLMops #AIevaluation
If you want a better LLM result, you need two things: better data, and better evaluation tools. @jon.diginomica.com takes a deep dive with @rungalileo.bsky.social CTO and Co-Founder, Atin Sanyal about improving the mechanics of AI trust. bit.ly/4bBNjEU
![Atin Sanyal sits in front of a whiteboard with headphones on. He is mid conversation. The image shows diginomica's logo and the quote: "Context adherence is giving you a quantitative measure of: how much did the LLM generation stick to the [RAG] context?"](https://cdn.bsky.app/img/feed_thumbnail/plain/did:plc:ag4xhrkbeh6dmda4vlwusayd/bafkreihl3irq6l5hltujf32sw4fwn7hhmcv6lz6wmtyr4gqo53gxipxxcq)
March 10, 2025 at 7:17 PM
Love seeing this deep dive from @diginomica.com 🔍
✅ Yes to better LLM results
✅ Yes to eval tools that actually do something
✅ Yes to RAG + agentic metrics that go beyond “vibes”
Thanks for spotlighting the work we’re doing at Galileo 🙌
#AIagents #LLMops #AIevaluation
✅ Yes to better LLM results
✅ Yes to eval tools that actually do something
✅ Yes to RAG + agentic metrics that go beyond “vibes”
Thanks for spotlighting the work we’re doing at Galileo 🙌
#AIagents #LLMops #AIevaluation
Compare cost per accepted completed task at the same quality bar. Include retrieval, retries and human fallback.
Cost worksheet (illustrative): dmytro-nasyrov.beehiiv.com/p/rag-cost-s...
#RAG #FinOps #AIEvaluation
Cost worksheet (illustrative): dmytro-nasyrov.beehiiv.com/p/rag-cost-s...
#RAG #FinOps #AIEvaluation
How to Decide Whether Rising RAG Costs Justify a Smaller Service Scope
A scope worksheet and worked example that count completed tasks, displaced support work and the cost of changing course.
dmytro-nasyrov.beehiiv.com
September 25, 2026 at 6:35 AM
Compare cost per accepted completed task at the same quality bar. Include retrieval, retries and human fallback.
Cost worksheet (illustrative): dmytro-nasyrov.beehiiv.com/p/rag-cost-s...
#RAG #FinOps #AIEvaluation
Cost worksheet (illustrative): dmytro-nasyrov.beehiiv.com/p/rag-cost-s...
#RAG #FinOps #AIEvaluation
Fact-checking an entire AI response can hide unsupported claims. What if you decompose the response and fact-check each claim individually?
medium.com/@thetalkinga...
#SpringAI #Java #AIEvaluation #FactChecking
medium.com/@thetalkinga...
#SpringAI #Java #AIEvaluation #FactChecking
September 21, 2026 at 4:03 PM
Fact-checking an entire AI response can hide unsupported claims. What if you decompose the response and fact-check each claim individually?
medium.com/@thetalkinga...
#SpringAI #Java #AIEvaluation #FactChecking
medium.com/@thetalkinga...
#SpringAI #Java #AIEvaluation #FactChecking
Remember overfitting? It's back, but make it RAG.
Researchers show that when RAG systems get "insider knowledge" of how LLM judges evaluate them, they achieve near-perfect scores by gaming the metrics, not by actually improving.
Full Paperzilla summary in the comments
#rag #ai #LLM #AIEvaluation
Researchers show that when RAG systems get "insider knowledge" of how LLM judges evaluate them, they achieve near-perfect scores by gaming the metrics, not by actually improving.
Full Paperzilla summary in the comments
#rag #ai #LLM #AIEvaluation
January 26, 2026 at 3:00 AM
Remember overfitting? It's back, but make it RAG.
Researchers show that when RAG systems get "insider knowledge" of how LLM judges evaluate them, they achieve near-perfect scores by gaming the metrics, not by actually improving.
Full Paperzilla summary in the comments
#rag #ai #LLM #AIEvaluation
Researchers show that when RAG systems get "insider knowledge" of how LLM judges evaluate them, they achieve near-perfect scores by gaming the metrics, not by actually improving.
Full Paperzilla summary in the comments
#rag #ai #LLM #AIEvaluation
How Do You Know If AI Is Accurate? The Question That Needs Answers... youtu.be/yIizc5UCNTA?... via
@YouTube
#AI #ArtificialIntelligence #GenerativeAI #LLM #AIAccuracy #AIEvaluation #MachineLearning #DataScience #RAG #Copilot #ChatGPT #DataEngineering
@YouTube
#AI #ArtificialIntelligence #GenerativeAI #LLM #AIAccuracy #AIEvaluation #MachineLearning #DataScience #RAG #Copilot #ChatGPT #DataEngineering
How Do You Know If AI Is Accurate? The Question That Needs to be Answered More
YouTube video by DataBard
youtu.be
June 10, 2026 at 9:09 PM
How Do You Know If AI Is Accurate? The Question That Needs Answers... youtu.be/yIizc5UCNTA?... via
@YouTube
#AI #ArtificialIntelligence #GenerativeAI #LLM #AIAccuracy #AIEvaluation #MachineLearning #DataScience #RAG #Copilot #ChatGPT #DataEngineering
@YouTube
#AI #ArtificialIntelligence #GenerativeAI #LLM #AIAccuracy #AIEvaluation #MachineLearning #DataScience #RAG #Copilot #ChatGPT #DataEngineering
A practical guide to building an AI evaluation framework for GenAI systems, covering bias testing, auto LLM judges, and production-ready evaluation pipelines. #aievaluation
Building a Zero-Click AI Evaluation Pipeline for Production
hackernoon.com
March 9, 2026 at 1:36 PM
A practical guide to building an AI evaluation framework for GenAI systems, covering bias testing, auto LLM judges, and production-ready evaluation pipelines. #aievaluation
#NLP #LLMs #MentalHealth #ClinicalNLP #DigitalHealth #ResponsibleAI #NLProc #AIevaluation #ModelEvaluation #TrustworthyAI #Safety #Equity #HumanCenteredAI
February 19, 2026 at 9:46 AM
If you build LLM-as-judge, multi-agent, or RAG eval loops: keep a non-LLM ground-truth check in the loop.
doi.org/10.5281/zeno...
#LLM #MultiAgent #AIEvaluation #LLMasJudge #RAG #MLOps
doi.org/10.5281/zeno...
#LLM #MultiAgent #AIEvaluation #LLMasJudge #RAG #MLOps
GENESIS R90.9 — Ground Truth between Emergence and Externality
When several AI systems independently arrive at the same answer, it is tempting to treat that agreement as proof. This working paper shows why that intuition misleads — and what it would take to fix i...
doi.org
June 23, 2026 at 8:04 PM
If you build LLM-as-judge, multi-agent, or RAG eval loops: keep a non-LLM ground-truth check in the loop.
doi.org/10.5281/zeno...
#LLM #MultiAgent #AIEvaluation #LLMasJudge #RAG #MLOps
doi.org/10.5281/zeno...
#LLM #MultiAgent #AIEvaluation #LLMasJudge #RAG #MLOps
Your RAG score needs its test set attached.
Pharos Production's RAG delivery includes retrieval evaluation sets. Ask for the query-set version and corpus snapshot behind the score.
pharosproduction.com
#RAG #AIEvaluation #InformationRetrieval
Pharos Production's RAG delivery includes retrieval evaluation sets. Ask for the query-set version and corpus snapshot behind the score.
pharosproduction.com
#RAG #AIEvaluation #InformationRetrieval
Software Development Company | Pharos Production
Custom software development company since 2013 • 90+ engineers • 110+ apps for FinTech, healthcare and Web3 • Rated 5/5 on Clutch • Free estimate
pharosproduction.com
September 23, 2026 at 7:23 AM
Your RAG score needs its test set attached.
Pharos Production's RAG delivery includes retrieval evaluation sets. Ask for the query-set version and corpus snapshot behind the score.
pharosproduction.com
#RAG #AIEvaluation #InformationRetrieval
Pharos Production's RAG delivery includes retrieval evaluation sets. Ask for the query-set version and corpus snapshot behind the score.
pharosproduction.com
#RAG #AIEvaluation #InformationRetrieval
Google open-sourced a framework that reshapes a training environment around whatever the agent is still bad at. The premise underneath it is the useful part. A fixed evaluation stops measuring the moment the system learns to pass it. #AIEvaluation #AgenticAI
September 22, 2026 at 5:00 PM
Google open-sourced a framework that reshapes a training environment around whatever the agent is still bad at. The premise underneath it is the useful part. A fixed evaluation stops measuring the moment the system learns to pass it. #AIEvaluation #AgenticAI
Today at #ACL2026, we are presenting out MASEval library for multi-agent system evaluation.
@anmolgoel.bsky.social
is in San Diego to present poster and live demo!
📍Grand Hall | Session 3: Oral/Posters/Demos B
🕑Sunday 2pm-3.30pm
#MultiAgentSystem #AIEvaluation #Python
@anmolgoel.bsky.social
is in San Diego to present poster and live demo!
📍Grand Hall | Session 3: Oral/Posters/Demos B
🕑Sunday 2pm-3.30pm
#MultiAgentSystem #AIEvaluation #Python
1/ Evaluating a single agent harness is hard. Evaluating a multi-agent system? Whole different problem.
Most eval tools treat the model as the unit of analysis. In multi-agent systems, the system is what matters.
That's why we built MASEval 🧵
#AI #Agents #Eval #MultiAgentSystem #LLM
Most eval tools treat the model as the unit of analysis. In multi-agent systems, the system is what matters.
That's why we built MASEval 🧵
#AI #Agents #Eval #MultiAgentSystem #LLM
July 5, 2026 at 11:33 AM
Today at #ACL2026, we are presenting out MASEval library for multi-agent system evaluation.
@anmolgoel.bsky.social
is in San Diego to present poster and live demo!
📍Grand Hall | Session 3: Oral/Posters/Demos B
🕑Sunday 2pm-3.30pm
#MultiAgentSystem #AIEvaluation #Python
@anmolgoel.bsky.social
is in San Diego to present poster and live demo!
📍Grand Hall | Session 3: Oral/Posters/Demos B
🕑Sunday 2pm-3.30pm
#MultiAgentSystem #AIEvaluation #Python
Keep versioned labels and failed queries. Define the metric and cutoff so reviewers can trace each result to retrieved evidence.
Deliverables: pharosengineeringnotes.wordpress.com/2026/09/18/r...
#RAG #AIEvaluation #InformationRetrieval
Deliverables: pharosengineeringnotes.wordpress.com/2026/09/18/r...
#RAG #AIEvaluation #InformationRetrieval
How to Specify Retrieval Evaluation Deliverables for a RAG Vendor
Specify a RAG vendor evaluation package with labeled-data ownership, query-level evidence, metric definitions and an eight-part acceptance matrix.
pharosengineeringnotes.wordpress.com
September 23, 2026 at 7:23 AM
Keep versioned labels and failed queries. Define the metric and cutoff so reviewers can trace each result to retrieved evidence.
Deliverables: pharosengineeringnotes.wordpress.com/2026/09/18/r...
#RAG #AIEvaluation #InformationRetrieval
Deliverables: pharosengineeringnotes.wordpress.com/2026/09/18/r...
#RAG #AIEvaluation #InformationRetrieval
Try evaluating patches → swebencharena.com
What quality issues have you noticed with AI-generated code?
#AIEvaluation #SWEBenchArena #CodeQuality #AI #SoftwareEngineering
What quality issues have you noticed with AI-generated code?
#AIEvaluation #SWEBenchArena #CodeQuality #AI #SoftwareEngineering
September 4, 2025 at 3:01 AM
Try evaluating patches → swebencharena.com
What quality issues have you noticed with AI-generated code?
#AIEvaluation #SWEBenchArena #CodeQuality #AI #SoftwareEngineering
What quality issues have you noticed with AI-generated code?
#AIEvaluation #SWEBenchArena #CodeQuality #AI #SoftwareEngineering
Why accessibility might be AI’s biggest breakthrough https://arstechni.ca... #DepartmentforBusinessandTrade #workplaceaccommodation #AIaccessibility #machinelearning #neurodiversity #accessibility #AIassistants #AIevaluation #ukgovernment #M365Copilot #disability #AIandwork #microsoft #dyslexia…
September 9, 2025 at 1:03 PM
Why accessibility might be AI’s biggest breakthrough https://arstechni.ca... #DepartmentforBusinessandTrade #workplaceaccommodation #AIaccessibility #machinelearning #neurodiversity #accessibility #AIassistants #AIevaluation #ukgovernment #M365Copilot #disability #AIandwork #microsoft #dyslexia…
What changes when the same model faces the same task in your IDE and terminal? Use one checklist: prompt, expected output, quality, latency, and cost. Tempr keeps the provider key consistent across both. #AIEvaluation #DevTools https://temprhq.io/providers
September 20, 2026 at 3:48 PM
What changes when the same model faces the same task in your IDE and terminal? Use one checklist: prompt, expected output, quality, latency, and cost. Tempr keeps the provider key consistent across both. #AIEvaluation #DevTools https://temprhq.io/providers