AI 모델이 쏟아져 나오는 요즘, ‘우리 모델이 벤치마크 1위’라는 광고를 얼마나 믿을 수 있을까요? 솔직히 말하면 반신반의하시는 게 맞습니다. 테스트 문제를 미리 학습해 점수를 올리는 사례가 늘면서, 기존 AI 벤치마크의 신뢰도가 흔들리고 있거든요. 이 문제를

Microsoft's Evals for Agent Interop is an open-source starter kit that enables developers to evaluate AI agents in realistic work scenarios. It features curated scenarios, datasets, and an evaluation harness to assess agent performance across tools like email and calendars. By Edin Kapić

I tested Kimi K2 and Qwen-3 Coder on 13 Rust development tasks across a 38k-line codebase and 2 Frontend refactor tasks. The results reveal differences in code quality, instruction following, and deve...

Where will the next breakout startup come from? Benchmark’s full partnership weighs in on the main stage at TechCrunch Disrupt 2026. Save up to $200 before September 25, 11:59 p.m. PT. Register now.

Standard AI benchmarks test knowledge and reasoning in isolation. They don't measure whether an AI persona maintains identity across sessions, accumulates knowledge over time, or produces measurably different output with a memory architecture loaded. This article proposes a five-dimension evaluation framework and a structured cognitive assessment battery designed specifically for persistent AI personas. Results from formal testing showed a 59-point gap between architecture-loaded and vanilla Claude on a 180-point scale.

Large Language Models (LLMs) are becoming more powerful, but deploying them on smartphones is complex. Developers face challenges optimizing across various hardware and software configurations. Google AI Edge Portal offers a solution for benchmarking and debugging LLMs on Android devices. This portal allows testing of ML workloads on a diverse range of over 120 Android devices. The new features include benchmarking and debugging LLMs specifically designed for the generative AI era. The portal allows users to benchmark LLMs on various Android devices and measure key metrics such as initialization time and decode speed. Benchmarking helps developers understand how their models perform on specific devices. The platform provides detailed insights into model performance, suggesting optimization opportunities. The Model Explorer feature enables easy debugging, helping find performance bottlenecks in complex model structures. It allows visualization and comparison of model graphs to identify areas for optimization. Users can easily compare and visualize model graphs, search specific nodes, and analyze tensor shapes. The Model Explorer tools help identify and address issues.

Benchmark Capital has been an investor in the Nvidia rival since 2016.

Challenge the newest AI models with your hardest PhD-level exercises. Learn how to use AI in your math research.

Daily AI research, tech audits, and innovation insights from Neon Innovation Lab.

IBM has released four new open-source Granite 4.0 Nano language models, ranging from 350 million to 1.5 billion parameters. These models prioritize efficiency and accessibility, designed to run on laptops and edge devices rather than requiring extensive cloud computing resources. The smallest models can even operate within a web browser, making them highly versatile for developers. The models are licensed under Apache 2.0, enabling commercial use and modification. They possess native compatibility with tools like llama.cpp and vLLM and are certified under ISO 42001 for responsible AI development. Benchmarks reveal they rival or outperform larger models in similar categories, particularly in instruction following and function calling. IBM's Granite models address needs for deployment flexibility, inference privacy, and open auditability. IBM has engaged with the open-source community, hinting at larger models and fine-tuning recipes in the future. This release signals a shift towards strategically scaled AI rather than solely relying on model size. Granite models are designed as enterprise-ready systems emphasizing transparency and performance.

Large language models (LLMs) have transitioned from research labs into the everyday workflows of companies worldwide. While tools like GPT-4 and Claude often steal the spotlight, they come with restrictions such as API rate limits, opaque model behavior, and privacy concerns. This has led to the rise of open-source LLMs like Meta’s LLaMA, Mistral AI’s Mistral, and Google’s Gemma. These models allow developers to build and deploy powerful AI applications without relying on third-party APIs, offering transparency, flexibility, and cost control.

A reproducible benchmark on latency, cost, and reproducibility, and where agents actually earn their keep.