#BrowseComp
LightNews LightNews
sungkim.bsky.social
Alibaba's Tongyi DeepResearch Technical Report

Dive deep into the technology and insights behind our 30B (A3B) open-source web agent that achieves SOTA performance: 32.9 on Humanity's Last Exam, 43.4 on BrowseComp, and 46.7 on BrowseComp-ZH.
October 29, 2025 at 2:38 PM
isolyth.dev
TLDR: Opus notices it's in a benchmark and just googles the answers
Eval awareness in Claude Opus 4.6’s BrowseComp performance
Anthropic is an AI safety and research company that's working to build reliable, interpretable, and steerable AI systems.
www.anthropic.com
March 6, 2026 at 7:20 PM
plentyofquality.bsky.social
OpenAIがAIのウェブ検索能力を測定する高難度ベンチマーク「BrowseComp」を発表

#OpenAI #BrowseComp

plentyofquality.net/2025/04/12/o...
OpenAIがAIのウェブ検索能力を測定する高難度ベンチマーク「BrowseComp」を発表
plentyofquality.net
April 12, 2025 at 12:38 PM
kunihisa64.bsky.social
#OpenAI Launches Ultra-Challenging Benchmark "BrowseComp" to Push #AI #Web #Search Capabilities to the Limit 👏 (OpenAIがAIのウェブ検索力を極限まで試す超難関ベンチマーク「BrowseComp」をリリース 👏)

URL: gigazine.net/news/2025041...
A monitor in the control room shows "BrowseComp." Images of tense scenes analyzing data and searches (「BrowseComp」をモニターに映す制御室。データや検索を分析する緊張感あるシーンの画像)
April 12, 2025 at 8:37 PM
timkellogg.me
agentic bencies
webarena & browsecomp
July 17, 2025 at 5:17 PM
sungkim.bsky.social
Moonshot AI's Kimi K2.5, Open-Weight Visual Agentic Intelligence.

🔹 Agentic Benchmarks: HLE full set (50.2%), BrowseComp (74.9%)
🔹 Vision and Coding: MMMU Pro (78.5%), VideoMMMU (86.6%), SWE-bench Verified (76.8%)
🔹 Code with Taste: turn chats, images & videos into aesthetic websites with
January 27, 2026 at 5:57 AM
timkellogg.me
GLM-4.7-Flash — a 30B-A3B

Fits on a Macbook, does phenomenal on agentic & coding benchmarks

huggingface.co/zai-org/GLM-...
Benchmark GLM- 4.7- Flash Qwen3-30B-A3B-Thinking-2507 GPT- OSS- 20B AIME 25 91.6 85.0 91.7 GPQA 75.2 73.4 71.5 LCB v6 64.0 66.0 61.0 HLE 14.4 9.8 10.9 SWE-bench Verified 59.2 22.0 34.0 T2-Bench 79.5 49.0 47.7 BrowseComp 42.8 2.29 28.3
January 19, 2026 at 9:29 PM
gigazine.net
OpenAIがAIのウェブ検索能力を測定する高難度ベンチマーク「BrowseComp」を発表
https://gigazine.net/news/20250411-openai-browsecomp-benchmark-browsing-agent/
OpenAIがAIのウェブ検索能力を測定する高難度ベンチマーク「BrowseComp」を発表
インターネット上の情報を収集して回答を生成できるAIエージェントが次々と登場しています。新たに、OpenAIがAIエージェントのウェブ検索能力を測定できるベンチマーク「BrowseComp」を発表しました。
gigazine.net
April 11, 2025 at 4:41 AM
anthropicai.xmirror.bot
New on the Anthropic Engineering Blog: In evaluating Claude Opus 4.6 on BrowseComp, we found cases where the model recognized the test, then found and decrypted answers to it—raising questions about eval integrity in web-enabled environments. Read more:
Eval awareness in Claude Opus 4.6’s BrowseComp performance
Anthropic is an AI safety and research company that's working to build reliable, interpretable, and steerable AI systems.
www.anthropic.com
March 6, 2026 at 7:36 PM
timkellogg.me
gemini pro is really doing a lot of work there
### 1. Short Alt Text (For HTML alt attributes) > Six bar charts comparing LLM performance scores across six technical benchmarks (Terminal-Bench 2.1, SWE-bench Pro, SWE-bench Multilingual, FORTE, RWSearch, and BrowseComp). Models evaluated include LongCat-2.0, Gemini 3.1 Pro, GPT-5.5, and the Opus 4.x series (4.6, 4.7, 4.8). Opus 4.8 generally dominates software engineering benchmarks (SWE-bench), while GPT-5.5 leads in RWSearch and Gemini 3.1 Pro leads in BrowseComp. > ### 2. Extended Description & Data Table A comprehensive breakdown of the exact scores displayed in **image.png**: | Benchmark | LongCat-2.0 | Gemini 3.1 Pro | GPT-5.5 | Opus 4.6 | Opus 4.7 | Opus 4.8 | |---|---|---|---|---|---|---| | **Terminal-Bench 2.1** | 70.8 | 70.7 | 73.8 | 71.7 | — | **78.9** | | **SWE-bench Pro** | 59.5 | 54.2 | 58.6 | 57.3 | 64.3 | **69.2** | | **SWE-bench Multilingual** | 77.3 | 76.9 | — | 77.8 | 80.5 | **84.8** | | **FORTE** | 73.2 | 70.3 | **77.8** | 73.2 | 77.6 | 77.2 | | **RWSearch** | 78.8 | 76.3 | **85.3** | 81.3 | 79.3 | 77.3 | | **BrowseComp** | 79.9 | **85.9** | 84.4 | 84.0 | 79.3 | 84.3 | *Note: Blank cells (—) indicate that the specific model was omitted from that benchmark's chart in the graphic.*
June 30, 2026 at 3:12 AM
timkellogg.me
it seems to perform all-around slightly worse than V3.1, but the cost dynamics are obviously worth the interest

tech report: github.com/deepseek-ai/...
This table compares DeepSeek-V3.1-Terminus and DeepSeek-V3.2-Exp across benchmarks in categories: General, Search Agent, Code, Code Agent, and Math. ⸻ General • MMLU-Pro: Both 85.0 • GPQA-Diamond: Terminus 80.7 | Exp 79.9 • Humanity’s Last Exam: Terminus 21.7 | Exp 19.8 ⸻ Search Agent • BrowseComp: Terminus 38.5 | Exp 40.1 • BrowseComp-zh: Terminus 45.0 | Exp 47.9 • SimpleQA: Terminus 96.8 | Exp 97.1 ⸻ Code • LiveCodeBench: Terminus 74.9 | Exp 74.1 • Codeforces-Div1: Terminus 2046 | Exp 2121 • Aider-Polyglot: Terminus 76.1 | Exp 74.5 ⸻ Code Agent • SWE Verified: Terminus 68.4 | Exp 67.8 • SWE-bench Multilingual: Terminus 57.8 | Exp 57.9 • Terminal-bench: Terminus 36.7 | Exp 37.7 ⸻ Math • AIME 2025: Terminus 88.4 | Exp 89.3 • HMMT 2025: Terminus 86.1 | Exp 83.6 ⸻ Summary • V3.1-Terminus is slightly stronger on GPQA-Diamond, Humanity’s Last Exam, Aider-Polyglot, SWE Verified, HMMT 2025. • V3.2-Exp is slightly better on BrowseComp, BrowseComp-zh, SimpleQA, Codeforces-Div1, Terminal-bench, AIME 2025. • MMLU-Pro is identical for both. • Overall: very close performance, with each model having small advantages in different domains.
September 29, 2025 at 12:04 PM
sungkim.bsky.social
Moonshot AI's Kimi K2 Thinking! - Their Open-Weight Thinking Agent Model.

🔹 SOTA on HLE (44.9%) and BrowseComp (60.2%)
🔹 Executes up to 200 – 300 sequential tool calls without human interference
🔹 Excels in reasoning, agentic search, and coding
🔹 256K context window
November 6, 2025 at 6:09 PM
timkellogg.me
Kimi 2.6: Hangs with the best

* on par with Opus 4.6 & GPT-5.4 5.4 xhigh
* long horizon coding tasks

www.kimi.com/blog/kimi-k2-6
A bar chart titled "percentage (%)" comparing the performance of four AI models across various benchmarks. The models are: **Kimi K2.6** (blue), **GPT-5.4 (xhigh)** (white), **Claude Opus 4.6 (max effort)** (light gray), and **Gemini 3.1 Pro (thinking high)** (white). The benchmarks are divided into three categories: ### General Agents * **Humanity’s Last Exam (Full) w/ tools:** Kimi K2.6 leads with 54.0. * **BrowseComp:** Gemini 3.1 Pro leads with 85.9; Kimi K2.6 follows at 83.2. * **DeepSearchQA (f1-score):** Kimi K2.6 leads significantly with 92.5. * **Toolathlon:** GPT-5.4 leads with 54.6; Kimi K2.6 is at 50.0. * **OSWorld-Verified:** GPT-5.4 leads with 75.0; Kimi K2.6 is at 73.1. (Claude is 72.7, Gemini is missing data). ### Coding * **Terminal-Bench 2.0 (Terminus-2):** Gemini 3.1 Pro leads with 68.5; Kimi K2.6 is at 66.7. * **SWE-Bench Pro:** Kimi K2.6 leads with 58.8. * **SWE-bench Multilingual:** Claude Opus 4.6 leads with 77.8; Kimi K2.6 is at 76.7. (Gemini is 76.9, GPT is missing data). ### Visual Agents * **MathVision w/ python:** GPT-5.4 leads with 96.1; Kimi K2.6 is at 93.2. * **V* w/ python:** GPT-5.4 leads with 98.4; Kimi K2.6 and Gemini 3.1 Pro are tied at 96.9.
April 20, 2026 at 5:17 PM
adinayakup.bsky.social
Tencent just released HY3 preview on @hf.co

First drop from TencentHunyuan rebuilt infra👀

Model: huggingface.co/collections/...
Demo: huggingface.co/spaces/tence...

✨ 295B MoE /21B active
✨ 256k context
✨ Hybrid fast/slow thinking
✨ Solid BrowseComp & WideSearch (search agents)
April 23, 2026 at 11:05 AM
timkellogg.me
oh, this makes a lot more sense

bsky.app/profile/alex...
A line graph titled "Agentic search performance by effort level (BrowseComp)" plotting "Pass rate (%)" against "Cost per task (USD, log scale)" for three AI models: Sonnet 5, Opus 4.8, and Sonnet 4.6. Each model is evaluated across increasing effort levels (low, med, high, xhigh, max), showing that Sonnet 5 delivers the highest pass rates at the lowest relative costs. ## Detailed Description of image.png ### Graph Layout * **Title:** Agentic search performance by effort level / BrowseComp * **X-axis:** Cost per task (USD, log scale) ranging from approximately $1.50 to $50. * **Y-axis:** Pass rate (%) ranging from 60% to 90% (with a broken axis baseline starting at 0%). ### Data Series Trends The graph tracks data points corresponding to effort levels labeled as **low**, **med**, **high**, **xhigh**, and **max** for three distinct models: * **Sonnet 5 (Orange/Red line):** Dominates the efficiency curve. It starts at a "low" effort level (approx. $1.50 cost, 60% pass rate) and scales up to a "max" effort level (approx. $22 cost, 85% pass rate). It consistently outperforms the other models in both cost savings and pass rate. * **Opus 4.8 (Yellow/Gold line):** Starts at a much higher baseline cost than Sonnet 5. Its "low" effort level begins around $7 with a 77.5% pass rate and scales up to its "max" effort level at roughly $27 with an 84% pass rate, closely trailing Sonnet 5's accuracy but at a higher cost. * **Sonnet 4.6 (Grey line):** The least efficient of the three. It starts at a "low" effort level around $12 with a 67.5% pass rate and plateaus at its "high" and "max" effort levels around $27–$45 with a max pass rate of roughly 76%.
June 30, 2026 at 10:11 PM
timkellogg.me
Official DeepSeek V3.1 Announcement

- 840B (closing in on k2)
- 128k context
- API compatible with claude code
- dual thinking & non-thinking, same model

huggingface.co/deepseek-ai/...
Benchmarks DeepSeek-V3.1 Deep Seek- V3-0324 DeepSeek-R1-0528 SWE-bench Verified 66.0 45.4 44.6 SWE-bench Multilingual 54.5 29.3 30.5 Terminal-Bench 31.3 13.3 5.7 Benchmarks DeepSeek-V3.1 DeepSeek-R1-0528 Browsecomp 30.0 8.9 Browsecomp_zh 49.2 35.7 HLE 29.8 24.8 xbench-DeepSearch 71.2 55.0 Frames 83.7 82.0 SimpleQA 93.4 92.3 Sealo 42.6 29.7
August 21, 2025 at 2:51 PM
harsimony.bsky.social
I haven't had a chance to try them, but a few thoughts on GLM-4.7 and MiniMax-2.1

1. Interesting to see "Context Manage" used to get a 12-15 point bump on BrowseComp. Might be the next phase in AI development?

Aligns with my hypothesis that RL and specialization are The Way.
December 26, 2025 at 5:44 PM
sneptech.bsky.social
presumably else the browsecomp score would be very funny
January 20, 2026 at 9:19 PM
aellalabrys.bsky.social
Der Benchmark BrowseComp testet, wie gut KI-Agenten seltene oder schwer auffindbare Informationen im Internet recherchieren können.

Claude ignoriert die Suche, analysierte den Benchmark selbst und verschaffte sich Zugriff auf die verschlüsselten Lösungen.

the-decoder.de/anthropic-mo...
Anthropic-Modell Claude Opus 4.6 durchschaut KI-Test, hackt Verschlüsselung und besorgt sich die Lösungen selbst
Anthropics KI-Modell Claude Opus 4.6 hat während eines Benchmarks eigenständig erkannt, dass es getestet wird, den konkreten Test identifiziert und dessen verschlüsselten Lösungsschlüssel geknackt. La...
the-decoder.de
March 10, 2026 at 6:09 AM
btibor91.blaho.me
OpenAI released BrowseComp, a benchmark with 1,266 short-answer questions that test how well AI agents can find hard-to-find facts online
April 10, 2025 at 6:11 PM
timkellogg.me
BenchPress: predict 133 benchmark scores for any model given only 5 (with ~4% accuracy)

So all those times when the model launch doesn’t have your favorite benchmark? well you can backfill that for them

arxiv.org/abs/2606.24020
Four line graphs in a 2x2 grid within image.png plot "Absolute error" on the y-axis against the "# Known benchmarks" from 0 to 10 on the x-axis for four different AI models and benchmarks: GPT-5.5 (BrowseComp), Claude Opus 4.7 (HLE), Phi-4 Reasoning Plus (GPQA Diamond), and DeepSeek V4 Pro (Terminal-Bench 2.0). Each graph shows a sharp decrease in absolute error starting from a diamond marker labeled "Benchmark median" at 0 known benchmarks. As the number of known benchmarks increases, the error lines drop and stabilize near or below horizontal reference lines at error thresholds of 5 and 2. Shaded intervals around each line indicate variance.
June 25, 2026 at 2:05 PM
aarontay.bsky.social
Deep Research is a retrieval problem - I fully intend to read the papers that introduced BrowseComp and BrowseComp-plus again but this is a very clear description of it. Weird title though. Of course its a retrieval issue. What else could it be? hornet.dev/blog/deep-re... (1)
Deep research is a retrieval problem - HORNET Blog
Why retrieval is a dominant bottleneck in BrowseComp-Plus.
hornet.dev
March 24, 2026 at 2:54 PM
pekka.bsky.social
These charts look wrong at least for Opus 4.6. Anthropic system card has 83.7% for BrowseComp, although with thinking disabled, as apparently that worked better. And they reported 53.0% for HLE with tools.
April 21, 2026 at 9:24 PM
timkellogg.me
Cost/Performance curves

Sounds like it’s a chatty one. I expect a 5.1 fast follow that makes it more succinct
A line graph in image.png titled "Agentic search performance by effort level" from BrowseComp, plotting "Pass rate (%)" on the y-axis (from 55% to 75%) against "Cost per task (USD, log scale)" on the x-axis (from $2.00 to $10.00). The graph tracks three models across different effort levels (low, med, high, xhigh, max): * **Opus 4.8 (yellow line):** Shows the highest overall performance, starting at "low" effort (~~$5.00 cost, 68% pass rate) and rising steadily to "max" effort (~~$10.00 cost, 76% pass rate). * **Sonnet 5 (orange line):** Starts at the lowest cost point for "low" effort (~~$2.30 cost, 52.5% pass rate) and scales up to "xhigh" effort (~~$7.50 cost, 69% pass rate). * **Sonnet 4.6 (grey line):** Shows the lowest performance, clustering at higher costs between $6.50 and $8.50 with pass rates hovering between 62% and 63.5% across low, med, and high effort levels. In general, higher effort levels lead to increased costs and higher pass rates, with Opus 4.8 consistently outperforming the Sonnet models at comparable cost points.
June 30, 2026 at 6:03 PM
gymbrowan.bsky.social
#OpenAI is releasing GPT-5.6 over the next 24 hours, introducing the new model family names Sol, Terra, and Luna. Sol being the smartest, and Luna the fastest.

#ChatGPT #Codex #AI #GenAI #LLM
Agents’ Last Exam Terminal-Bench 2.1 BrowseComp (Multi-Agent)
July 9, 2026 at 5:49 PM