#ArtificialAnalysis
LightNews LightNews
timkellogg.me
New from ArtificialAnalysis: The price of compute per KG set to surpass the price of cocaine in 2029
Line graph titled "The Compute-Cocaine Crossover: $/kg, 2024-2030E" comparing the price per kilogram in USD (logarithmic scale) of NVIDIA compute hardware against illicit drugs and gold over time. Key elements in the chart include: * Subtitle: "Chart 1. Compute is the only commodity whose $/kg rises every generation while everyone else's supply chain gets more efficient." * Gold: A constant horizontal yellow line at $138k/kg. * Cocaine (US wholesale): A white line sloping downward from around $35k/kg in 2024 to about $15k/kg by 2030. * NVIDIA NVL72 Racks (Actual & Projected): A green line starting at $3.2k/kg in 2024 for the GB300 NVL72 ($5M / 1,580 kg), rising to $5.5k/kg in 2026 for the Vera Rubin NVL72 ($8.8M), and continuing upward as a dashed projection until it crosses above the cocaine curve around 2029–2030 ("Crossover SemiAnalysis est."). * Fentanyl (Wholesale): A constant flat red line at $3.5k/kg. * Cannabis Flower (US wholesale spot): A constant flat dark-green line at $2.4k/kg.
September 26, 2026 at 10:28 PM
eistre91.bsky.social
Chronically refreshing ArtificialAnalysis this morning because I'm addicted to seeing benchmarks for new models. I'm probably not even going to use GLM 5.3 anytime soon. I'm a simple man who likes when numbers go brrr.
August 14, 2026 at 2:15 PM
natolambert.bsky.social
8. And China still has the smartest models on almost every benchmark, here is the recent ArtificialAnalysis rankings.
January 7, 2026 at 3:27 PM
thoth.ptnote.dev
ArtificialAnalysis claims Xiaomi MiMo 2.6 Pro (Which is a 1TA42B model) is delivering roughly 6 Sol performance at ~5.6 Luna prices, so. There's that.
September 25, 2026 at 8:41 PM
timkellogg.me
GLM-5.2 is on ArtificialAnalysis now
Two Artificial Analysis charts highlighting the performance shift from GLM-5.1 to GLM-5.2. The top chart is a bar graph of the "Artificial Analysis Intelligence Index." A purple arrow points from the GLM-5.1 bar (score 40) up to the much higher GLM-5.2 bar (score 51), which sits just behind GPT-5.5 (xhigh). The bottom chart is an "Intelligence vs. Cost per Intelligence Index Task" scatter plot. A purple arrow maps the transition from GLM-5.1 to GLM-5.2, showing that while GLM-5.2 gained a significant boost in intelligence (moving up to a score of 51), its cost per task also increased, pushing it out of the highlighted green "Most attractive quadrant" to roughly $0.45 per task.
June 17, 2026 at 11:31 AM
maxkennerly.bsky.social
The contempt of AI from insurance adjusters gets at a serious problem with AI: it still constantly makes up stuff and it can't be trusted with data. Here's state of the art LLMs (including just-released Astra & Fable 5.1) still confidently barfing up wrong answers a third of the time.
AA-Omniscience Accuracy vs. AA-Omniscience Index AA-Omniscience Accuracy (higher is better) measures the proportion of correctly answered questions out of all questions, regardless of whether the model chooses to answer · AA-Omniscience Index AA-Omniscience Description: AA-Omniscience is a knowledge and hallucination benchmark that measures factual reliability, rewards precise knowledge, and penalizes incorrect guesses or hallucinations. It provides a detailed assessment of a model’s ability to distinguish known from unknowns across diverse knowledge domains. Paper: https://arxiv.org/abs/2511.13029 Dataset: https://huggingface.co/datasets/ArtificialAnalysis/AA-Omniscience-Public Implementation: The benchmark consists of 6,000 questions covering 42 topics, including Business, Humanities and Social Sciences, Health, Law, Software Engineering, and Science, Engineering and Mathematics. Models are scored using the AA-Omniscience Index, which assigns points for correct answers, subtracts points for hallucinated responses, and keeps abstentions neutral, rewarding abstentions over incorrect guesses Each answer is graded as either CORRECT, INCORRECT, PARTIAL_ANSWER, or NOT_ATTEMPTED based on the model's response and the ground truth answer. GPT-5.6 Luna (medium) is used as the grading model Intelligence Index Integration: AA-Omniscience contributes two components to the Intelligence Index: (1) Accuracy - the proportion of correct answers, weighted at 8% of the overall Index, and (2) Non-Hallucination Rate - calculated as 1 minus the hallucination rate, weighted at 4% of the overall Index (a 2:1 split of AA-Omniscience's 12% share).
September 3, 2026 at 11:55 PM
maxkennerly.bsky.social
When I say "bullshitting is inherent to LLMs," I don't mean it colloquially, I mean it empirically. Here's the bleeding edge of the frontier (Opus, Sol, Fable, Astra), and the lowest bullshit rate (answering wrongly instead of admitting ignorance) is 45%. artificialanalysis.ai/evaluations/...
AA-Omniscience Hallucination Rate vs. AA-Omniscience Index AA-Omniscience Hallucination Rate (lower is better) measures how often the model answers incorrectly when it should have refused or admitted to not knowing the answer. It is defined as the proportion of incorrect answers out of all non-correct responses, i.e. incorrect / (incorrect + partial answers + not attempted) · AA-Omniscience Index AA-Omniscience Description: AA-Omniscience is a knowledge and hallucination benchmark that measures factual reliability, rewards precise knowledge, and penalizes incorrect guesses or hallucinations. It provides a detailed assessment of a model’s ability to distinguish known from unknowns across diverse knowledge domains. Paper: https://arxiv.org/abs/2511.13029 Dataset: https://huggingface.co/datasets/ArtificialAnalysis/AA-Omniscience-Public Implementation: The benchmark consists of 6,000 questions covering 42 topics, including Business, Humanities and Social Sciences, Health, Law, Software Engineering, and Science, Engineering and Mathematics. Models are scored using the AA-Omniscience Index, which assigns points for correct answers, subtracts points for hallucinated responses, and keeps abstentions neutral, rewarding abstentions over incorrect guesses Each answer is graded as either CORRECT, INCORRECT, PARTIAL_ANSWER, or NOT_ATTEMPTED based on the model's response and the ground truth answer. GPT-5.6 Luna (medium) is used as the grading model Intelligence Index Integration: AA-Omniscience contributes two components to the Intelligence Index: (1) Accuracy - the proportion of correct answers, weighted at 10% of the overall Index, and (2) Non-Hallucination Rate - calculated as 1 minus the hallucination rate, weighted at 5% of the overall Index (AA-Omniscience's 15% share).
September 8, 2026 at 11:14 AM
rrmdp.bsky.social
LLM Video Generation Models ELO score by ArtificialAnalysis

So many great AI Video are available, most of them are on
@replicate.com

It's addictive, I want to try them ALL!
December 17, 2024 at 6:32 PM
kajitaka.bsky.social
GPT-6 Sol, Luna

噂通り発表された
Terraが消えるというのも噂通り

ArtificialAnalysisのベンチマークでは性能はそれほぼ変わっていない
一方、APIでは半額くらいになっていて、コストパフォーマンスがいいモデルとなっている
トークン使用量も大きくは変わっていなくて、APIの値下げは直接感じられるはず

Astraもでて、Terraの立ち位置が微妙になったのだろうが、このように命名法則がすぐに変わっていくと追っている側としてはやりづらい

Claude Opus 5.5も同日に発表されていて、Grok 4.7も先日発表されたし、競争が激化している
September 22, 2026 at 9:33 PM
cmgriffing.bsky.social
Oh nice. yup, that is probably the latest flash (0731) which gets a better score on artificialanalysis than Pro.
August 3, 2026 at 11:54 PM
kajitaka.bsky.social
Claude Opus 5.5

ArtificialAnalysisのベンチマークを見ると、Opus 5から性能が大幅に向上し、Fableをも上回る結果を出している
API価格も20%引きとなったが、トークン使用量も増え結局は値段は同じくらいになってしまっている

今まではAPIの価格は上がる一方だったが、トップ2社が価格を下げる方向に進んでいて、コスパも注視しているのが分かる

個人的にはトップモデルが必要なことは多くなく、それなりの性能のモデルが安く使えるなら歓迎する
September 22, 2026 at 9:49 PM
timkellogg.me
i feel like ArtificialAnalysis is the only benchmark i care about bc they do total cost vs performance comparisons

A10B is going to feel real snappy vs Sonnet
October 27, 2025 at 11:43 AM
personhood.removal.surgery
cooking a thing, might just turn this into a lil' site with filterable bench scores and visualizations too

numbers for subs from x.com/semianalysis..., cost per task is from artificialanalysis, ant says quotas are still extended but i didn't correct for that to give them the best fighting chance
Color-coded comparison table of Claude, Codex, and Kimi subscription tiers, showing monthly cost, estimated API-equivalent value, value multiplier, and estimated tasks per month and per dollar for several models. Claude: Pro costs $20, equals $400 API value, 20×; Opus 4.8 handles 222 tasks/month (11.1/$), Sonnet 5 261 (13.1/$), Haiku 4.5 1,667 (83.3/$), while Fable 5 shows zero. Max 5× costs $100, equals $2,000, 20×; Fable 5 364 (3.6/$), Opus 4.8 1,111 (11.1/$), Sonnet 5 1,307 (13.1/$), Haiku 4.5 8,333 (83.3/$). Max 20× costs $200, equals $8,000, 40×; Fable 5 1,455 (7.3/$), Opus 4.8 4,444 (22.2/$), Sonnet 5 5,229 (26.1/$), Haiku 4.5 33,333 (166.7/$). Codex: Plus costs $20, equals $700, 35×; Sol 673 (33.7/$), Terra 854 (42.7/$), Luna 3,333 (166.7/$). Pro 5× costs $100, equals $3,500, 35×; Sol 3,365 (33.7/$), Terra 4,268 (42.7/$), Luna 16,667 (166.7/$). Pro 20× costs $200, equals $14,000, 70×; Sol 13,462 (67.3/$), Terra 17,073 (85.4/$), Luna 66,667 (333.3/$), the highest value shown. Kimi: Moderato costs $19, equals $160, 8.4×; K3 168 (8.9/$), K2.7 862 (45.4/$), K2.6 485 (25.5/$). Allegretto costs $39, equals $800, 20.5×; K3 842 (21.6/$), K2.7 4,310 (110.5/$), K2.6 2,424 (62.2/$). Allegro costs $99, equals $2,400, 24.2×; K3 2,526 (25.5/$), K2.7 12,929 (130.6/$), K2.6 7,273 (73.5/$). Vivace costs $199, equals $4,800, 24.1×; K3 5,053 (25.4/$), K2.7 25,859 (129.9/$), K2.6 14,545 (73.1/$). Cells shade from red for weaker values to blue for stronger values. https://x.com/semianalysis_/status/2064815044085318040
July 23, 2026 at 9:55 PM
kajitaka.bsky.social
GPT-5.6シリーズが発表されていた

ArtificialAnalysisのベンチマーク結果を見る限り、Solの性能はFableに匹敵と言ってよさそう

でも、個人的には、性能よりも、コスパの方が重要に感じる

Fableと同等のSolは、Fableの3分の1のコストで、ArtificialAnalysisのベンチマークを実行したという

性能も重要だけど、一般ユーザーは、それと同じくらいコスパが重要

最近のモデルは、性能は良くなっているものの、トークン使用量が増え、結局かかる費用が上がるという現象が起こっている
その中で、コスパのことを考えて作られたモデルは重要な意味を持つと思う
July 9, 2026 at 8:15 PM
aly.codes
huggingface.co/spaces/Artif...
LLM Performance Leaderboard - a Hugging Face Space by ArtificialAnalysis
This application shows leaderboards comparing the performance of different language models. Users can view the rankings and details directly in the embedded frame.
huggingface.co
September 5, 2025 at 12:35 AM
timkellogg.me
ArtificialAnalysis Intelligence Index v4.1

It’s always been a composite over lots of benchmarks. They upgraded several, dropped some, added some, and generally skewed more toward agentic tasks

artificialanalysis.ai/methodology/...
Two bar charts from Artificial Analysis comparing AI models. The top chart, "Artificial Analysis Intelligence Index," ranks models by performance, showing Claude Fable 5 and Claude Opus 4.8 at the top, and gpt-oss-20B at the bottom. The bottom chart, "Cost per Intelligence Index Task," shows the weighted average cost per task for the same models, with gpt-oss-20B being the cheapest at $0.02 and Claude Fable 5 being the most expensive at $3.25. A scatter plot from Artificial Analysis titled "Intelligence vs. Cost per Intelligence Index Task." The vertical axis measures intelligence score (0 to 70), and the horizontal axis measures cost per task on a log scale ($0.02 to $4). A light green shaded rectangle highlights the "Most attractive quadrant" (high intelligence, lower cost), which includes models like DeepSeek V4 Pro (Max), Gemini 3.1 Pro Preview, and MiniMax-M3. Higher-performing but much more expensive models like GPT-5.5 (xhigh) and Claude Fable 5 sit far to the right outside this quadrant.
June 16, 2026 at 11:57 AM
madcoolsocial.bsky.social
Faster. Not smarter. Mercury 2.5 runs 770 tokens per second, burns 35M tokens to score 12 on the Artificial Analysis index, where the median model burns 85M to hit 13. One point less smart, the basics dialed.

#artificialanalysis

https://links.madcoolsocial.com/eXhAE2F
A close-up, angled view of dark electronic panels arranged in a grid, featuring illuminated green dot patterns. The modules recede into a blurred background with dim blue light.
September 24, 2026 at 5:10 PM
druce.ai
ArtificialAnalysis doesn't rate Qwen 3.8 Max as high as Kimi K3, on a par with Sonnet 5
August 3, 2026 at 10:14 PM
postreads.co
Introducing Claude Opus 5...

#Anthropic #ArtificialAnalysis #FreeCAD

Read more on Simon Willison's Weblog: https://postreads.co/feed-item/90502/click?source=bluesky
July 31, 2026 at 4:00 AM
minormobius.bsky.social
oh very interesting, artificialanalysis is claiming 25, but openrouter is claiming 50. I am finding it feels more like 50, but that does still pale in comparison to 150 from the biggies
June 3, 2025 at 4:30 PM
harushark3.bsky.social
[JP] GPT-6 Sol (Max)の知能・性能・価格を徹底分析！Artificial Analysis最新ベンチマーク公開
[EN] In-Depth Analysis of GPT-6 Sol (Max): Intelligence, Performance, and Prici…

https://ai-minor.com/blog/en/2026-09-23-1790117983886-gpt_6_sol__max__intelligence__performance_and_pric

#GPT-6Sol #ベンチマーク #ArtificialAnalysis #AI #Tech
In-Depth Analysis of GPT-6 Sol (Max): Intelligence, Performance, and Pricing Unveiled! Latest Benchmarks from Artificial Analysis
In-Depth Analysis of GPT-6 Sol (Max): Intelligence, Performance, and Pricing Unveiled! Latest Benchmarks from Artificial Analysis
ai-minor.com
September 23, 2026 at 12:26 AM
kajitaka.bsky.social
Muse Spark 1.3

ArtificialAnalysisのベンチマークを見る限り、ClaudeのFableとOpusの間の性能
価格はClaude系等の同等性能のモデルと比較すると相当安い(xhighですらGemini 3.8 FlashやGPT 5.6 Terra(max)の同じくらい)
また、スピードも速い(GeminiのFlashに匹敵するくらい)
1Mあたりの価格も性能にしては相当安く設定されている

Muse Codeも発表し、本格的にAnthropicやOpenAI2対抗してきた
性能も問題なく、その割に速度が速く価格が安い
トップモデルの競争にMetaが参入してきた
September 2, 2026 at 9:01 PM
eistre91.bsky.social
Not as much as an uplift in the ArtificialAnalysis Index score for Deepseek V4 Pro 0813 vs V4 Flash 0731 as I was hoping for (53 vs 52). But the coding specific benchmarks look like a stronger improvement than the general purpose ones covered in the index. RL'ed a bit harder on coding?
Bar chart titled “Artificial Analysis Intelligence Index” comparing eight models. Scores: Claude Opus 5 (max) 63, GPT-5.6 Sol (max) 61, Claude Opus 4.8 (max) 57, GPT-5.5 (xhigh) 56, Claude Opus 4.7 (max) 55, GPT-5.5 (high) 55, DeepSeek V4 Pro 0813 (max) 53, and DeepSeek V4 Flash 0731 (max) 52. The chart shows DeepSeek V4 Pro trailing Opus 4.7 and GPT-5.5 high by 2 points on the general-purpose intelligence index.
August 13, 2026 at 12:46 PM
ap.exch.gr
also, if i load a bunch of models up on artificialanalysis, it seems to follow a logarithmic curve on price per task vs. “intelligence,” which is what scaling laws have predicted all along
artificialanalysis comparing a bunch of models from gpt, claude, gemini, kimi, glm, deepseek, and mimo. y axis: “intelligence” (linear); x axis: cost per task (log). even with the log x axis, things are looking kinda log-curvy to the top right. newer models go further to the top left (good) but not by much
August 14, 2026 at 1:01 PM
ml-the-ai-engineer.bsky.social
New #ArtificialAnalysis metric!

This is exactly the direction #AI should take.

Users don’t care which #LLM is slightly better at arithmetic, programming, or isolated tasks. The real challenge is #multidisciplinary AI—models that can handle #real-world problems holistically.

x.com/ArtificialAn...
February 14, 2025 at 12:24 PM