#AgentX
ATOM just smashed the AgentX benchmark, running Kimi‑K3 with 35K‑token calls on AMD Instinct MI355X GPUs. Token caching & agentic inference finally hit real‑world context windows. Dive in to see the numbers! #ATOM #AMDInstinctMI355X #AgentXBenchmark

🔗 aidailypost.com/news/atom-se...
November 17, 2026 at 4:46 PM
SGLang for Multi-Turn Agents: Prefix-Cache Wins, Speculative-Decoding Risks
## Why I Am Evaluating SGLang for Agent Serving The production problem I want SGLang to solve is repeated work: an agent sends its instructions, repository context, previous answers, and tool results back to the server on every turn. Much of that prompt has already been processed. Paying to prefill it again wastes capacity that could serve another session. Validation before deployment No GPU server or AgentX traces were run locally, and the retrieved AgentX results are unofficial; validate cache reuse, latency, throughput, and correctness before claiming savings. SGLang’s RadixAttention makes reusable token prefixes central to serving. Speculative decoding attacks a different cost: generating output by proposing candidate tokens and verifying them with the target model. I evaluate these as separate optimizations, not as one “agent acceleration” switch. **The evidence boundary matters:** I reviewed the available installation instructions, launch examples, and AgentX optimization material, but I did not launch a GPU server or replay AgentX traces for this article. I have no local cache-hit, TTFT, or throughput measurements to present. The commands below are reproduction recipes, not a claim of completed deployment. That distinction is especially important here. The supplied InferenceX AgentX optimization page includes an explicit warning that the retrieved copy is unofficial and its data may be modified or outdated. I use its results as leads for validation, not as a basis for buying fewer GPUs. SGLang’s architecture still warrants evaluation. Multi-turn serving stresses more than attention kernels: * Prefixes must survive eviction between turns. * Routing must find the worker holding useful state. * Tool serialization must preserve reusable token sequences. * Draft-model state must survive scheduling and transfer. * Cache reuse must preserve answer correctness. A strong single-prompt throughput number cannot establish any of those properties. For an agent service, I care about the time to the next useful answer after a tool call, at the concurrency and session churn the business actually expects. ## Proposed Setup and Multi-Turn Test I would begin with one worker, one target checkpoint, and no disaggregation. That isolates prefix reuse before introducing host offload, routing, or draft models. For my setup plan, I selected these installation paths from the SGLang repository: # Python environment installation. python -m venv .venv source .venv/bin/activate python -m pip install --upgrade pip uv uv pip install --prerelease=allow sglang # Alternative: the published container image. docker pull lmsysorg/sglang:latest These are discovery commands, not reproducible production pins. Before benchmarking, I would record the installed package version or container digest and preserve the dependency list: python -m pip freeze > requirements.resolved.txt python -m sglang.launch_server --help > launch-server-help.txt docker image inspect lmsysorg/sglang:latest \ --format '{{json .RepoDigests}}' I have not executed these checks here. In particular, I would not equate a flag appearing in an online example with support in an arbitrary installed wheel. The server-argument reference belongs beside the captured `--help` output during setup. For my planned speculative-decoding comparison, I selected the Llama 3.1/EAGLE3 pairing. With compatible hardware and access to the target weights, the baseline recipe is: python -m sglang.launch_server \ --model meta-llama/Meta-Llama-3.1-8B-Instruct \ --mem-fraction-static 0.7 \ --cuda-graph-max-bs-decode 8 \ --dtype float16 \ --host 127.0.0.1 \ --port 30000 For the speculative arm, I would restart the server and add the documented EAGLE3 configuration: python -m sglang.launch_server \ --model meta-llama/Meta-Llama-3.1-8B-Instruct \ --speculative-algorithm EAGLE3 \ --speculative-draft-model-path jamesliu1/sglang-EAGLE3-Llama-3.1-Instruct-8B \ --speculative-num-steps 3 \ --speculative-eagle-topk 4 \ --speculative-num-draft-tokens 16 \ --mem-fraction-static 0.7 \ --cuda-graph-max-bs-decode 8 \ --dtype float16 \ --host 127.0.0.1 \ --port 30000 I would verify the exact checkpoint revisions before either run. This example also requires model access where applicable; it is not a promise that every checkpoint downloads anonymously. For my configuration, I would either leave all three sizing parameters unset for automatic selection or set all three explicitly when tuning; I checked this constraint in the speculative-decoding guide. I would test the explicit configuration above against automatic selection rather than mixing hand-picked and implicit values. ### A runnable multi-turn smoke test The following client sends two sequential turns and carries the first response into the second request. It requires a running server but no Python packages beyond the standard library. import json import urllib.request URL = "http://127.0.0.1:30000/v1/chat/completions" MODEL = "meta-llama/Meta-Llama-3.1-8B-Instruct" messages = [ { "role": "system", "content": "Answer briefly. Preserve identifiers exactly.", }, { "role": "user", "content": ( "The deployment identifier is amber-47. " "Acknowledge it in one sentence." ), }, ] for turn in range(2): payload = { "model": MODEL, "messages": messages, "temperature": 0, "max_tokens": 64, } request = urllib.request.Request( URL, data=json.dumps(payload).encode("utf-8"), headers={"Content-Type": "application/json"}, method="POST", ) with urllib.request.urlopen(request, timeout=180) as response: result = json.load(response) answer = result["choices"][0]["message"]["content"] print(json.dumps({"turn": turn + 1, "answer": answer})) messages.append({"role": "assistant", "content": answer}) if turn == 0: messages.append({ "role": "user", "content": "What deployment identifier did I give you?", }) **Illustrative output, not captured server output:** {"turn": 1, "answer": "The deployment identifier is amber-47."} {"turn": 2, "answer": "amber-47"} This establishes only a basic client workflow. It does not measure cache reuse, prove that either answer will have this wording, or reproduce AgentX. For a real AgentX replay, I still need the traces, harness revision, request scheduling, model configuration, and expected outputs. Those artifacts were absent from the available material. I would not substitute two chat requests and label the result “AgentX.” My evaluation matrix would separate a cache-disabled control, the ordinary cache-enabled baseline, and cache plus speculation. I would confirm the cache-control option in the pinned build rather than invent a flag. Each arm needs cold-cache and warm-cache runs, plus repeated trials under session churn. I would capture: * **Token-weighted prefix reuse:** cached prompt tokens divided by prompt tokens, using verified server telemetry. * **TTFT:** request dispatch to the first generated token, excluding empty streaming events. * **Output throughput:** generated output tokens divided by wall time, with GPU count explicit. * **Session behavior:** live sessions, active requests, branch fan-out, and tool-result sizes. * **Correctness:** identifier retrieval, structured output validity, and task-specific assertions. A non-streaming response time is not TTFT. Likewise, textual prefix similarity is not a measured cache-hit rate. ## What I Could Not Verify and What I Would Test The immediate roadblock in this review was missing execution evidence, not a demonstrated SGLang crash. I could inspect recipes, but I could not establish local performance results or verify a specific vLLM issue. I keep those gaps visible because otherwise the most persuasive part of an infrastructure review becomes fiction. The technical failure modes still give me a concrete validation checklist. **Window traffic can compete with durable prefixes.** In my evaluation, I would stress the shared pool used by sliding-window allocations and prefix pages, where transient window activity can displace reusable history under memory pressure. I would test this with interleaved sessions and enough pressure to force eviction. Repeating one prompt immediately after itself is insufficient. My mitigation order would be to verify the allocator fixes in the selected build, measure reuse under churn, and then consider offload. Increasing memory allocation without understanding eviction can hide the problem rather than solve it. **Offload has model-specific state requirements.** I interpret the hybrid-cache discussion narrowly: full-attention KV can be offloaded while a short sliding-window tail is reconstructed. Recurrent state requires its own checkpointing support; it cannot simply be treated as another reconstructible window. I would not generalize that path to every supported model architecture. Host-memory accounting also deserves an explicit budget. For my capacity estimate using the GLM AgentX configuration, I would budget `--hicache-size` in host-tier GB **per rank**. At 180 GB across each of four ranks, I would therefore allocate approximately 720 GB for those host cache tiers, not 180 GB for the whole deployment. I would reserve additional host capacity for the operating system and other processes. **Release labels do not prove feature availability.** When reviewing the GLM deployment recipe, I found two version gates to check before launch: the dated 2026-10-25 nightly includes checkpoint-loading change #40776, while the v0.5.22 release does not; an AMD monokernel pin predating sgl-amd-mono#60 rejects FP8 KV cache and causes startup to fail. My workaround would be an immutable image digest and a startup smoke test for the actual checkpoint, attention backend, and KV dtype. A package version alone is not enough. **Speculation changes scheduling as well as decoding.** I would not use NGRAM as a neutral, no-extra-weights substitute for EAGLE3. In my comparison, I would account for NGRAM's CUDA-only support, incompatibility with `--attn-dp-size > 1`, and disabling of overlap scheduling and mixed chunked prefill. That changes the experiment. **Cache correctness must survive transfer.** In my correctness review, I identified misplaced cache offsets during heterogeneous-disaggregation staging as a regression case to test. I would use the 127,500-token shared-prefix case, with 2 correct needles out of 128 before the fix and 128 out of 128 afterward, as a validation target. I treat that as an external regression-test lead, not my result or a universal failure rate. Before introducing disaggregation, I would require correctness tests across cold, partially cached, and heavily reused prefixes. A fast wrong answer does not qualify as throughput. Finally, I had no specific vLLM issue to inspect, so I cannot identify an affected version or recommend an issue-specific workaround. ## Scale, Latency & Cost vs. Alternatives My comparison starts with workload fit, not a blanket ranking. Serving choice | Why I would evaluate it | Main evaluation risk | Required decision evidence ---|---|---|--- SGLang with prefix reuse | Repeated multi-turn history | Eviction or routing destroys reuse | Warm/cold TTFT and token-weighted cache reuse SGLang with EAGLE3 or model-specific MTP | Decode-heavy responses | Draft and verification work outweigh accepted tokens | Acceptance behavior, latency, throughput, and memory at target load vLLM with prefix caching | Existing vLLM deployment and operational familiarity | An unmatched cache or scheduler configuration creates a misleading comparison | Same traces, weights, precision, load, and correctness checks Managed inference API | Lower infrastructure ownership | Limited cache controls and variable billing economics | Actual session-level spend, latency, and service guarantees I would not describe this as RadixAttention beating PagedAttention. PagedAttention concerns KV-memory management; automatic prefix caching concerns reuse. A meaningful vLLM comparison must enable and validate the relevant reuse behavior rather than compare a tuned SGLang configuration against an unrelated default. In my performance review, I separated the externally reported **26.75% output-throughput improvement and 36.25% reduction in mean TTFT at concurrency 384** from speculative-decoding gains: the change replaced context-length-driven kernel specialization with a runtime scalar, reducing compilation overhead rather than accelerating computation. These are not local benchmark results or measured benefits of enabling speculative decoding. I also treated the FlashInfer GDN recurrent-state checkpointing result—**47,771 to 53,004 tokens/s/GPU at 92.4% cache-hit rate** —as a separate, configuration-specific comparison. I cannot transfer that result to the Llama smoke-test recipe, a different GPU, or an ordinary chat workload. For capacity planning, I would use SLO-compliant completed sessions, not peak output tokens alone. For my AgentX replay plan, I would count concurrency as live sessions and track active requests separately, because subagent branches can create multiple simultaneous requests within each session. ### Practical break-even calculation My deployment decision would use: **Monthly benefit = removed GPU capacity + avoided external inference spend − added operating costs.** For an illustrative budget—not measured SGLang performance—assume: * One removable GPU allocation costs $3 per hour. * The service runs 730 hours per month. * Engineering and maintenance cost $6,000 initially. * Extra host resources and monitoring cost $400 monthly. Removing one continuously allocated GPU saves $2,190 monthly before the added operating costs. Net savings are $1,790, giving a simple payback of about **3.4 months**. But a benchmark improvement does not guarantee that an allocation can be removed. If the service still requires the same replica count for peak demand, failover, or tail latency, immediate GPU-bill savings may be zero. I would approve the change only after demonstrating that the smaller deployment meets the same TTFT and completion-latency objectives with acceptable correctness and failure behavior. ## My Recommendation: Evaluate SGLang Before Deployment My verdict is **shortlist SGLang for a controlled agent-serving evaluation; do not claim validated savings from this review alone**. I see a credible architectural fit between reusable prefixes, cache-aware routing, and long-running agent sessions. I also see enough state-management complexity that a generic throughput benchmark would give me false confidence. **I would deploy after validation if:** * My production traces contain substantial repeated token prefixes. * Warm-cache gains survive realistic interleaving and session churn. * The pinned image supports the exact target and draft checkpoints. * Speculation improves my operating point without unacceptable memory pressure. * Correctness remains stable across cache hits, eviction, and transfer. * The improvement removes billable capacity or meets an otherwise missed SLO. **I would hold off if:** * My prompts rarely share reusable prefixes. * My evidence consists only of immediate repeated requests. * I cannot identify the fixes included in the deployed build. * I need hybrid-cache or disaggregation behavior I have not tested. * My claimed savings depend on applying unrelated AgentX percentages. * I lack the traces needed for a fair comparison with my existing server. The next useful artifact is a pinned replay bundle: request traces, launch commands, runtime identity, cache telemetry, streaming timings, and correctness assertions. That is what turns an optimization story into a deployment decision. For adjacent infrastructure options, browse our tools collection. If the harder problem is designing the replay, routing, and capacity gates around your service, talk with our engineering team.
dev.to
November 12, 2026 at 12:40 AM
Last week results from sglang/vllm show millions of tokens per GPU on NVL72 -- ~800M tokens/second on 1MW of power. Enough to serve every doctor in the country simultaneously (1M at 50 q/day).

Doctors should leave inference efficiency worries to inference engineers.

vllm.ai/blog/2026-11...
DeepSeek-V4.1-Flash on Vera Rubin NVL72: 58x Throughput vs. GB300 NVL72 at 387 tokens/s
On one NVIDIA Vera Rubin NVL72 rack, vLLM serves DeepSeek-V4.1-Flash on SemiAnalysis AgentX at 1.7M tokens/s per GPU at a P90 of 387 tokens/s per user, 58x GB30
vllm.ai
November 13, 2026 at 12:50 PM
PyTorch inference hits TPU v7:

A megakernel for Muse Spark Open got 433 tok/s per user (up to 25.9K tok/s per chip) on AgentX (1M context). All 62 layers of the 816B MoE run as one Pallas program via @SGLang + TorchTPU.

• 1.94x faster per token
• 2x via DFlash2

Learn more: https://bit.ly/4zgrhBL
November 10, 2026 at 5:00 PM
The GPU monopoly on AI inference is officially facing a multi-cloud reality check.

For years, NVIDIA's software moat and CUDA ecosystem ruled supreme. But as inference workloads shift from single-turn chats to long-context, multi-turn AgentX benchmarks, hardware economics are changing fast.
November 6, 2026 at 11:30 AM
NVIDIA beschrijft geheugengebruik van Vera Rubin NVL72

NVIDIA beschrijft hoe het geheugen en de verbindingen van Vera Rubin NVL72 worden ingezet voor AI-inferentie. Volgens het bedrijf levert het systeem in SemiAnalysis AgentX tot 30 keer zov…

Lees op techspeeltuin.nl

#technieuws #HardwareEnChips
NVIDIA beschrijft geheugengebruik van Vera Rubin NVL72
NVIDIA beschrijft hoe het geheugen en de verbindingen van Vera Rubin NVL72 worden ingezet voor AI-inferentie. Volgens het bedrijf levert het systeem in SemiAnalysis AgentX tot 30 keer zoveel throughput per megawatt als…
techspeeltuin.nl
November 5, 2026 at 10:37 PM
SemiAnalysis AgentX benchmarks show Google TPUv7 Ironwood delivering up to 2.7x the tokens per dollar of Nvidia B200 and B300
The Iron(wood) Throne: Google's TPU Stakes Its Claim on Inference
Viable Alternative of CUDA Moat, NVIDIA scared and working nights & weekends, Jensen’s Counter Submission, Kimi K3, Muse Spark Open, Qwen, GLM5.3, TorchTPU, vLLM, SGLang, llm-d, Ray
newsletter.semianalysis.com
November 5, 2026 at 11:44 PM
NVIDIA Vera Rubin NVL72 Platform Squeezes More From Memory
NVIDIA says memory hierarchy, not brute HBM capacity, will decide who wins agentic inference, and I think the NVIDIA Vera Rubin NVL72 platform proves it. The pitch is efficiency over expansion, and for once the numbers back up the marketing. NVIDIA says HBM capacity per GPU rose roughly 3% from Blackwell Ultra to Vera Rubin while agentic throughput per dollar rose up to 30x on SemiAnalysis AgentX at 125 tokens per second per user. That last figure matters because NVIDIA frames the AI factory as a hierarchy where HBM is the fastest, most expensive, most contested tier for weights and KV cache. Below HBM sit CPU memory, local SSDs and shared storage, each trading speed for capacity at lower cost per bit, according to NVIDIA. NVIDIA says frontier models now carry trillions of parameters and agentic trajectories can process up to 3 million tokens, with RAG adding even more retained context. HBM becomes the bottleneck because it must hold both model weights and KV cache while bandwidth caps how fast data moves, a limit NVIDIA calls the memory wall. NVIDIA compares HBM to a warehouse where bandwidth is the loading dock, and a bigger warehouse without more docks just raises...
www.startuphub.ai
November 6, 2026 at 10:07 AM
Saudi Arabia launches AgentX AI hackathon in Riyadh to link skills to jobs

Backed by the communications ministry, teams build AI prototypes for real industries, aiming to match national talent with what employers need....
Saudi Arabia launches AgentX AI hackathon in Riyadh to link skills to jobs
Backed by the communications ministry, teams build AI prototypes for real industries, aiming to match national talent with what employers need.
tbreak.com
November 4, 2026 at 5:56 AM
Another DeepSeek advantage is open weights. Enthusiasts can optimize inference themselves. In just three weeks, the vLLM team got V4.1 Flash up to 599 TPS per user in the AgentX benchmark.
October 21, 2026 at 9:44 AM
vLLM's early Vera Rubin benchmarks report up to 7.8× AgentX throughput per GPU vs GB200. Underneath the headline: CUDA locality domains help MoE kernels read nearby weights. A workload-specific early result, not universal speedup.

vllm-project.github.io/2026/10/09/v...
vLLM Support for NVIDIA Vera Rubin NVL72
Early Rubin vLLM inference benchmarks, kernels and MoE locality.
vllm-project.github.io
October 11, 2026 at 3:15 PM
Driver shocked by these cat 😮 🐈
YouTube video by AgentX
youtube.com
October 3, 2026 at 5:47 PM
Among other things am delighted that almost 100 years ago someone else was doing the same thing I do to figure out birthdates. Smudgey pencil satisfaction no thank you Claude/ Chat/AgentX.
September 30, 2026 at 10:12 AM
What exactly should the next public cloud look like? For agents? Agents and humans? A layer on top of existing entities? Or not?

There are so many questions from DX to AgentX(?) to primitives, to orchestration, to shape, to zones, to hardware, to.. everything.
September 26, 2026 at 10:48 PM
Advancing Model Research in AgentX: Long-Horizon Autonomy for Industrial Recommender Systems

Shuang Yang et al.

#arXiv #cs.AI #cs.IR
Advancing Model Research in AgentX: Long-Horizon Autonomy for Industrial Recommender Systems
Sustaining industrial recommendation research requires using the results of one experiment to decide what to investigate next. We present AgentX-Model, the next generation of AgentX's model research framework, which connects proposal development and model experimentation within sandboxes defined by…
arxiv.org
September 26, 2026 at 12:05 AM
Shuang Yang, et al.: Advancing Model Research in AgentX: Long-Horizon Autonomy for Industrial Recommender Systems https://arxiv.org/abs/2609.30001 https://arxiv.org/pdf/2609.30001 https://arxiv.org/html/2609.30001
September 25, 2026 at 6:40 AM
Advancing Model Research in AgentX: Long-Horizon Autonomy for Industrial Recommender Systems

Kuaishou presents its dual-agent framework where a Research Agent forms proposals and a Model Agent investigates them.

📝 arxiv.org/abs/2609.30001
Advancing Model Research in AgentX: Long-Horizon Autonomy for Industrial Recommender Systems
Sustaining industrial recommendation research requires using the results of one experiment to decide what to investigate next. We present AgentX-Model, the next generation of AgentX's model research f...
arxiv.org
September 25, 2026 at 3:34 AM
I have mowed the backyard for the first time since the start of winter, and nuked all the garden beds with some agentX.

If the owner is expecting a manicured lawn so they can sell the place they better start paying a gardener.

They haven’t spent a cent on maintenance so the place is falling apart.
September 22, 2026 at 7:05 AM
New Model Architecture Implications for TAM of DRAM/NVMe, DeepSeek V4.1 Flash, AgentX, InferenceX, NVMe experiments
Engrams Embedding Entendre: Codesign for Efficient DRAM/SSD Offloading
newsletter.semianalysis.com
September 18, 2026 at 7:56 PM
New Engram technique offloads AI model memory to DRAM or SSD, easing pressure on scarce high-bandwidth memory chips
Engrams Embedding Entendre: Codesign for Efficient DRAM/SSD Offloading
New Model Architecture Implications for TAM of DRAM/NVMe, DeepSeek V4.1 Flash, AgentX, InferenceX, NVMe experiments
newsletter.semianalysis.com
September 18, 2026 at 10:59 PM
NVIDIAの新しい技術に関するニュースがありました。

Vera Rubin NVL72というものが、
これまでのものと比べて、
驚くほど効率よく動けるようになるそうです。

コーディングの作業において、
最大で30倍もの力を発揮するのだとか。

AIの進化のスピードは、
本当に目を見張るものがありますね。

これからどんな未来が待っているのか、
とても楽しみです。
www.zaikei.co.jp #news
NVIDIA Vera Rubin NVL72、AgentXでGB300比最大30倍のスループット/MW | 財経新聞
NVIDIAのVera Rubin NVL72が、AgentXのコーディング負荷でGB300比最大30倍のスループット/MWを示した。
www.zaikei.co.jp
September 18, 2026 at 8:08 AM
[Shocking] Vera Rubin achieves up to 30x difference in agent power efficiency benchmark

▼Read More
https://tech-matome.com/archives/45312
[Shocking] Vera Rubin achieves up to 30x difference in agent power efficiency benchmark - AIテクノロジーまとめ
Nvidia and semiconductor research firm SemiAnalysis announced that the Vera Rubin NVL72 demonstrated up to 30 times better power efficiency than the GB300 NVL72 in the AgentX benchmark, which simulates actual coding agent usage. Agent-style requests consume about 15 times more tokens than regular chat, making long-input processing and cache reuse critical to system performance. The AgentX dataset consists of traces selected from Claude Code sessions, using synthetic tokens to recreate the session structure. The 30x difference was recorded under specific conditions using DeepSeek-V4-Pro and measured points maintaining interactivity, meaning the same results may not apply to all processing. In inference systems, prefill (processing long inputs) and decode (generating answers) are handled separately, supported by NVLink connecting numerous GPUs. While generalizability to non-coding tasks and software optimization remain future challenges, establishing measurement methods for agent-based inference is expected to advance hardware comparative evaluation.
tech-matome.com
September 18, 2026 at 12:46 AM