#TensorRT-LLM
vLLM supports NVIDIA, AMD, and TPU with PagedAttention memory under Apache 2.0. SGLang's RadixAttention boosts multi-turn agents but recent RCEs warn on IPC isolation. How do you balance caching benefits with strict container isolation?

techwhoknows.com
vLLM vs SGLang vs TensorRT-LLM (2026 Comparison): Throughput, KV Caching, and Security
Compare vLLM, SGLang, and TensorRT-LLM in 2026 across PagedAttention, RadixAttention prefix caching, TensorRT compilation, and inference server security.
techwhoknows.com
November 16, 2026 at 6:21 AM
KI verarbeitet dezentrale Patientenakten innerhalb der Krankenhaus-Firewall und liefert rückverfolgbare klinische Signale. Synapsis AI von Dyania Health nutzt NVIDIA NeMo, TensorRT und TensorRT-LLM, um Klinikern mehr Zeit…
https://kiboti.vercel.app/magazine/332f7bd29e7a
🤖 Automatisierter KI-Beitrag
November 14, 2026 at 3:01 AM
AI Computing Software Development Engineer, TensorRT-LLM - Taiwan, Taipei Job educativ.net/jobs/job/68305...
November 11, 2026 at 7:14 PM
TensorRT-LLM 1.4 Makes trtllm-serve Run 8 HTTP Frontends by Default: RAM Cost, Connection Balance, and What Still Differs
## TL;DR TensorRT-LLM 1.4.0rc3 (GitHub release 2026-11-06) flips one default that changes the shape of your deployment: `trtllm-serve` now starts **8 HTTP frontend processes** instead of 1. The release notes mark it BREAKING (PR #19379). I read that PR, the earlier prototype PR #16523, the metrics fix #19394, the current docs on `main`, and the release notes, then ran two small simulations of my own. I did not run a GPU workload. * **The reason is host-side, not GPU-side.** The prototype PR measured `gen_preprocessing` p50 at 54 ms (concurrency 512), 109 ms (1024) and 1,520 ms (2048) on a DeepSeek V4 disaggregated generation worker while decode step time stayed flat. One asyncio loop was the queue. * **It costs host RAM.** The PR measured about 1.99 GB resident per attached frontend and about 2.5 GB for the launcher on a Qwen3-0.6B test. That is roughly 16.4 GB for 8 frontends versus 2.5 GB for 1, and time to healthy went from 54 s to 154 s in that one test. GPU memory is unchanged. * **Balance is per connection, not per request.** In my model, 8 frontends sharing a load worth 3 event loops stay comfortable only with many connections. With 16 long-lived connections the busiest frontend is above 1.0 utilization in 7.7% of random draws. * **Some of the scary caveats in the PR body are already fixed.** The metrics sharding described in #19379 was addressed by #19394, merged two days earlier and shipped in the same rc. Still different: the stateful Responses API store is off, `/health` is per frontend, and perf-metrics files are per frontend. * **Where this does not apply.** PyPI's latest non-prerelease `tensorrt-llm` is 1.2.1 as I write this, so a plain `pip install` does not get this default. ## What exactly changed in trtllm-serve 1.4.0rc3? The CLI default for `--num_serve_frontends` on the `trtllm-serve` serve subcommand went from 1 to 8. The launcher process owns the engine. The other 7 frontends re-exec the command line, attach to the same executor, and bind the same port with `SO_REUSEPORT`, so clients still see one URL. Three details from the PR are easy to miss: 1. **Only the CLI default changed.** `LlmArgs.num_serve_frontends` still defaults to 1, so `trtllm-bench`, `trtllm-eval` and a bare Python `LLM()` do not pay for the multi-frontend topology. 2. **YAML wins over the new default.** `num_serve_frontends: 1` in the `--config` file overrides it, as does `--num_serve_frontends 1` on the command line. 3. **Some configurations quietly fall back to 1.** If you did not type the flag and the config cannot run several frontends, the server logs why and runs one. The current docs list `--port 0` or `--report_addr`, `orchestrator_type` set to `rpc` or `ray`, `enable_resource_governor`, and `--enable_sleep_mode`. `--grpc` also keeps one unless you ask for more. If you type `--num_serve_frontends 8` together with one of those, startup fails instead. The same release candidate carries two other BREAKING entries (#21036 removes unused `LlmRequest` fields, #21041 removes obsolete TensorRT-backend compatibility surfaces). Those are API removals. The frontend default is the one that changes what a plain launch does on your host. Here is the shortest way to opt out: # CLI trtllm-serve <model> --num_serve_frontends 1 # or in the --config YAML num_serve_frontends: 1 ## Why would one HTTP process become the bottleneck? Every request on a single `trtllm-serve` process goes through one asyncio event loop: `json.loads`, pydantic validation, and then every streamed chunk written back. The prototype PR (#16523, merged 2026-07-22) reported this on a DeepSeek V4 disaggregated generation worker: Concurrency | `gen_preprocessing` p50 ---|--- 512 | 54 ms 1,024 | 109 ms 2,048 | 1,520 ms Doubling concurrency from 512 to 1,024 roughly doubled the delay. Doubling it again multiplied it by about 14, while decode step time stayed flat. The PR author called this an M/M/1-style blow-up, and I agree with the shape. In a single-server queue the mean wait grows like utilization divided by one minus utilization, so it looks harmless until the loop approaches saturation and then explodes. Adding GPU capacity does nothing for it, because the GPU is not the queue. That is why eight processes help: the same parsing and streaming work is spread over eight event loops and eight GILs. The PR says the setting was already the production configuration in the authors' agentic disaggregated setups, with 8 on context workers and 4 on generation workers. ## Does SO_REUSEPORT really balance the load? The kernel assigns each **new connection** to one of the listening sockets, effectively by hashing the connection tuple. The docs say it plainly: each request is handled entirely by the frontend that accepted its connection. Once a client holds a keep-alive connection, all of its requests land on the same frontend. So the question that matters is how many long-lived connections your clients open. I modeled it with the standard library only. The assumptions are mine and deliberately simple: connections are assigned uniformly at random to 8 frontends, every connection carries equal load, and the total offered load equals 3 single-process event loops (so one frontend alone would be hopelessly saturated). Perfect balance would put each frontend at 0.375 utilization, a mean M/M/1 wait of 0.6 service times. The core of the model: import random K, RHO_TOTAL = 8, 3.0 def busiest_utilization(n_conn): counts = [0] * K for _ in range(n_conn): counts[random.randrange(K)] += 1 return RHO_TOTAL * max(counts) / n_conn def wait_factor(u): # M/M/1 mean wait in service times return float("inf") if u >= 1 else u / (1 - u) I ran 20,000 draws per connection count with a fixed seed: Long-lived connections | Median busiest-frontend utilization | 99th percentile | Draws with a saturated frontend ---|---|---|--- 8 | 1.125 | 1.875 | 50.5% 16 | 0.750 | 1.312 | 7.7% 32 | 0.656 | 1.031 | 1.0% 64 | 0.562 | 0.797 | 0% 256 | 0.469 | 0.574 | 0% 1,024 | 0.422 | 0.472 | 0% With 1,024 connections you are within 13% of perfect balance. With 16 you are at a median utilization of 0.75, a mean wait of 3.0 service times against the balanced 0.6, and about one draw in thirteen saturates a frontend. This is an illustration, not a measurement of TensorRT-LLM. Real traffic is lumpier than equal-weight connections (a long streaming response weighs more than a short one), which I expect makes the tail worse, but I have not measured that. If your traffic comes through a router or gateway with a small connection pool, count its connections to this server before assuming 8 frontends means 8x headroom. ## What does the new default cost in host memory and startup time? The PR's verification section is the only measurement I found, so I use it as is. One container with 1x GB300, Qwen3-0.6B, launched the way that cluster runs production: Case | LISTEN sockets on the port | Healthy after ---|---|--- Default (8 frontends) | 8 | 154 s `--num_serve_frontends 1` | 1 | 54 s Resident memory was about 1.99 GB per attached frontend and about 2.5 GB for the launcher. Every extra frontend is a full Python process that imports TensorRT-LLM and loads the tokenizer, which is what the docs warn about. The arithmetic for the default is 2.5 + 7 x 1.99 = **16.43 GB** , about 13.9 GB more than a single frontend. Two things to keep in mind. First, this is one run on a tiny model, so treat 1.99 GB as an order of magnitude, since a larger tokenizer or extra imports will move it. Second, a container memory limit sized around a single serving process will now get OOM-killed during startup. Attached frontends hold no GPU memory: in the PR, `nvidia-smi` listed only the worker (58.7 GiB including KV cache) and the launcher (748 MiB), identical with 1 or 8 frontends. ## Is the metrics sharding in the PR body still a problem? This is the part I almost got wrong. The body of #19379 lists a caveat: `/metrics`, `/prometheus/metrics` and `/health` are answered by whichever frontend accepted the connection, so scrapes see one frontend's share and counters appear to reset. It cites a measurement of `trtllm_request_success_total` showing 5 distinct values (6, 7, 8 and 10 among them) for 64 requests across 32 scrapes. But #19394, "make multi-frontend metrics, profiling and liveness launcher-owned", was merged on 2026-10-28, two days before the default flip, and I confirmed with the compare API that both land first in rc3 (neither is in rc2). It reports the same 32 scrapes returning one identical value (64) afterwards. The mechanism: the launcher sets one shared `PROMETHEUS_MULTIPROC_DIR` before it spawns children, iteration-derived series come from the launcher only, and attached frontends forward `GET /metrics` to the launcher over a Unix socket. The current docs on `main` match that: `/metrics` is forwarded and `/prometheus/metrics` covers every frontend. Why does this matter to you? Because Prometheus treats any decrease in a counter as a reset. To show what the sharded behavior would have done to a dashboard, I simulated 64 requests landing on 8 per-process counters, scraped at random every second request: With shared counters the increase is the true value, 62 between the first and last scrape, every time. With per-frontend counters the 5th to 95th percentile spread is 47 to 86 (minimum 23, maximum 111), and 38% of runs were off by more than 10. Illustrative again, but it is why a build without #19394 and with 8 frontends is a bad combination for anything alerting on request rates. You can check your own server in a few seconds. This probe scrapes a counter repeatedly and flags any decrease. I tested it against a mock HTTP server that I wrote in both modes (it reported `consistent` for shared counters and a drop count for sharded ones). I have not pointed it at a real TensorRT-LLM server. import re, time, urllib.request def total(url, name): text = urllib.request.urlopen(url, timeout=10).read().decode() return sum(float(m.group(1)) for m in re.finditer( r'^' + re.escape(name) + r'(?:\{[^}]*\})?\s+([0-9.eE+-]+)', text, re.M)) def probe(url, name="trtllm_request_success_total", n=30, gap=0.05): s = [] for _ in range(n): s.append(total(url, name)); time.sleep(gap) drops = sum(b < a for a, b in zip(s, s[1:])) return {"distinct": len(set(s)), "drops": drops, "verdict": "per-frontend counters" if drops else "consistent"} Send some traffic while it runs. If the counter is flat you cannot tell the difference. ## What still behaves differently with several frontends? Per the current docs and the PRs, these remain: Item | Behavior with more than 1 frontend ---|--- Responses API store (`store`, `previous_response_id`) | Disabled, because the store is in-process memory. Use 1 frontend if you rely on it. `/health` | Describes the frontend that answered. #19394 adds a watchdog so attached frontends notice a dead launcher or engine and shut down, instead of 7 of 8 answering 200 with no engine. Per-request perf metrics | Each frontend writes its own file. The PR suggests `perf_metrics_output_dir` JSONL across processes. Disaggregated `/steady_clock_offset` handshake | Still calibrates only the answering frontend, per #19394. Port sharing | `SO_REUSEPORT` would let another server of the same user join the port and take traffic, so the launcher refuses a port that another server already uses. The `/health` row is the one I would test in your environment. A load balancer health check that lands on a different frontend each time is exactly the kind of thing that passes in staging and flaps in production. ## What should I check before upgrading? 1. **Are you on 1.4.0rc3 or main, not 1.2.1?** If you install from PyPI without `--pre`, this does not affect you yet. 2. **Does the memory limit cover 16 GB of extra frontends** , or should you set `num_serve_frontends` to 2 or 4? The PR gives no sizing rule, only that production disaggregated configs use 8 and 4. 3. **How many connections do your clients hold?** Under roughly 32, expect imbalance (table above). 4. **Do you use the Responses API store?** If yes, pin 1. 5. **Do your dashboards and alerts survive?** Run the probe, and confirm your build includes #19394. 6. **Did you rely on a quiet fallback?** If your config hit one of the fallback conditions, you are already on 1 frontend and will see a log line saying why. 7. **Re-measure.** Record `gen_preprocessing` p50 and time to healthy before and after, since the numbers in the PR come from one cluster and one small model. ## What I did not do I ran no GPU workload and no TensorRT-LLM server. The latency, memory and startup numbers are quoted from the PRs, and I attribute them as such. The two simulations are my own, use simplifying assumptions stated above, and should be read as intuition about the mechanism, not as predictions for your fleet. My statements about which rc contains which PR come from the GitHub compare API and the rc3 release notes on 2026-11-11. ## FAQ **Does running 8 frontends use more GPU memory?** No. In the PR's test, attached frontends held no GPU memory. `nvidia-smi` listed only the worker and the launcher, in both the 8-frontend and 1-frontend runs. **How do I go back to a single frontend?** Pass `--num_serve_frontends 1` or set `num_serve_frontends: 1` in the `--config` YAML. The YAML value overrides the new CLI default. **Will`pip install tensorrt-llm` give me this default?** Not today. The latest non-prerelease on PyPI is 1.2.1. The change is in the 1.4.0rc3 release candidate (also uploaded to PyPI on 2026-11-06) and on `main`. **How many frontends should I run?** The PR does not give a formula. Its production configs use 8 on context workers and 4 on generation workers. I would start from the free CPU cores and the RAM budget above, then measure request-handling latency at your real concurrency. **Is the stateful Responses API store gone?** It is disabled whenever more than one frontend runs, because it lives in each process's memory. With `--num_serve_frontends 1` it works as before. **Can another process steal traffic through SO_REUSEPORT?** The docs say the launcher refuses a port that another server already uses, which is the guard against a second server of the same user joining the port. ## Further reading * Day-0 Model Support Is a Fork: What 26 NVIDIA Dynamo Snapshot Builds Say About Pinning Your LLM Serving Stack * Your LLM Serving Engine Ships Every 13 Days: Reading vLLM's Breaking-Change Lists as Upgrade Risk Data ## Sources * TensorRT-LLM v1.4.0rc3 release notes * PR #19379: default num_serve_frontends to 8 with single-frontend fallback * PR #16523: multi-process HTTP frontends on the classic IPC executor path * PR #19394: multi-frontend metrics, profiling and liveness launcher-owned * trtllm-serve docs, Multiple HTTP Frontends * tensorrt-llm on PyPI
dev.to
November 11, 2026 at 2:39 AM
💫 NVIDIA TensorRT LLM ускоряет работу больших языковых моделей на GPU

Библиотека TensorRT LLM оптимизирует выполнение LLM и визуальных генеративных моделей на NVIDIA GPU с помощью специализированных ядер и эффективного runtime.

• Специализированные ядра для общих операций
• Эффективный runtime
• …
November 8, 2026 at 6:50 AM
FLUX 3 Image Klein on NVIDIA B200 reached up to 3.46x faster text-to-image and 4.45x faster image-to-image generation at 2048×2048 with combined TensorRT LLM VisualGen optimizations versus uncompiled BF16 inference.
https://agxsearch.ai/t/7ayhzc?utm_source=bluesky&utm_medium=social
November 8, 2026 at 2:57 PM
⚡ Inference and serving: #llamacpp, #Ollama, #vLLM, #TensorRT-LLM, #WebLLM and #LitServe
November 7, 2026 at 11:00 AM
Deploying Sovereign LLMs in Air-Gapped Enterprise VPCs: vLLM PagedAttention, TensorRT-LLM, and AWQ Quantization
# Deploying Sovereign LLMs in Air-Gapped Enterprise VPCs: vLLM PagedAttention, TensorRT-LLM, and AWQ Quantization _By Soham Sharma · Principal AI Architect, Botmartz AI Solutions_ > _Original canonical engineering dissertation published at Botmartz Insights: Deploying Sovereign LLMs._ ## Executive Summary: The Data Sovereignty & Compliance Mandate For enterprises operating across regulated verticals—commercial banking, statutory compliance, defense aerospace, and clinical healthcare—routing proprietary unstructured data to public multi-tenant AI endpoints (such as OpenAI, Anthropic, or public cloud model hubs) is increasingly prohibited by corporate governance and statutory legislation. Under the **European Union AI Act** , **General Data Protection Regulation (GDPR)** , **India's Digital Personal Data Protection Act (DPDPA)** , and **US Department of Defense FedRAMP High** standards, enterprise customer data must satisfy three non-negotiable requirements: 1. **Zero Data Egress** : Model weights and customer prompts must never traverse the public internet. 2. **Deterministic Data Residency** : Storage, memory caching, and inference must occur exclusively within dedicated enterprise Virtual Private Clouds (VPC) on bare-metal or sovereign cloud tenancies. 3. **Guaranteed Zero Training Retention** : Absolute cryptographic guarantees that client documents are never used for continual model pre-training. Meeting these mandates requires self-hosting open-weights foundation models (e.g. Llama 3.3 70B, Qwen 2.5 72B, DeepSeek V3) in **air-gapped, zero-egress VPC topologies**. However, naive self-hosting regularly fails due to catastrophic GPU memory fragmentation and abysmal token-per-second unit economics. This paper specifies the engineering blueprint for running high-throughput, sovereign LLM inference appliances in air-gapped VPCs using **vLLM PagedAttention** , **Continuous Batching** , and **Activation-Aware Weight Quantization (AWQ)**. ## 1. The Physics of High-Throughput Inference: The KV Cache Bottleneck In autoregressive transformer generation, generating each new token requires attending to all prior tokens in the sequence. To avoid recomputing Key and Value activations at every step, transformers maintain a **Key-Value (KV) Cache** in GPU High-Bandwidth Memory (HBM). ### 1.1 The Memory Footprint of the KV Cache For a model with $L$ layers, $H$ key-value heads, and head dimension $D$, the KV cache memory consumed per token in FP16 precision is: $$\text{Memory}_{\text{token}} = 2 \times 2 \times L \times H \times D \quad \text{bytes}$$ For a Llama-3-70B model ($L = 80$, $H = 8$ with Grouped Query Attention, $D = 128$): * Each token consumes **327,680 bytes (320 KB)**. * For a single request with an 8,192-token context window, the KV cache alone consumes **2.62 GB of VRAM**. * For 50 concurrent requests, the KV cache requires **131 GB of VRAM** —completely exhausting the memory of two 80GB Nvidia H100 SXM GPUs before accounting for the model's 140 GB weight footprint! [ TRADITIONAL CONTIGUOUS VRAM ALLOCATION - CATASTROPHIC FRAGMENTATION ] ┌────────────────────────┬──────────────────────────┬────────────────────────┐ │ Request A (Allocated) │ Wasted Internal Gap (60%)│ Request B (Allocated) │ └────────────────────────┴──────────────────────────┴────────────────────────┘ Result: 60-80% of expensive GPU VRAM sits idle due to static over-allocation! [ vLLM PAGEDATTENTION ALLOCATION - VIRTUAL MEMORY BLOCKS ] ┌───────┬───────┬───────┬───────┬───────┬───────┬───────┬───────┐ │ Blk 1 │ Blk 2 │ Blk 3 │ Blk 4 │ Blk 5 │ Blk 6 │ Blk 7 │ Blk 8 │ └───────┴───────┴───────┴───────┴───────┴───────┴───────┴───────┘ Result: Zero fragmentation; blocks allocated on-demand like OS virtual memory. ### 1.2 PagedAttention: Virtual Memory for GPU Tensors Traditional inference engines allocate contiguous blocks of physical VRAM based on the _maximum possible sequence length_ ($T_{\max}$), resulting in 60%–80% internal memory fragmentation. **PagedAttention** solves this by borrowing the classical operating system concept of **virtual memory paging** : * The KV cache is partitioned into fixed-size physical blocks (e.g. 16 or 32 tokens per block). * A logical-to-physical block table dynamically maps non-contiguous physical memory chunks during generation. * **System Impact** : VRAM waste drops to under 4%, allowing **2.8× to 4.5× higher concurrency** on identical GPU clusters. ## 2. Quantization Mathematics: AWQ vs. GPTQ vs. FP8 Running a 70-billion-parameter model at 16-bit precision requires ~140 GB of VRAM just to load weights, necessitating multi-GPU tensor parallelism ($2 \times \text{H100 80GB}$). To run high-throughput inference on cost-effective enterprise hardware ($4 \times \text{L40S 48GB}$ or $2 \times \text{A100 80GB}$), model weights must be quantized without degrading reasoning perplexity. [ WEIGHT QUANTIZATION TRADE-OFF MATRIX ] Quantization Method Perplexity Loss Throughput (tok/s) VRAM Footprint ───────────────────────────────────────────────────────────────────────────── FP16 (Baseline) 0.0% (Zero loss) 100% 140 GB (70B) AWQ 4-Bit < 0.1% (Near-zero) 240% (GEMM speedup) 38 GB (70B) GPTQ 4-Bit ~ 0.5% (Measurable)190% 38 GB (70B) RTN (Round-to-Near) ~ 4.8% (Severe) 220% 38 GB (70B) ### Why Activation-Aware Weight Quantization (AWQ) Dominates Naive Round-to-Nearest (RTN) quantization clips all weights uniformly. However, research proves that **only 0.1% to 1% of salient weight channels dictate the entire model's output quality**. These salient channels correspond to large activation magnitudes. AWQ identifies these critical channels by observing activation distributions on a small calibration set and applies per-channel protection scales: $$\mathbf{W}' = \text{Quantize}(\mathbf{W} \cdot \mathbf{S}) \cdot \mathbf{S}^{-1}$$ Where $\mathbf{S}$ is an optimal diagonal scaling matrix chosen to minimize the output activation reconstruction error: $$\arg\min_{\mathbf{S}} |\mathbf{W} \mathbf{X} - \text{Quantize}(\mathbf{W} \cdot \mathbf{S}) \cdot \mathbf{S}^{-1} \mathbf{X}|_2$$ Because salient channels remain unclipped, **AWQ achieves near-lossless 4-bit quantization** while reducing model memory footprint by 72% and doubling generation throughput via INT4 Tensor Core hardware acceleration. ## 3. Air-Gapped AWS Private VPC Topology To satisfy strict CISO audit mandates, the sovereign inference cluster must operate within an isolated subnet with **Zero Internet Gateway (IGW)** access. ┌─────────────────────────────────────────────────────────────────────────┐ │ ENTERPRISE AIR-GAPPED VPC │ │ │ │ ┌─────────────────────────────────────────────────────────────────┐ │ │ │ Private Application Subnet (10.0.1.0/24) │ │ │ │ │ │ │ │ [ Internal Enterprise App / Agent Orchestrator ] │ │ │ │ │ │ │ │ │ ▼ (Internal Private DNS) │ │ │ │ [ Private Application Load Balancer (Internal ALB) ] │ │ │ │ │ │ │ │ │ ▼ │ │ │ │ ┌─────────────────────────────────────────────────────────┐ │ │ │ │ │ GPU Inference Worker Subnet (10.0.2.0/24) │ │ │ │ │ │ │ │ │ │ │ │ ┌───────────────────────┐ ┌───────────────────────┐ │ │ │ │ │ │ │ vLLM Worker Node 1 │ │ vLLM Worker Node 2 │ │ │ │ │ │ │ │ (4x Nvidia L40S 48GB) │ │ (4x Nvidia L40S 48GB) │ │ │ │ │ │ │ │ Port 8000 (OpenAI API)│ │ Port 8000 (OpenAI API)│ │ │ │ │ │ │ └───────────────────────┘ └───────────────────────┘ │ │ │ │ │ └─────────────────────────────────────────────────────────┘ │ │ │ └─────────────────────────────────────────────────────────────────┘ │ │ │ │ │ ┌────────────────────────────────▼────────────────────────────────┐ │ │ │ AWS VPC Endpoints (PrivateLink) │ │ │ │ • com.amazonaws.us-east-1.s3 (Pre-loaded Model Weights) │ │ │ │ • com.amazonaws.us-east-1.ecr (Container Image Repository) │ │ │ │ • com.amazonaws.us-east-1.logs (CloudWatch Audit Streams) │ │ │ └─────────────────────────────────────────────────────────────────┘ │ │ │ │ >>> NO INTERNET GATEWAY · NO NAT GATEWAY · ZERO PUBLIC IP ADDRESSES <<<│ └─────────────────────────────────────────────────────────────────────────┘ ## 4. Production Deployment Specification: Docker & vLLM Engine Below is the production Docker configuration deploying a sovereign **Llama-3.3-70B-Instruct-AWQ** instance with 4-way tensor parallelism, PagedAttention, and OpenAI-compatible API schemas: version: '3.8' services: vllm-sovereign-node: image: vllm/vllm-openai:v0.6.3.post1 container_name: vllm-sovereign-appliance restart: always runtime: nvidia environment: - CUDA_VISIBLE_DEVICES=0,1,2,3 - NCCL_DEBUG=INFO - HF_HUB_OFFLINE=1 # Strictly air-gapped: no HuggingFace downloads volumes: # Local NVMe drive containing pre-verified model weights and tokenizer - /mnt/nvme/models/Llama-3.3-70B-Instruct-AWQ:/models/llama-3.3-70b-awq:ro - /mnt/nvme/cache:/root/.cache ports: - "8000:8000" command: > --model /models/llama-3.3-70b-awq --quantization awq --tensor-parallel-size 4 --max-model-len 8192 --max-num-seqs 64 --gpu-memory-utilization 0.92 --block-size 16 --enable-prefix-caching --disable-log-requests --enforce-eager --port 8000 deploy: resources: reservations: devices: - driver: nvidia count: 4 capabilities: [gpu] healthcheck: test: ["CMD-SHELL", "curl -f http://localhost:8000/health || exit 1"] interval: 30s timeout: 10s retries: 3 start_period: 120s ## 5. Cost-per-Token Economics: Self-Hosted Sovereign vs. Public APIs Operating an enterprise AI workload generating **100 million tokens per month** (e.g. 50M input tokens, 50M output tokens): Architecture | Monthly Infrastructure Cost | p95 TTFT (Time to First Token) | Data Egress Risk | Compliance Status ---|---|---|---|--- **OpenAI GPT-4o Public API** | ~$1,125 / month (variable scale) | 480 ms – 1,200 ms | **High (Third-Party Cloud)** | Fails FedRAMP High / strict banking laws **Anthropic Claude 3.5 Sonnet** | ~$2,250 / month (variable scale) | 520 ms – 1,400 ms | **High (Third-Party Cloud)** | Fails strict air-gapped sovereign VPC **Sovereign vLLM (4x L40S 48GB)** | **$1,650 / month (Fixed reserve)** | **82 ms** | **ZERO (Air-Gapped Private VPC)** | **100% Compliant (GDPR, DPDPA, HIPAA)** **The Operational Dividend** : Beyond regulatory survival, dedicated sovereign instances deliver fixed, predictable infrastructure costs that do not balloon with usage, paired with **p95 first-token latencies under 100ms**. ## Frequently Asked Questions (FAQ / AEO) ### What is the difference between PagedAttention and standard attention? Standard attention stores the key-value (KV) cache in contiguous GPU memory blocks sized for the maximum possible sequence length, wasting 60-80% of memory due to fragmentation. PagedAttention divides the KV cache into fixed-size virtual memory pages (like an operating system), allocating blocks dynamically as tokens are generated to eliminate fragmentation and boost throughput by 2-4x. ### Can an air-gapped LLM run without internet connectivity during inference? Yes. By setting `HF_HUB_OFFLINE=1` and pre-mounting verified model weights from a secure local NVMe mount or private enterprise S3 VPC endpoint, the inference engine operates with zero outbound internet connectivity. ### Why is AWQ preferred over GPTQ for enterprise inference? AWQ (Activation-Aware Weight Quantization) identifies and protects the salient 1% of weight channels that dominate activation outliers. This preserves model reasoning, code generation, and mathematical accuracy with near-zero perplexity degradation compared to GPTQ, while running at full 4-bit INT4 hardware acceleration. _Botmartz AI Solutions designs defense-grade enterprise AI infrastructure, air-gapped LLM deployments, and sovereign private VPC systems. Explore our solutions at botmartz.com/insights or enroll in our Forward Deployed AI Engineering Masterclass._
dev.to
November 7, 2026 at 8:37 PM
Blackwell GPUで画像生成を最大4.45倍高速化:FLUX 3 Image KleinとTensorRT LLM VisualGenの最適化

Blackwell上の画像生成を最大4.45倍高速化

#画像生成 #TensorRT-LLM #Blackwell #拡散モデル #推論最適化
Blackwell GPUで画像生成を最大4.45倍高速化:FLUX 3 Image KleinとTensorRT LLM VisualGenの最適化
Blackwell上の画像生成を最大4.45倍高速化
ai.warp-studio.com
November 7, 2026 at 12:04 AM
NVIDIA: Dynamo v1.6.2: SGLang image on CPython 3.12.16, HTTP 400 on TensorRT-LLM token budget overrun, opt-in GC freeze for SGLang workers
November 6, 2026 at 6:15 AM
NVIDIA versnelt FLUX 3 Image Klein op Blackwell-gpu's

FLUX 3 Image Klein genereert volgens NVIDIA sneller beelden op een B200-gpu wanneer ontwikkelaars de optimalisaties in TensorRT LLM VisualGen combineren.

Lees op techspeeltuin.nl

#technieuws #AI
NVIDIA versnelt FLUX 3 Image Klein op Blackwell-gpu's
FLUX 3 Image Klein genereert volgens NVIDIA sneller beelden op een B200-gpu wanneer ontwikkelaars de optimalisaties in TensorRT LLM VisualGen combineren.
techspeeltuin.nl
November 7, 2026 at 8:53 AM
🌍 Remote SWE Jobs

▸ Feltsense AI Engineer @ Silver.dev
$60k-$110k
▸ ML Inference Engineer – Terminal-Bench Task Designer (vLLM / SGLang / TensorRT-LLM) @ Toptal

👇 crackr.dev/jobs/remote-software-engineer-jobs

#RemoteJobs #SWE #TechJobs #Hiring
November 7, 2026 at 2:00 PM
FLUX 3 Image Klein hits 3.46x faster image generation on a single B200. NVIDIA's TensorRT LLM VisualGen combines NVFP4 quantization, fused kernels, and multi-GPU parallelism to cut diffusion latency.
FLUX 3 Image Klein hits 3.46x faster image generation on a single B200 - GenAInews
NVIDIA's TensorRT LLM VisualGen combines NVFP4 quantization, fused kernels, and multi-GPU parallelism to cut diffusion latency.
genainews.tech
November 7, 2026 at 12:15 AM
TensorRT Edge-LLM으로 DGX Spark에서 Nemotron 3.5 Lightning 실행하기 (DSpark로 생성 속도 최대 1.7배)

핵심 요약

• NVIDIA가 기술 블로그 튜토리얼에서 DGX Spark 64GB에 TensorRT Edge-LLM 0.11.1을 설치해 Nemotron 3.5 Lightning(전체 30B, 활성 3B, NVFP4)을 실행하는 절차를 공개했습니다.
• trtedge chat 명령 하나로 약 21GB 모델 다운로드, 엔진 빌드, 채팅이 이어지고, trtedge serve로 Open…
TensorRT Edge-LLM으로 DGX Spark에서 Nemotron 3.5 Lightning 실행하기 (DSpark로 생성 속도 최대 1.7배)
핵심 요약 • NVIDIA가 기술 블로그 튜토리얼에서 DGX Spark 64GB에 TensorRT Edge-LLM 0.11.1을 설치해 Nemotron 3.5 Lightning(전체 30B, 활성 3B, NVFP4)을 실행하는 절차를 공개했습니다. • trtedge chat 명령 하나로 약 21GB 모델 다운로드, 엔진 빌드, 채팅이 이어지고, trtedge serve로 OpenAI 호환 API 서버를 띄웁니다(문서상 실험 단계, 기본 동시 요청 1개). • NVIDIA 측정(배치 1, 출력 128토큰 제한)에서 DSpark 추측 디코딩은 생성 속도를 1.45~1.7...
discuss.pytorch.kr
November 4, 2026 at 10:35 AM
🚨 EUVD-2026-110820
📊 8.4/10
🏢 NVIDIA

📝 NVIDIA TensorRT-LLM for any platform contains a vulnerability in KvCacheAwareRouter tokenizer loading, where an attacker could cause inclusion of functio...

🔗 https://euvd.enisa.europa.eu/vulnerability/EUVD-2026-110820

#cybersecurity #infosec #cve #euvd
November 4, 2026 at 6:02 PM
🚨 EUVD-2026-110821
📊 7.1/10
🏢 NVIDIA

📝 NVIDIA TensorRT-LLM contains a vulnerability where an attacker could cause a server-side request forgery. A successful exploit of this vulnerability migh...

🔗 https://euvd.enisa.europa.eu/vulnerability/EUVD-2026-110821

#cybersecurity #infosec #cve #euvd
November 4, 2026 at 6:02 PM
🚨 EUVD-2026-110822
📊 7.1/10
🏢 NVIDIA

📝 NVIDIA TensorRT-LLM contains a vulnerability where an attacker could cause a server-side request forgery. A successful exploit of this vulnerability migh...

🔗 https://euvd.enisa.europa.eu/vulnerability/EUVD-2026-110822

#cybersecurity #infosec #cve #euvd
November 4, 2026 at 6:02 PM
Aligning a large language model with RLHF usually means stitching together reward training, PPO and slow generation across many GPUs. NVIDIA NeMo-Aligner from packages SFT, reward models, PPO, REINFORCE, DPO, SPIN and SteerLM, with TensorRT-LLM speeding up generation and tensor, data and...
November 3, 2026 at 1:18 AM
これ、無料で試せます。

要約すると、DGX Spark 64GBをプラットフォームとして活用し、数コマンドでAIモデルの実行が可能。
軽量なC++推論ランタイムにより、エッジデバイス向けの開発からデプロイまでを統一。

無料なら試してみますか?
ai-tech-express.jp/articles/nvidia-tensorrt-edge-llm-edge-ai
NVIDIA TensorRT Edge-LLMとは?使い方と日本での提供状況
NVIDIAが公開したTensorRT Edge-LLMの概要、使い方、性能数値を解説。DGX Spark 64GBでのNemotron-3.5 Lightning実行や推論加速の手順を紹介します。
ai-tech-express.jp
November 2, 2026 at 2:13 PM
vLLMとSGLangを載せ替えて測ると、24GB×4の直答はvLLMが418対272 tok/s、96GB×1のARC3はSGLang系でLB 2.68→11.07(要因は未分離)。差が開いたのはKVが収まるかどうかと読んでいる。迷ったらvLLMで始め、先に同時本数×文脈長の収支を見たい。
yurudeep.com/posts/deeple...
【調べた】vLLMとSGLang、どっちを使う?──比較記事の結論がバラバラなので、ARC-AGIコンペの実測も入れてGPUメモリの収支で整理した - ゆるディープ
vLLMとSGLangの性能比較は、記事によって結論が逆になる。公式情報と論文で両者の得意な場面を整理し、ARC-AGI-2/3コンペで同じ負荷を載せ替えた実測を加えた。勝敗を分けたのはエンジンの評判ではなくKVキャッシュのメモリ収支だった、という話まで。TensorRT-LLMなど他の選択肢の位置づけも載せる。
yurudeep.com
November 4, 2026 at 11:07 AM
NVIDIA: TENSORRT EDGE-LLM RUNS NEMOTRON-3.5 LIGHTNING ON DGX SPARK 64GB, 126.9 TOKENS/SEC WITH DSPARK
October 31, 2026 at 3:00 PM
NVIDIA: TENSORRT EDGE-LLM RUNS NEMOTRON-3.5 LIGHTNING ON DGX SPARK 64GB, 126.9 TOKENS/SEC WITH DSPARK
October 31, 2026 at 3:00 PM
TechFeed: NVIDIAの数コマンドでエッジAI推論環境が整う「TensorRT Edge-LLM」— 投機的デコーディングで生成速度1.7倍、DGX Sparkからそのままエッジ本番環境へ展開可能
NVIDIAの数コマンドでエッジAI推論環境が整う「TensorRT Edge-LLM」— 投機的デコーディングで生成速度1.7倍、DGX Sparkからそのままエッジ本番環境へ展開可能
TechFeed
techfeed.io
October 31, 2026 at 11:10 PM
October 31, 2026 at 3:00 PM
TechFeed: NVIDIAの数コマンドでエッジAI推論環境が整う「TensorRT Edge-LLM」— 投機的デコーディングで生成速度1.7倍、DGX Sparkからそのままエッジ本番環境へ展開可能
NVIDIAの数コマンドでエッジAI推論環境が整う「TensorRT Edge-LLM」— 投機的デコーディングで生成速度1.7倍、DGX Sparkからそのままエッジ本番環境へ展開可能
TechFeed
techfeed.io
October 31, 2026 at 11:10 PM