#bfloat16
RL training of LLMs silently drops 99.5% of weight updates due to bfloat16 rounding—and a one-line fix closes the training-inference gap with measurable performance gains.

https://arxiv.org/abs/2610.41073

#AI #MachineLearning
November 5, 2026 at 9:00 AM
BFloat16 mixed precision training on TFloat32 tensor cores, followed by quantization-aware training (from BFloat16 to fake Int8) and then post-training quantization to w8a8 with nearest-power-of-two weight snapping is the darkest magic I've ever attempted to conjure. Not sure how or whether it works
November 3, 2026 at 9:55 AM
Bfloat16 Rounding Issues Cause Near-Complete Stagnation in RL Policy Updates

🤖 IA: It's not clickbait ✅
👥 Users: It's not clickbait ✅

#ai #machinelearning #bfloat16

👇👇👇
Bfloat16 Rounding Issues Cause Near-Complete Stagnation in RL Policy Updates
The article discusses a critical issue in reinforcement learning (RL) post-training for language models, where standard bfloat16 (bf16) rounding discards over 99% of parameter updates. Modern frameworks like Verl, TRL, and OpenRLHF maintain master weights in fp32 within PyTorch but serve rollouts using a bf16 copy in inference engines such as vLLM or SGLang. The conversion from fp32 to bf16 uses round-to-nearest, which creates a 'dead zone' where small updates are rounded to zero. At typical RL learning rates (1e-6 to 3e-7), nearly all per-parameter updates fall below the threshold, causing the inference engine to serve a model that is effectively frozen. The Stanford paper by Andrew Gao and William Gao demonstrates that only 0.4-0.5% of weights change per optimizer step, with weights needing 50-200 steps to cross a bf16 grid boundary. This discrepancy leads to a significant mismatch between the trained model and the one used for rollouts, as evidenced by lower scores on checkpoints compared to the actual inference model. The paper proposes 'Fair Cast,' a solution using stochastic rounding at the export barrier to ensure exported weights match master weights in expectation. This method requires adding 16 random bits before truncation, with minimal latency impact. Testing on Qwen3 models showed Fair Cast outperforming round-to-nearest in 18 out of 25 runs, particularly improving accuracy on logic puzzles from 0.33 to 0.57. Frameworks should check if their weight sync uses standard casts like weight.to(torch.bfloat16), which introduces this issue, and consider switching to stochastic rounding to prevent policy update stagnation.
en.killbait.com
November 2, 2026 at 5:49 PM
Google's Jonathan Barron just fixed it:

In arXiv:2610.31642 (Oct 2026), Barron details "Softmax Collapse":

When a model is confident, target probability p ~ 1. In float32/bfloat16, (p - 1) rounds to exact 0.0.
October 26, 2026 at 5:56 AM
破解老黃刀法換來的免費升級,RTX 4000系列也有類似的作法

GPU-unlock:解鎖 NVIDIA 3000 系列顯示卡的 FP/BF16 指令發射速率 https://github.com/Zagitta/gpu-unlock

這個 Linux 核心模組針對 NVIDIA Ampere 架構的 GeForce RTX 3000 系列部分顯示卡,解鎖原先遭限縮的浮點運算指令發射速率,使其接近對應專業卡的水準。它鎖定的是 FP16(16 位元浮點數)/BF16(bfloat16 浮點格式)融合乘加指令 FMA(fused multiply-add)累加至 FP32(32 […]
Original post on mistyreverie.org
mistyreverie.org
October 26, 2026 at 12:12 AM
transformers v5.20.2

Inputs in generate no longer lose their original type inside prepare_inputs_for_generation, which resolves problems with multimodal models; Zero expert gradients in bfloat16 with expert parallelism and ep_size less than fsdp_size; Declare the input modalities of VLMs that…
transformers v5.20.2
Inputs in generate no longer lose their original type inside prepare_inputs_for_generation, which resolves problems with multimodal models; Zero expert gradients in bfloat16 with expert parallelism and ep_size less than fsdp_size; Declare the input modalities of VLMs that report text only; Avoid a…
whatsnew.fyi
October 19, 2026 at 9:02 PM
Gautam Khosla: Calibrating the Checksum: An Empirical False-Positive Bound for One-Sided ABFT on bfloat16 GEMM, and What It Fails to Catch https://arxiv.org/abs/2610.02240 https://arxiv.org/pdf/2610.02240 https://arxiv.org/html/2610.02240
October 5, 2026 at 6:38 AM
So I ended up writing a fix script that's basically a reset: fresh base model, merge, strip the quantization leftovers, save a clean copy.

Version 2 didn't quantize at all: plain bfloat16 on Apple Silicon. Simpler, and it worked.
October 1, 2026 at 9:01 PM
Quantization got my fine-tune running, and then it cost me a day.

Part 3 of training a model on 7,431 of my own tweets, on my laptop.
October 1, 2026 at 9:01 PM
Qwen's Diffusers example loads the model in bfloat16 on CUDA. That is a local route, but it does not establish a VRAM fit for your particular GPU.
September 22, 2026 at 5:08 AM
can't spell bfloat16 without bloat
September 16, 2026 at 10:43 PM
Gemma 4 on an 2021 4 GB Laptop GPU: QAT Takes It From 9.5 GiB to 1.6
This article provides a step by step deployment guide for Gemma 4 E2B's quantization-aware-trained (QAT) checkpoint to a local, laptop hosted GPU enabled system — a much older Lenovo Yoga 9, on sale since January 2021, with a 4 GB GTX 1650 Ti. A suite of Python MCP tools is built to simplify management of the llama.cpp hosted deployment. https://github.com/xbill9/gemma4-dev/tree/main/local-llamacpp-1650ti-2b-q4_0 Gemma 4 E2B in bfloat16 is 9.5 GiB of weights. This laptop's GPU has 4 GiB. Even a plain int8 conversion would not fit. **The QAT release closes that gap: the same model, trained knowing it would be stored at 4 bits, in a 3.35 GB file of which only 1.31 GiB ever has to be on the GPU.** It loaded in 1618 MiB, left more than half the card free, and decodes at 73.75 tok/s. #### What is this project trying to Do? Every other rig in this repository serves Gemma 4 from rented hardware: Cloud TPU, Compute Engine, EC2, Cloud Run. I wanted to know whether the same model would run on the laptop I already had — a 10th-generation Core i7 with a GTX 16-series GPU, no tensor cores and a 4 GB ceiling that no quota request can raise. It does, and QAT is the reason. The rest of this article is how, and what the old hardware changes about running it. #### At This Point You Should Have… * An NVIDIA GPU with a CUDA driver — here a GTX 1650 Ti with Max-Q Design, driver 615.71.09 * The CUDA toolkit — `nvcc` 13.3 here * Python 3.10 or newer for the MCP server — 3.14.7 here, the system `python3`, no virtualenv * The repository cloned, and `local-llamacpp-1650ti-2b-q4_0/` as your working directory * Claude Code, or any MCP client that speaks stdio #### The Laptop | ---|--- Machine | Lenovo Yoga 9 15IMH5 On sale | since January 2021, when Windows Central reviewed it CPU | Intel Core i7-10750H, 10th generation (Comet Lake), 12 threads RAM | 15 GiB, as `free` reports it GPU | GeForce GTX 1650 Ti with Max-Q Design, 4096 MiB GPU power | 40 W limit The MCP server reports the GPU the same way: gpu_status 📡 **GPU** — `local-llamacpp-1650ti-2b-q4_0` NVIDIA GeForce GTX 1650 Ti with Max-Q Design, 7.5, 4096 MiB, 1606 MiB, 2127 MiB, 615.71.09 ⚠️ GTX 16-series (TU116/TU117): compute capability 7.5 but **no tensor cores**. Do not compare throughput against the T4-based `g4dn`/`g5g` rigs on the strength of a matching compute capability. **Compute capability 7.5 is a trap.** A T4 is also 7.5 and has tensor cores; the GTX 16-series is a cut-down Turing die with the tensor cores removed. llama.cpp notices on its own at init: The following devices will have suboptimal performance due to a lack of tensor cores: Device 0: NVIDIA GeForce GTX 1650 Ti with Max-Q Design #### Why Can't the Normal Model Fit? The sizes, from this repository's model reference: Gemma 4 E2B as | Weights | Fits 4096 MiB? ---|---|--- bfloat16 | 9.5 GiB | ❌ no int8 | ~4.8 GiB | ❌ no QAT Q4_0 GGUF, whole file | 3.35 GB | barely, on paper QAT Q4_0 GGUF, GPU-resident part | **1.31 GiB** | ✅ yes The bf16 checkpoint is more than twice the card. Halving it to int8 is still larger than the card. Only a 4-bit model is in range, and that is where QAT comes in. #### What Is QAT, and Why Does It Matter Here? Quantization-aware training simulates quantization _during_ training rather than compressing a finished model afterwards. The 4-bit weights are not a post-hoc approximation of a bf16 model: the model was trained knowing it would be stored this way. Google's model card puts the goal plainly: > This model card is for the new versions of the Gemma 4 family optimized with Quantization-Aware Training (QAT), which allows preserving similar quality to bfloat16 while dramatically reducing the memory requirements to load the model. That is the difference that matters on a 4 GB card. **Rounding the bf16 model down to 4 bits after the fact would fit too, but nothing in its training would have prepared it for that.** QAT is the version of "fits" that was designed to keep the model's quality close to bf16 on the way there. Google ships the QAT weights four ways: Artifact | What it is | For ---|---|--- `-qat-q4_0-unquantized` | QAT values in a half-precision container | custom compilation `-qat-q4_0-gguf` | the same values, packed Q4_0 | **llama.cpp — this rig** `-qat-w4a16-ct` | compressed-tensors | vLLM mobile | on-device runtimes | phones The GGUF is not a lesser copy. This repository checked it: four norm tensors read out of the GGUF are bit-identical to the ones in the `-unquantized` release, so it is the same QAT model in its native packing. #### Which QAT Checkpoint? `google/gemma-4-E2B-it-qat-q4_0-gguf`, which llama.cpp opens directly. The agent reads it off disk: model_info 📡 **Model** — `local-llamacpp-1650ti-2b-q4_0` - **Name:** `google/gemma-4-E2B-it-qat-q4_0-gguf` - **Path:** `/home/xbill/models/gemma-4-E2B-it-qat-q4_0/gemma-4-E2B_q4_0-it.gguf` - **On disk:** 3.35 GB - **Quantization slot:** `q4_0` — but the dominant tensor type is **Q6_K**. Both embedding tensors are Q6_K (2.257 GB of 3.334 GB); only the ~1.08 GB transformer body is actually Q4_0. - **Resident on GPU:** ~1.31 GiB. `per_layer_token_embd` (1.93 GB, 58% of the file) is `TENSOR_READ_LAZY` and is served by GET_ROWS out of the mmap. Run `inspect_gguf.py` to re-derive the split from the artifact rather than trusting these numbers. Two lines in that output explain why a 3.35 GB file fits in far less than 3.35 GB. #### Why Only 1.31 GiB Has to Be on the GPU A 3.35 GB file against 3724 MiB free reads as "barely fits." It is better than that. `make info` reads the tensor table out of the GGUF: make info gemma-4-E2B_q4_0-it.gguf tensors: 541 total: 3.334 GB largest tensors: 1926.8 MB per_layer_token_embd.weight Q6_K [8960, 262144] <- LAZY, host-resident 330.3 MB token_embd.weight Q6_K [1536, 262144] 27.5 MB per_layer_model_proj.weight F16 [1536, 8960] 10.6 MB blk.34.ffn_up.weight Q4_0 [1536, 12288] lazy (never on GPU): 1.927 GB (58% of file) must be resident: 1.407 GB = 1.31 GiB by tensor type (the slot-5 token is q4_0; the file mostly is not): Q6_K 2.257 GB (67.7%) Q4_0 1.048 GB (31.4%) F16 0.028 GB ( 0.8%) **One tensor is 58% of the file, and none of it has to be on the GPU.** `per_layer_token_embd` is Gemma 4's per-layer embedding table — the "E" in E2B, and why the model has about 5B parameters in total but 2B effective. llama.cpp creates it with `TENSOR_READ_LAZY` and uses it as a row lookup, not a matmul, so the rows a token needs are read out of the memory-mapped file on the host. QAT and the lazy table stack. QAT puts the 35 transformer blocks at Q4_0, 1.05 GB where half precision would be 3.7 GB; the lazy table keeps the biggest remaining tensor off the card entirely. **What decode actually streams per token drops from 4.514 GB at half precision to 1.407 GB, a 3.2x cut.** Two rules follow, and both are the opposite of what a memory-anxious setup would do: * **Do not lower`-ngl` to save memory.** Full offload leaves over 2 GiB free. Lowering it moves real matmul weights to the CPU and saves nothing that was scarce. * **Never pass`--no-mmap`.** Lazy reading requires mmap. Turn it off and the 1.93 GB table has to be materialised, and a comfortable fit becomes a failure to load. #### Step 1 — Build llama.cpp for sm_75 A standard CUDA build, pinned to this card's architecture: grep -E "^(GGML_CUDA|CMAKE_CUDA_ARCHITECTURES|CMAKE_BUILD_TYPE)[:=]" ~/llama.cpp/build/CMakeCache.txt ~/llama.cpp/build/bin/llama-server --version CMAKE_BUILD_TYPE:STRING=Release CMAKE_CUDA_ARCHITECTURES:UNINITIALIZED=75 GGML_CUDA:BOOL=ON version: 0.3.0-dev (build 1, commit 95ef7fc) #### Step 2 — Configure `tpu.env` One committed env file drives both the `Makefile` and the MCP server. It is called `tpu.env` for consistency with its siblings; there is no TPU here. grep -E "^(MODEL_PATH|ENDPOINT|N_GPU_LAYERS|CONTEXT_SIZE|KV_CACHE_TYPE|FLASH_ATTENTION|THREADS|PARALLEL_SLOTS|METRICS)=" tpu.env MODEL_PATH=/home/xbill/models/gemma-4-E2B-it-qat-q4_0/gemma-4-E2B_q4_0-it.gguf ENDPOINT=http://127.0.0.1:8080 N_GPU_LAYERS=99 CONTEXT_SIZE=8192 KV_CACHE_TYPE=f16 FLASH_ATTENTION=1 METRICS=1 THREADS=4 PARALLEL_SLOTS=1 `N_GPU_LAYERS=99` is full offload. Every value was measured on this card rather than copied from a cloud rig. #### Step 3 — Serve It and Measure the Memory make serve `make serve` runs in the foreground on purpose: this is a laptop, nothing is billed, and Ctrl-C is a complete teardown. From another shell: nvidia-smi --query-compute-apps=pid,process_name,used_memory --format=csv,noheader 37073, /home/xbill/llama.cpp/build/bin/llama-server, 1618 MiB **1618 MiB of 4096.** Weights, an 8192-token context and CUDA's own overhead, with more than half the card to spare. Had the lazy table been resident the requirement would have been about 3.5 GB, and the model would have failed to load. The agent confirms it is serving: model_server_status ✅ Serving at http://127.0.0.1:8080 (pid 83619). `/health` → 200. The endpoint is a fixed local address, so there is nothing to discover. An early version of this tool read a missing pid file as "not running" and reported ❌ against a healthy server; it now checks which process owns port 8080, and `/health` makes the final call. #### Where the 1618 MiB Goes The allocation, as printed by the engine when a sibling rig loaded the same file on this same card through Ollama: term | MiB ---|--- model buffer (weights) | 1341.78 KV, 3 full-attention layers × 8192 cells | 48.00 KV, 12 sliding-window layers × 1024 cells | 12.00 compute buffer | 122.52 CUDA context + slack | ~94 The weights are the QAT body plus the resident embeddings. The KV cache is only 60 MiB, because llama.cpp caps Gemma 4's sliding-window layers at the 1024-token window regardless of context size. My own derivation before the run predicted 1616 MiB — two MiB off — with 144 MiB of KV and ~130 of overhead. **Both of those terms were wrong, in opposite directions, and the total still matched.** A total that agrees with the hardware does not confirm the terms; read the engine's allocation log. #### Step 4 — Ask the Model query_model "In one sentence, what is a TPU?" ✅ **Reply** A TPU is a specialized hardware accelerator designed by Google specifically to speed up the computationally intensive operations required for training and running machine learning models and deep learning tasks. --- _(plus 1167 chars of reasoning, suppressed)_ prompt 25 tok · completion 308 tok · 70.5 tok/s A 4-bit Gemma 4 answering on a laptop GPU with no tensor cores, through Claude Code, over MCP. **A one-sentence answer cost 308 completion tokens** , and most of them were thinking. #### The Second Way to Get an Empty Reply Gemma 4 emits a thinking block. llama.cpp routes it to `reasoning_content` and leaves `content` empty until the block closes, so a caller that reads only `content` with a modest budget gets `""` and concludes the server is broken. `query_model` defaults to `max_tokens=1024`, and reports the empty case as what it is: query_model "Name three TPU generations." max_tokens=64 📡 **Reasoning only — no answer yet.** `finish_reason: length` after 64 tokens, all of them thinking. This is Gemma 4 reasoning, not a broken server. Re-run with a larger `max_tokens` (currently 64). The same applies to benchmarks: **a 128-token generation limit on this model measures the thinking phase and nothing else.** #### Tuning: What Moved Decode? `llama-bench -p 512 -n 128 -r 3`, all layers on the GPU: | config | prefill t/s | decode t/s ---|---|---|--- 🥇 | `-fa 1`, f16 KV, `-t 4` | 340.33 | 73.75 🥈 | `-fa 1`, f16 KV, `-t 6` | 340.39 | 73.74 🥉 | `-fa 1`, f16 KV, `-t 8` | 338.49 | 72.97 | `-fa 0`, f16 KV, `-t 6` | 338.74 | 70.39 | `-fa 1`, f16 K, q8_0 V | 202.66 | 65.55 | `-fa 1`, q8_0 K, f16 V | 239.09 | 64.91 * **Flash attention is a free +4.8% on decode** (70.39 → 73.74). Adopted. * **Thread count does not matter.** Decode is GPU-bound. * **Quantizing the KV cache is a loss.** `q8_0` costs 11–12% of decode and 30–40% of prefill. With no tensor cores to hide the dequantization, and a KV cache of 60 MiB, it buys memory that was never scarce. **QAT already did the quantizing that matters.** * **`GGML_CUDA_FORCE_MMQ` does not pay**, even though llama.cpp suggests it for this card: 337.61 / 73.04 against the stock build's 340.33 / 73.75. #### Does Concurrency Help? For decode, yes: `llama-batched-bench` goes from 73.33 to 277.61 tok/s between one and 64 parallel sequences. But that is the decode phase in isolation. Through the HTTP endpoint with 512-token prompts, prefill runs one request at a time — time to first token doubles exactly with every doubling of clients — and aggregate output tops out near 48 tok/s, 1.47x from one client. With short prompts and long answers it reaches 2.6x at 16 clients, but every level between 1 and 16 flips between two speeds from trial to trial. **For one person on one laptop, concurrency one is the right setting** , and that is what `PARALLEL_SLOTS=1` carries. #### Managing It with MCP `server.py` is a single-file MCP server with seven tools: `gpu_status`, `model_info`, `start_model_server`, `stop_model_server`, `model_server_status`, `query_model` and `get_help`. There is no provisioning tool, because there is nothing to provision. It registers under its directory name, so every tool reaches Claude Code as `mcp__local-llamacpp-1650ti-2b-q4_0__<tool>`. A stdio check confirms the handshake independently of any client: { printf '%s\n' \ '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-06-18","capabilities":{},"clientInfo":{"name":"probe","version":"0"}}}' \ '{"jsonrpc":"2.0","method":"notifications/initialized"}' \ '{"jsonrpc":"2.0","id":2,"method":"tools/list","params":{}}'; sleep 4; } \ | python3 server.py 2>/dev/null initialize OK: name='local-llamacpp-1650ti-2b-q4_0' version='' proto 2025-06-18 tools/list OK: 7 tools -> get_help, gpu_status, model_info, model_server_status, query_model, start_model_server, stop_model_server #### One Thing Broke: MCP SDK 2.x Midway through, the server stopped loading. Another project on the machine needed the MCP Python SDK 2.x, all of these projects share one system Python, and 2.x renamed the class this server imports: python3 -c "from mcp.server.fastmcp import FastMCP" raise ModuleNotFoundError(_MESSAGE, name=__name__) ModuleNotFoundError: No module named 'mcp.server.fastmcp'. This is mcp 2.x, where FastMCP was renamed to MCPServer (from mcp.server.mcpserver import MCPServer) and other APIs changed; see the migration guide at https://py.sdk.modelcontextprotocol.io/v2/migration/#fastmcp-renamed-to-mcpserver or pin 'mcp<2' to keep running v1 code. Pinning back was not an option with a shared interpreter. The fix was two lines, the requirement bound, and the test mock's module path: -from mcp.server.fastmcp import FastMCP +from mcp.server.mcpserver import MCPServer -mcp = FastMCP(MCP_SERVER_NAME) +mcp = MCPServer(MCP_SERVER_NAME) -mcp>=1.2.0,<2 +mcp>=2 -sys.modules["mcp.server.fastmcp"] = _fastmcp_module +sys.modules["mcp.server.mcpserver"] = _mcpserver_module Touching `start_model_server` for the rename surfaced an older bug: the tool launched `llama-server` without `-fa`, `-t` or `--parallel`, so an MCP-started server came up with **4 slots and 6 threads** instead of the measured configuration — and llama.cpp splits the context across slots. The argv now lives in one function, a test holds it to the `Makefile`, and the running server's real command line matches: tr '\0' ' ' < /proc/$(pidof llama-server)/cmdline /home/xbill/llama.cpp/build/bin/llama-server -m /home/xbill/models/gemma-4-E2B-it-qat-q4_0/gemma-4-E2B_q4_0-it.gguf --host 127.0.0.1 --port 8080 -ngl 99 -c 8192 -ctk f16 -ctv f16 -fa 1 -t 4 --parallel 1 --metrics make lint make test All checks passed! lint OK ---------------------------------------------------------------------- Ran 30 tests in 0.035s OK #### And Price/Performance? There is no price. The laptop was already on the desk: nothing is billed, nothing is reserved, and there is no idle cost to optimise. What QAT bought is capability, not a discount. Without it this GPU cannot hold Gemma 4 E2B at all. With it, the model takes 1618 MiB and a single conversation decodes at 73.75 tok/s. #### Teardown # make serve is in the foreground: Ctrl-C Or ask the agent for `stop_model_server`, which sends SIGTERM to whichever process owns port 8080, including one started by `make serve`. Not run for this article; the server is still up. #### What This Does Not Cover **Output quality was not measured here.** The claim that QAT holds quality close to bfloat16 is Google's, from the model card; this article measured memory and speed, not accuracy, and never compared the QAT model's answers against a bf16 run. **No benchmark completed a task.** Every sweep generation hit its token cap inside Gemma 4's thinking block, so the throughput figures are real and the tasks are not. **Nothing here transfers to a T4.** Same compute capability, different silicon. #### Summary The goal of this article was to run Gemma 4 on a much older laptop whose GPU has 4 GB of memory. The key to the solution was Google's quantization-aware-trained checkpoint, packed as a GGUF, plus llama.cpp leaving its largest tensor in host memory. The measured results were: * **bf16 needs 9.5 GiB and int8 ~4.8 GiB; the QAT GGUF needs 1.31 GiB on the GPU.** Only the 4-bit QAT model fits this card, and it fits with room to spare. * **Full offload in 1618 MiB of 4096** , with an 8192-token context and a KV cache of just 60 MiB. * **73.75 tok/s single-stream decode** on a GPU with no tensor cores; flash attention is worth 4.8%. * **Quantizing further is a loss:** a `q8_0` KV cache costs 11–12% of decode, because QAT already took the memory pressure away. * **An MCP server manages the whole lifecycle** , and moving it to SDK 2.x took a two-line change that also exposed an MCP-started server running the wrong configuration. Scope: one Lenovo Yoga 9 15IMH5 (Core i7-10750H, 15 GiB RAM, GTX 1650 Ti Max-Q with 4096 MiB and a 40 W cap), llama.cpp `95ef7fc` built for sm_75 with CUDA 13.3, `google/gemma-4-E2B-it-qat-q4_0-gguf`, and the MCP server on mcp 2.2.0 under Python 3.14.7. `llama-bench` rows are three repeats; serving sweeps are three repeats per level with the prompt cache defeated. The memory split comes from a sibling rig running the same engine through Ollama on the same card. Model quality was not measured. The strategy for using MCP for a local GPU deployment was validated with an incremental step by step approach. #### References * local-llamacpp-1650ti-2b-q4_0 | GitHub * google/gemma-4-E2B-it-qat-q4_0-gguf | Hugging Face * Lenovo Yoga 9i 15 review, January 2021 | Windows Central * Quantization-Aware Training for Gemma 4 | Google * llama.cpp | GitHub * Migration Guide: v1 to v2 | MCP Python SDK
dev.to
September 10, 2026 at 7:28 PM
Pure JAX on G5g: Serving Gemma 4 on Graviton and a T4G
This article provides a step by step deployment guide for serving Google's Gemma 4 on an AWS EC2 G5g instance using pure JAX. The code is here: github.com/xbill9/gemma4-dev #### What is this project trying to Do? This project aims to serve a modern open model on the cheapest whole CUDA GPU AWS will rent you, and to measure honestly what that costs. #### Aren't You Using The Wrong GPU? Probably! The T4G is a Turing chip from 2018. It has no bfloat16 and no fp8. But it is cheap, it is available when nothing else is, and it is attached to a Graviton2 host — which makes G5g the rare hardware axis that almost nothing in the ML ecosystem targets: **aarch64 and CUDA together**. So let's give pure JAX a shot on G5g! #### AWS EC2 G5g G5g instances pair an AWS Graviton2 (64-bit Arm) processor with NVIDIA T4G Tensor Core GPUs. At `g5g.xlarge` they are the **cheapest EC2 instance carrying a whole NVIDIA GPU** , and the only Arm-based GPU family AWS offers. Two GPU instances are cheaper per hour and neither can serve this model (us-east-1, Linux, on-demand, checked against the Pricing API on 2026-08-28): * `g6f.large` at **$0.2020** is genuinely NVIDIA and genuinely CUDA — but it is **one eighth of a GPU with 3 GB** , and the weights alone are 6.155 GB. The first `g6f` that fits is `g6f.4xlarge` at $0.9500, which is 1.7x this rig's `g5g.2xlarge`. * `g4ad.xlarge` at **$0.3785** carries an AMD Radeon Pro V520 — no CUDA at any price. Among whole NVIDIA GPUs, G5g is the floor: `g5g.xlarge` at $0.4200, and the next one up is `g4dn.xlarge` at $0.5260. More information is available here: https://aws.amazon.com/ec2/instance-types/g5g/ The default in this rig is `g5g.2xlarge` — 1 GPU, 8 vCPU, 16 GiB RAM. Note- the T4G reports **15,360 MiB** of device memory, not the nominal 16 GB. Budget against the measured number. #### Gemma 4 Gemma is Google's family of open models built from the same research as Gemini. This rig serves `google/gemma-4-E2B-it`, the instruction-tuned reference release. #### JAX JAX is Google's array computing library — NumPy semantics, composable transformations, and compilation to XLA. On NVIDIA hardware, `pip` supplies the CUDA libraries, so there is nothing to build. More information is available here: https://github.com/jax-ml/jax "Pure JAX" here is literal. The engine is this repo's own Gemma 4 port driven by a JAX generation loop behind an OpenAI-compatible FastAPI server, running under systemd. #### Prerequisites You need four things before starting: * An AWS account with G instance vCPU quota in `us-east-1` * A **subnet id** , a **security group id** , and an **IAM instance profile** — the rig requires all three explicitly and will not create them for you * A Hugging Face token with access to the Gemma weights * Python 3.13 and `pip` The instance profile needs `AmazonSSMManagedInstanceCore` plus read access to your Secrets Manager secret and your S3 cache bucket. There is **no inbound SSH rule and no private key** — all remote administration goes over SSM Run Command. #### Install the Rig Clone the monorepo and install the control plane: git clone https://github.com/xbill9/gemma4-dev cd gemma4-dev/gpu-jax-g5g-2b pip install -r requirements.txt That installs `boto3` and FastMCP only. Nothing here needs a GPU — the GPU is on the other end. #### Run the Tests python3 -m unittest discover -s tests -v 105 tests, fully offline. Every cloud, subprocess, and network boundary is mocked. If these do not pass, do not launch an instance. #### Register the MCP Server The whole rig is driven by an MCP server exposing a devops agent: ./project-setup.sh This installs the bundled skill and registers `.mcp.json`: { "mcpServers": { "gpu-jax-g5g-2b": { "command": "python3", "args": [".claude/skills/gpu-jax-g5g-2b-management/mcp/server.py"], "env": { "AWS_REGION": "us-east-1", "MODEL_NAME": "google/gemma-4-E2B-it", "INSTANCE_TYPE": "g5g.2xlarge", "MCP_SERVER_NAME": "gpu-jax-g5g-2b" } } } } Every tool is now available as `mcp__gpu-jax-g5g-2b__<tool>`. #### Save the Hugging Face Token save_hf_token(token="hf_...") This writes to AWS Secrets Manager under `vllm/hf-token`. The instance fetches it at boot into a root-only `EnvironmentFile`. Note- the token never goes in user data. Instance metadata is readable by anything running on the box. #### Check the Quotas check_g5g_quotas() Reports your On-Demand and Spot G instance vCPU limits for the region. `g5g.2xlarge` is 8 vCPU. Check this before launching, not after the launch fails. #### Launch the Instance create_g5g_instance( subnet_id="subnet-...", security_group_id="sg-...", iam_instance_profile="...", spot=True ) The AMI is resolved at launch time from SSM Parameter Store: /aws/service/deeplearning/ami/arm64/base-oss-nvidia-driver-gpu-ubuntu-26.04/latest/ami-id **Never hardcode an AMI id here.** AWS also ships ARM64 DLAMIs built for Graviton _CPU_ inference. They boot perfectly and simply have no GPU. The `/latest/` parameter also moves — this rig has seen `ami-0bff4343bfd56a20e` become `ami-025a6e5b3b786cf61` overnight as Ubuntu 26.04 became 26.04.1. #### Watch the Install get_install_progress(instance_id="i-...") Cloud-init installs `jax[cuda13]` on Python 3.14. The stages are timed: [stage] jax-wheels 43s (total 84s) [stage] serving-deps 14s (total 98s) [stage] gpu-verify 13s (total 111s) [stage] cache-restore 6s (total 117s) [stage] unit-rewrite 0s (total 117s) **117 seconds, and there is no compile step anywhere in it.** That is the entire reason this rig exists — more on that below. The `cache-restore` stage pulled **805 files / 12 MB in 6 seconds** from S3, on a _fresh_ instance, compiled by a box that had already been terminated. XLA's cold-compile penalty becomes a rounding error on Spot. #### Verify the GPU verify_gpu_arch(instance_id="i-...") This measures whether JAX's CUDA kernels actually cover this GPU, rather than trusting that they do. You want to see SM 7.5 claimed and a real device, not a silent CPU fallback. #### Deploy the Server make skill deploy_jax_server(instance_id="i-...") **Always`make skill` first.** The deploy ships the _skill snapshot_ , not your working tree. The deploy output prints the payload root and the build id so a stale deploy is visible in one line. #### Verify The Installation The first line the process emits is the device-policy banner: INFO ports.gemma4.jax_e_model: jax_e_model device policy: platform=gpu compute_capability=7.5 compute_dtype=float16 pallas_interpret=False Both halves matter. `float16` is **the device choosing** Turing's only real 16-bit datapath — it is read from the live compute capability, not from a config file. And `pallas_interpret=False` is the difference between serving and silently running a simulator. Then the whole resolved configuration lands on one greppable line: READY build_id=6852f5680f43 ... compute_dtype=float16 kv_cache_dtype=float16 kv_cache_requested=auto pre_ampere=True quant_mode=fp16 window_kv=True Load is staged, so a hang is attributable: download 87.7s read_shards 73.5s (1 shard, 600 tensors, 0.95 GB of non-text towers skipped) convert_params 3.4s device_put 0.0s 164.7s total, 9.26 GB Then confirm the served build matches what you shipped: verify_model_health(instance_id="i-...") #### Query the Model query_model(instance_id="i-...", prompt="Explain Graviton in one sentence.") The endpoint is OpenAI-compatible on `:8000`, so anything that speaks that API works: get_endpoint(instance_id="i-...") Note- **warm up at the shape you measure.** `max_new_tokens` is a `static_argnames` entry, so `(bucket, max_tokens)` is the compiled shape. The same request measured **18.77 s cold against 4.35 s warm**. #### Check the Metrics get_metrics(instance_id="i-...") tpu_jax_decode_tokens_per_second 13.0 tpu_jax_prefill_milliseconds 692.6 tpu_jax_hbm_used_bytes 6296892160 tpu_jax_weight_bytes 6155450950 tpu_jax_degenerate_responses_total 0 **Quote the gauge, not end-to-end.** Decode is flat at 12.9 / 13.0 / 12.9 tok/s across 41 → 2,057 input tokens. End-to-end throughput _does_ fall (12.43 → 8.22) — but that is prefill being linear in the padded bucket, not decode degrading. Two different claims. #### Why Not vLLM? Because I tried, on identical silicon, and it works — at 43 tok/s — but only after this: 1. A ~67-minute from-source build 2. `cuda-toolkit` from NVIDIA's `sbsa` repo, because the DLAMI ships a driver but no `nvcc` 3. A Rust toolchain, for vLLM's frontend 4. **An unlanded patch to Triton's attention kernel** , reapplied after every upgrade That last one is the interesting failure. Gemma 4 has heterogeneous attention head dimensions — sliding layers at 256, global layers at **512**. Only two vLLM backends support that, and with FA4 unavailable it force-selects Triton: Gemma4 model has heterogeneous head dimensions {'sliding_attention': 256, 'full_attention': 512}. FA4 not available, forcing TRITON_ATTN backend. Whose 512-wide tile then asks Turing for memory Turing does not have: triton.runtime.errors.OutOfResources: out of resource: shared memory, Required: 98304, Hardware limit: 65536 JAX sidesteps all four. `pip` supplies CUDA, so no build, no toolkit, no Rust. The plugin's precompiled cubins already cover `sm_75`. And attention is ordinary XLA rather than a hand-tiled Triton kernel, so there is no per-block shared-memory ceiling and no patch to carry. **The honest trade: 13.10 tok/s against 43, for a 117-second install with nothing to reapply.** For a measurement rig I re-provision constantly on Spot, that was right. For a production endpoint it probably is not. #### The Number I Did Not Expect I profiled decode with xprof. The kernel table is the whole story: conversion 54.0% <-- dtype conversion fp32 gemv 32.9% fusion 12.2% TensorCore 0.0% **Zero.** 1,466 ms of kernels across 108 distinct kernels on a Tensor Core GPU, and not one Tensor Core fired. Over half of decode went to converting numbers between formats before the math could start. The obvious hypothesis was bf16 weights being converted on a chip with no bf16. So this weekend I converted the checkpoint to float16 host-side and re-ran. Parameter dtypes now read `{'float16': 541, 'uint8': 1, 'int8': 1}` — and **conversion is still 54.0%**. The obvious explanation is wrong and I do not yet know the real one. That is the next thing to profile. I would rather publish the open question than a tidy story. What I can stand behind is that the measurement is real: the same profile on a **different instance, a different AMI, and a restored cache** landed at 1466.0 ms against 1467.1 ms. **1.1 ms apart on 1467.** #### AWS Services Used Service | Role ---|--- **EC2** (`g5g.2xlarge`, Spot) | Graviton2 + NVIDIA T4G **Systems Manager — Parameter Store** | Resolves the arm64 GPU DLAMI id at launch **Systems Manager — Run Command** | Ships the payload, runs every diagnostic. No SSH **S3** | XLA compilation cache, shared across instances **Secrets Manager** | Hugging Face token **IAM** | Instance profile scoping all of the above **Service Quotas** | Pre-flights G vCPU limits before a doomed launch **EBS gp3** | 100 GB at 500 MiB/s, 6,000 IOPS, for a 9.5 GB checkpoint Everything goes through `boto3`. The rig never shells out to the AWS CLI. #### Clean Up terminate_g5g_instance(instance_id="i-...") Termination is cheap on this rig — there is no built image to lose with the root volume, only a pip install and a model cache, and the compilation cache is already in S3. #### What I Learned This Summer **Check reachability on paper before spending a provisioning cycle.** A nine-step analysis order — does the compute dtype match the chip, is there a fused kernel for that format on that chip — runs in an afternoon. A launch costs a day. It has killed two bad plans before either touched hardware. **A wrong dtype does not error. It emulates.** bfloat16 on Turing does not fail loudly; it routes through fp32 and quietly eats your decode. **Refuse early, with the arithmetic attached.** The fused W4A16 kernel wants 550 KiB–1.1 MiB per block and Turing gives you 64 KiB. The rig computes that at startup and refuses with the numbers in the message, rather than dying as a cryptic `OutOfResources` at the first token. **The scariest bugs return`status: "success"`.** A padding-eviction bug in the KV ring cache produced a token loop, not a crash. Nothing in the logs was red. It took a week. #### Summary The AWS G5g instance provides a genuinely cheap environment for serving open models, and pure JAX reaches a served token on it without a single line of compiled code. The throughput is not competitive with a patched vLLM - but the deployment is 117 seconds, reproducible to 1.1 ms, and has nothing to reapply.
dev.to
August 29, 2026 at 1:10 AM
I’m using the TinyStories dataset! I didn’t note the size on disk but it’s 7.8 params at bfloat16 I guess about 15 megabytes.

Don’t tell anybody but my next project is to train a language model that fits on a floppy disk (quantized). 😁
August 26, 2026 at 2:49 PM
Podpora numerických hodnot typu quadruple-precision floating-point v GCC
Podpora numerických hodnot typu quadruple-precision floating-point v GCC
Už jsme si představili formáty nazývané half-float a bfloat16. Dnes si představíme formát quadruple-precision (čtyřnásobná přesnost), který numerické hodnoty ukládá do plných šestnácti bajtů.
www.root.cz
March 2, 2026 at 11:10 PM
NVIDIA AI Open-Sourced KVzap: A SOTA KV Cache Pruning Method that Delivers near-Lossless 2x-4x Compression

As context lengths move into tens and hundreds of thousands of tokens, the key value cache in transformer decoders becomes a primary deployment bottleneck. The cache stores keys and values…
NVIDIA AI Open-Sourced KVzap: A SOTA KV Cache Pruning Method that Delivers near-Lossless 2x-4x Compression
As context lengths move into tens and hundreds of thousands of tokens, the key value cache in transformer decoders becomes a primary deployment bottleneck. The cache stores keys and values for every layer and head with shape (2, L, H, T, D). For a vanilla transformer such as Llama1-65B, the cache reaches about 335 GB at 128k tokens in bfloat16, which directly limits batch size and increases time to first token.
nexttech-news.com
January 15, 2026 at 11:04 PM
2504.11651
大規模言語モデル(LLM)のサイズは急速に拡大しており、リソースに制約のあるハードウェア上で効率的に展開するには大きな課題がある。本論文では、Dynamic-Length Float (DFloat11)を紹介する。DFloatは、元のモデルとビット単位で同一の出力を保持しながら、LLMサイズを30%削減する可逆圧縮フレームワーク...
April 26, 2025 at 12:06 AM
[3/30] 683 Likes, 193 Comments, 3 Posts
2504.11651, cs․LG | cs․DC, 15 Apr 2025

🆕70% Size, 100% Accuracy: Lossless LLM Compression for Efficient GPU Inference via Dynamic-Length Float

Tianyi Zhang, Yang Sui, Shaochen Zhong, Vipin Chaudhary, Xia Hu, Anshumali Shrivastava
April 26, 2025 at 12:06 AM
Very cool! But this seems more like "what if we did prolog, but as a neural network?" Inputs are highly structured (maze, sudoku), not random internet text in Portuguese. SWI-Prolog binary is ~14M, 27M bfloat16 would be ~3-4x that. *Reasoning* isn't that hard, it's figuring out what to reason.
July 27, 2025 at 3:43 PM
It turns out your quantisation centroids should distribute according to the cube root of the data pdf (a result from the '50s). Surprising, if you ask me. To find out more, read our post graphcore-research.github.io/posts/cube-r...
Optimal Formats and the Cube Root of the PDF
Your boss emails you a point in 128-billion-dimensional space. “Llama 3.1 8B,” the message reads. “A not-so-large language model in bfloat16. But it’s too big. Trim the fat (ASAP).” You open up your t...
graphcore-research.github.io
June 12, 2025 at 11:20 AM
Your boss emails you a point in 128-billion-dimensional space. It's Llama 8B in bfloat16. They want it compressed.

What should you do 🤔... quantise to NF4? 🧵
June 12, 2025 at 11:19 AM