LightNews — News, ranked by scholars

mr. TIM

@timkellogg.me

astonishing: using fp16 instead of bf16 results in more stable training runs as well as a smaller performance gap between training & inference

this is critical for RL, which is mostly inference and very sensitive to reproducible results

A 3×4 grid of line charts comparing BF16 (blue) and FP16 (green) across various training setups.
Each subplot shows Training Steps (x-axis) vs. a performance metric (y-axis, 0.0–1.0).

Top row:
(a) Sanity GRPO – FP16 consistently outperforms BF16; smooth upward trend to ~0.95.
(b) Sanity GRPO-Token-TIS – similar behavior; FP16 steadier and higher.
(c) Sanity GRPO-Seq-MIS – FP16 reaches ~0.95; BF16 lags below 0.85.
(d) Sanity GSPO – FP16 again higher and smoother; BF16 converges slower.

Middle row:
(e) Sanity PG-Seq-IS and (f) Sanity PG-Seq-MIS – both show FP16 > BF16; FP16 reaches near 1.0.
(g) OctoThinker GRPO – FP16 stable near 1.0; BF16 spikes early, then collapses.
(h) Lora GRPO-Token-TIS – FP16 stable around 0.8; BF16 fluctuates sharply and drops after ~800 steps.

Bottom row:
(i) MoE GRPO-Seq-MIS, (j) MoE GRPO-Token-TIS, and (k) MoE PG-Seq-TIS – both precisions close; FP16 slightly faster early on.
(l) Dense-14B DAPO – both curves rise smoothly; FP16 maintains a small advantage.

In nearly all cases, FP16 (green) trains more stably and converges to higher performance than BF16 (blue).

October 31, 2025 at 6:53 PM

モンモン(AIイラスト)

@almondfish.bsky.social

Krea2 BF16で割と足りる感じね
#セリーナちゃん

September 17, 2026 at 12:10 AM

#BruceSterling

@bruces.bsky.social

"The unit was equipped with an Ampere-generation NVIDIA accelerator, lacking native BF16 execution. It is believed that the reasoning process was derailed by CUDA's sloppy rescaling of BF16 tensors (used in training) during inference."
#AI #bloodshed #death #destruction

January 17, 2026 at 5:14 PM

Astra ⭐

@astrra.space

LLM runtime that takes in a BF16 model file and an imatrix and quantizes it to fit exactly into your available VRAM

August 13, 2026 at 4:09 PM

Alexander Doria

@dorialexander.bsky.social

bf16 halloween might be already ending. according to a bytedance engineer could just have been another flash-attention bug.

November 2, 2025 at 1:30 PM

xjdr

@xjdr.bsky.social

now that people are paying attention again, here is your periodic reminder. Always run in bf16. always apply ROPE and attention softmax at float32 (as shown here)

github.com/xjdr-alt/ent...

November 24, 2024 at 5:23 PM

Tom Aarsen

@tomaarsen.com

I'm very excited to share that NVIDIA just released Nemotron-3-Embed: two multilingual embedding models for retrieval

The 8B takes the top spot on RTEB.

- Nemotron-3-Embed-1B-BF16 (2048-dim)
- Nemotron-3-Embed-8B-BF16 (4096-dim)

Both OpenMDW-1.1, ready for commercial use.

🧵

July 16, 2026 at 4:06 PM

nina 🪷

@ninachaubal.com

as of last night, I'm able to compress BF16 weights by 30%

chart showing various models compressed 29.17-31.44% for BF16 and 10.62-15.30% for FP8

September 16, 2026 at 5:56 PM

村

@abfbfygo.bsky.social

BF16周年
クリムゾンクライシス16周年おめでとう！！！！！

November 14, 2024 at 10:14 PM

SE Gyges

@segyges.bsky.social

bf16 is a better nonlinearity

May 15, 2026 at 3:14 AM

rev. howard arson

@theophite.bsky.social

never mind i just changed the default epsilon from 1e-8 to 1e-5 because i think that bf16 training is causing pink and green color artifacts

rev. howard arson @theophite.bsky.social · May 15

there is nothing which makes me feel like a more unprincipled mathematical pervert than resorting to changing an optimizer's betas

June 27, 2025 at 5:02 AM

rev. howard arson

@theophite.bsky.social

"doesn't that mean that you've quanted everything to shit?" no. :)

bf16, more or less.

August 15, 2026 at 3:35 PM

SHU@VAPE

@shus1214.bsky.social

BF16+Dvarw16

February 17, 2025 at 12:23 AM

rev. howard arson

@theophite.bsky.social

rank one scale envelope with two mantissae. dequants to bf16 with shifts, adds, and one GEMM transfer. oh yeah. nnh.

August 22, 2026 at 8:59 PM

mr. TIM

@timkellogg.me

these graphs are nuts

bf16 has been the only way training has been done for nearly a decade

all this year tons of resources have been dumped into RL, and this is saying most of that was wasted bc we chose the “cool” float format

mr. TIM @timkellogg.me · Oct 31

astonishing: using fp16 instead of bf16 results in more stable training runs as well as a smaller performance gap between training & inference

this is critical for RL, which is mostly inference and very sensitive to reproducible results

October 31, 2025 at 9:28 PM

Sung Kim

@sungkim.bsky.social

Alibaba Ant's Ling-3.0-tiny is now available as open-weight:

BF16: huggingface.co/inclusionAI/...
FP8: huggingface.co/inclusionAI/...
INT4: huggingface.co/inclusionAI/...

August 10, 2026 at 5:24 PM

xjdr

@xjdr.bsky.social

405B Base (bf16) will withstand the test of time and remain eternal (18 months)

November 25, 2024 at 7:14 AM

Sung Kim

@sungkim.bsky.social

You can try out the newest and greatest Google TPU:

v6e-1 (Trillium) TPUs!! 2x the high bandwidth memory as v5e-1 (32GB) and a whopping peak rating of 918 BF16 TFLOPS (nearly 3x A100)!

on Google Colab.