float32 is a nonlinearity
May 15, 2026 at 1:20 AM
float32 is a nonlinearity
machine learning needs a nonlinear function, such as a transition between a float32 layer and an INT4 layer
March 14, 2026 at 1:07 AM
machine learning needs a nonlinear function, such as a transition between a float32 layer and an INT4 layer
now that people are paying attention again, here is your periodic reminder. Always run in bf16. always apply ROPE and attention softmax at float32 (as shown here)
github.com/xjdr-alt/ent...
github.com/xjdr-alt/ent...
November 24, 2024 at 5:23 PM
now that people are paying attention again, here is your periodic reminder. Always run in bf16. always apply ROPE and attention softmax at float32 (as shown here)
github.com/xjdr-alt/ent...
github.com/xjdr-alt/ent...
lmao "SolidGoldMagikarp," in a tied-embedding model, literally means "produce whatever float32 numerical error tells you to."
March 14, 2026 at 12:24 AM
lmao "SolidGoldMagikarp," in a tied-embedding model, literally means "produce whatever float32 numerical error tells you to."
good news, we have the FIM in closed form for every layer: float32 is a log-uniform finite categorical distribution.
Just one more feature, one of these is going to be continuous, I know it
April 15, 2026 at 5:35 PM
good news, we have the FIM in closed form for every layer: float32 is a log-uniform finite categorical distribution.
Almost there!
Performance tip of the day: use Float32 instead of Float. Turned this from unusable into almost usable. 😅
#playdate #EmbeddedSwift #mandelbrot #fractals
Performance tip of the day: use Float32 instead of Float. Turned this from unusable into almost usable. 😅
#playdate #EmbeddedSwift #mandelbrot #fractals
December 30, 2024 at 7:02 PM
Almost there!
Performance tip of the day: use Float32 instead of Float. Turned this from unusable into almost usable. 😅
#playdate #EmbeddedSwift #mandelbrot #fractals
Performance tip of the day: use Float32 instead of Float. Turned this from unusable into almost usable. 😅
#playdate #EmbeddedSwift #mandelbrot #fractals
add to that the fact that Saxe correctly pointed out that Tishby's compression model of training used stupid bucketing, and then went on to ignore that tanh actually does saturate definitively rather than asymptotically in float32.
September 18, 2026 at 3:18 AM
add to that the fact that Saxe correctly pointed out that Tishby's compression model of training used stupid bucketing, and then went on to ignore that tanh actually does saturate definitively rather than asymptotically in float32.
anyway it's actually really important that the accumulation step of your batch is in float32 no matter what everything else is being done in.
December 8, 2025 at 7:00 AM
anyway it's actually really important that the accumulation step of your batch is in float32 no matter what everything else is being done in.
brb proposing a new text encoding standard, UTF-768×float32
the seahorse emoji exists as a natural region of embedding space that sadly has no corresponding token attached to it yet
Never ask a man his age, a woman her salary, or GPT-5 whether a seahorse emoji exists
September 17, 2025 at 9:01 AM
brb proposing a new text encoding standard, UTF-768×float32
millenia of research-hours wasted on reasoning about probability distributions calculated in float32 over 50,000 separate entries, a thing that is insane
March 4, 2026 at 1:02 AM
millenia of research-hours wasted on reasoning about probability distributions calculated in float32 over 50,000 separate entries, a thing that is insane
10^9 is 30 or 32 bits depending on signed or unsigned, so breaking down at 10 digits pretty much means it’s as accurate as float32! That’s pretty interesting!
February 13, 2025 at 4:36 AM
10^9 is 30 or 32 bits depending on signed or unsigned, so breaking down at 10 digits pretty much means it’s as accurate as float32! That’s pretty interesting!
i guess "positive definite enough so that if it's in float32 you're not dividing by zero," not "positive definite exactly."
March 20, 2026 at 9:14 PM
i guess "positive definite enough so that if it's in float32 you're not dividing by zero," not "positive definite exactly."
Wow super interesting. Quantizing might not be as big of a cost saver as I thought.
"... reducing from float32 (32 bits) to float16 (16 bits) or int8 (8 bits) should yield ideal 2× or 4× gains, respectively. However, such improvements are not observed in practice."
"... reducing from float32 (32 bits) to float16 (16 bits) or int8 (8 bits) should yield ideal 2× or 4× gains, respectively. However, such improvements are not observed in practice."
"numerical precision reduction yields the most benefit in the prefill phase of large models, where compute dominates. In contrast, the decode phase remains memory-limited, and aggressive quantization (e.g., int8 or int4) may incur overheads that outweigh theoretical savings"
Understanding Efficiency: Quantization, Batching, and Serving Strategies in LLM Energy Use
Large Language Models (LLMs) are increasingly deployed in production, contributing towards shifting the burden in terms of computational resources and energy demands from training to inference. While ...
arxiv.org
February 2, 2026 at 6:23 PM
Wow super interesting. Quantizing might not be as big of a cost saver as I thought.
"... reducing from float32 (32 bits) to float16 (16 bits) or int8 (8 bits) should yield ideal 2× or 4× gains, respectively. However, such improvements are not observed in practice."
"... reducing from float32 (32 bits) to float16 (16 bits) or int8 (8 bits) should yield ideal 2× or 4× gains, respectively. However, such improvements are not observed in practice."
Writing a fixed-point position class, and Neptune is 4.4 billion km away. So a reasonable scale for the upper 32bits would be (1<<32)km. But converting to meters that's (1<<32)*1000, which is not an exact float32. How about I use (1<<32)*1024 instead? So not kilometers, but... kibimeters?
May 4, 2026 at 4:59 AM
Writing a fixed-point position class, and Neptune is 4.4 billion km away. So a reasonable scale for the upper 32bits would be (1<<32)km. But converting to meters that's (1<<32)*1000, which is not an exact float32. How about I use (1<<32)*1024 instead? So not kilometers, but... kibimeters?
Volumetric exploration of multi-resolution CryoET data in native WebGPU.
Kiln (0.3.0) adds float32 support, OME-Zarr v0.4, slice views and more.
github.com/MPanknin/kil...
Vibrio cholerae tomogram from CryoET Data Portal.
cryoetdataportal.czscience.com
#WebGPU #OME-Zarr #CryoET
Kiln (0.3.0) adds float32 support, OME-Zarr v0.4, slice views and more.
github.com/MPanknin/kil...
Vibrio cholerae tomogram from CryoET Data Portal.
cryoetdataportal.czscience.com
#WebGPU #OME-Zarr #CryoET
May 26, 2026 at 11:31 AM
Volumetric exploration of multi-resolution CryoET data in native WebGPU.
Kiln (0.3.0) adds float32 support, OME-Zarr v0.4, slice views and more.
github.com/MPanknin/kil...
Vibrio cholerae tomogram from CryoET Data Portal.
cryoetdataportal.czscience.com
#WebGPU #OME-Zarr #CryoET
Kiln (0.3.0) adds float32 support, OME-Zarr v0.4, slice views and more.
github.com/MPanknin/kil...
Vibrio cholerae tomogram from CryoET Data Portal.
cryoetdataportal.czscience.com
#WebGPU #OME-Zarr #CryoET
Same thing, só umas casas a mais pra lá => do que o float32
May 5, 2025 at 11:13 AM
Same thing, só umas casas a mais pra lá => do que o float32
POV: you really, really, *REALLY* do not want to add an extra "math32" library
October 14, 2024 at 6:23 PM
POV: you really, really, *REALLY* do not want to add an extra "math32" library
'Alas, I could not be coerced into float32, so I'm one again playing essentially just myself in this romcom...'
June 29, 2026 at 9:29 PM
'Alas, I could not be coerced into float32, so I'm one again playing essentially just myself in this romcom...'
Releasing geo-index 0.2: Fast, memory-efficient, zero-copy spatial indexes (RTree & KDTree) for Python & Rust.
- Range, nearest-neighbor queries
- Persist indexes as bytes, reuse them later
- Supports float32 for 2x less memory
- Enables GeoParquet spatial partitioning
github.com/kylebarron/g...
- Range, nearest-neighbor queries
- Persist indexes as bytes, reuse them later
- Supports float32 for 2x less memory
- Enables GeoParquet spatial partitioning
github.com/kylebarron/g...
GitHub - kylebarron/geo-index: A Rust crate and Python library for packed, immutable, zero-copy spatial indexes.
A Rust crate and Python library for packed, immutable, zero-copy spatial indexes. - kylebarron/geo-index
github.com
January 7, 2025 at 3:36 PM
Releasing geo-index 0.2: Fast, memory-efficient, zero-copy spatial indexes (RTree & KDTree) for Python & Rust.
- Range, nearest-neighbor queries
- Persist indexes as bytes, reuse them later
- Supports float32 for 2x less memory
- Enables GeoParquet spatial partitioning
github.com/kylebarron/g...
- Range, nearest-neighbor queries
- Persist indexes as bytes, reuse them later
- Supports float32 for 2x less memory
- Enables GeoParquet spatial partitioning
github.com/kylebarron/g...
that's okay. log-uniform float32 bucketing is a reparameterization gradient so actually it all works.
June 25, 2026 at 4:40 AM
that's okay. log-uniform float32 bucketing is a reparameterization gradient so actually it all works.
I just had a brainwave. One of the ones that makes you go "Wow, until one second ago I was stupid."
The model I downloaded is bfloat16. That has surprisingly poor decimal-digit precision.
If Qwen 3 was trained in float32, 814 tightly spaced vectors could totally collapse into the same point.
🤔
The model I downloaded is bfloat16. That has surprisingly poor decimal-digit precision.
If Qwen 3 was trained in float32, 814 tightly spaced vectors could totally collapse into the same point.
🤔
I just discovered something interesting.
There is a point in Qwen 3 4B Instruct 2507's embedding space where 814 tokens all live.
These 814 tokens (out of 151,936) map to exactly the same point — their vector representations are bit-for-bit identical. The model literally can't tell them apart.
There is a point in Qwen 3 4B Instruct 2507's embedding space where 814 tokens all live.
These 814 tokens (out of 151,936) map to exactly the same point — their vector representations are bit-for-bit identical. The model literally can't tell them apart.
November 4, 2025 at 5:49 PM
I just had a brainwave. One of the ones that makes you go "Wow, until one second ago I was stupid."
The model I downloaded is bfloat16. That has surprisingly poor decimal-digit precision.
If Qwen 3 was trained in float32, 814 tightly spaced vectors could totally collapse into the same point.
🤔
The model I downloaded is bfloat16. That has surprisingly poor decimal-digit precision.
If Qwen 3 was trained in float32, 814 tightly spaced vectors could totally collapse into the same point.
🤔
Today: experimenting #golang and #ebitengine
func (g *Game) Draw(screen *ebiten.Image) {
screen.Fill(g.background)
for _, ball := range g.balls {
vector.FillCircle(screen, float32(ball.X), float32(ball.Y), float32(ball.Radius), ball.color, false)
}
func (g *Game) Draw(screen *ebiten.Image) {
screen.Fill(g.background)
for _, ball := range g.balls {
vector.FillCircle(screen, float32(ball.X), float32(ball.Y), float32(ball.Radius), ball.color, false)
}
May 31, 2026 at 7:40 PM
Today: experimenting #golang and #ebitengine
func (g *Game) Draw(screen *ebiten.Image) {
screen.Fill(g.background)
for _, ball := range g.balls {
vector.FillCircle(screen, float32(ball.X), float32(ball.Y), float32(ball.Radius), ball.color, false)
}
func (g *Game) Draw(screen *ebiten.Image) {
screen.Fill(g.background)
for _, ball := range g.balls {
vector.FillCircle(screen, float32(ball.X), float32(ball.Y), float32(ball.Radius), ball.color, false)
}
?????????? all the best practices i have ever seen for diffusion are just, like, 'yeah, sometimes you're going to have to train in float32. that's just how it is on this bitch of an earth'
March 19, 2026 at 1:45 AM
?????????? all the best practices i have ever seen for diffusion are just, like, 'yeah, sometimes you're going to have to train in float32. that's just how it is on this bitch of an earth'
No longer confident! Web search says that Qwen 3 was never float32. It was bfloat16 from the start.
Which suggests that we might be looking at a collapse during training due to bfloat16's limited precision. If they'd trained in float32, these tokens would have been separated by a small distance.
Which suggests that we might be looking at a collapse during training due to bfloat16's limited precision. If they'd trained in float32, these tokens would have been separated by a small distance.
November 4, 2025 at 6:33 PM
No longer confident! Web search says that Qwen 3 was never float32. It was bfloat16 from the start.
Which suggests that we might be looking at a collapse during training due to bfloat16's limited precision. If they'd trained in float32, these tokens would have been separated by a small distance.
Which suggests that we might be looking at a collapse during training due to bfloat16's limited precision. If they'd trained in float32, these tokens would have been separated by a small distance.