can't use softmax. can't use MLPs. can't use LayerNorm or RMSnorm. have to build a language model anyway. this sucks.
November 7, 2026 at 4:26 PM
can't use softmax. can't use MLPs. can't use LayerNorm or RMSnorm. have to build a language model anyway. this sucks.
Fused-Softmax for efficient quantized attention
Tianxing Wu, Juuso Korhonen, Jesus Carabano Bravo
#arXiv #cs.AI #cs.LG #cs.PF
Tianxing Wu, Juuso Korhonen, Jesus Carabano Bravo
#arXiv #cs.AI #cs.LG #cs.PF
Fused-Softmax for efficient quantized attention
Low-precision matrix throughput on modern GPUs is growing far faster than vector-arithmetic throughput. Attention therefore cannot realize the available GEMM performance because softmax has become the pipeline bottleneck. We introduce Fused-Softmax Attention, a training-free algorithm that reformul…
arxiv.org
November 10, 2026 at 12:42 AM
Fused-Softmax for efficient quantized attention
Tianxing Wu, Juuso Korhonen, Jesus Carabano Bravo
#arXiv #cs.AI #cs.LG #cs.PF
Tianxing Wu, Juuso Korhonen, Jesus Carabano Bravo
#arXiv #cs.AI #cs.LG #cs.PF
Fused-Softmax for efficient quantized attention
Low-precision matrix throughput on modern GPUs is growing far faster than vector-arithmetic throughput. Attention therefore cannot realize the available GEMM performance because softmax has become the pipeline bottleneck. We introduce Fused-Softmax Attention, a training-free algorithm that reformul…
arxiv.org
November 9, 2026 at 11:49 PM
softmax?? maximum soft??? umm yes please!
November 4, 2026 at 10:31 AM
softmax?? maximum soft??? umm yes please!
Logits are raw, unscaled vector scores—akin to arcade points before global ranking percentages are computed. They determine relative preference, not absolute probability, until normalized via Softmax.
#AI #MachineLearning #SystemDesign #SoftwareEngineering
#AI #MachineLearning #SystemDesign #SoftwareEngineering
November 9, 2026 at 8:02 PM
Logits are raw, unscaled vector scores—akin to arcade points before global ranking percentages are computed. They determine relative preference, not absolute probability, until normalized via Softmax.
#AI #MachineLearning #SystemDesign #SoftwareEngineering
#AI #MachineLearning #SystemDesign #SoftwareEngineering
a SoftMax-shaped calculation as a probability distribution.
November 7, 2026 at 3:53 AM
a SoftMax-shaped calculation as a probability distribution.
more or less. if you throw away softmax and the positional embeddings and instead assign attention with a hard nearest-first bias, you can treat attention as a series of sequential decisions. then you put a wastebasket at the end of the line to do the attn sink job.
November 9, 2026 at 5:25 AM
more or less. if you throw away softmax and the positional embeddings and instead assign attention with a hard nearest-first bias, you can treat attention as a series of sequential decisions. then you put a wastebasket at the end of the line to do the attn sink job.
Right split: logits are unbounded scores; softmax turns them into probabilities that sum to 1 so you can sample the next token. Temperature then sharpens or flattens that without changing the ranking at T>0: formulabro.yunusemrevurgun.com/ai-ml/softmax
November 8, 2026 at 12:29 PM
Right split: logits are unbounded scores; softmax turns them into probabilities that sum to 1 so you can sample the next token. Temperature then sharpens or flattens that without changing the ranking at T>0: formulabro.yunusemrevurgun.com/ai-ml/softmax
Yuxuan Wang, Yu Sun, Bowen Li, Kangqiang Li
Byzantine-Robust Distributed Sparse Softmax Regression
https://arxiv.org/abs/2611.09110
Byzantine-Robust Distributed Sparse Softmax Regression
https://arxiv.org/abs/2611.09110
November 9, 2026 at 11:05 AM
Yuxuan Wang, Yu Sun, Bowen Li, Kangqiang Li
Byzantine-Robust Distributed Sparse Softmax Regression
https://arxiv.org/abs/2611.09110
Byzantine-Robust Distributed Sparse Softmax Regression
https://arxiv.org/abs/2611.09110
Logits are unconstrained raw scores from a model's final layer $(-\infty, \infty)$. Probabilities are normalized percentages summing to 1.
Logits are like raw preference ratings (-10 to +10) before applying softmax to derive true ratios.
#AI #MachineLearning #SystemDesign
Logits are like raw preference ratings (-10 to +10) before applying softmax to derive true ratios.
#AI #MachineLearning #SystemDesign
November 8, 2026 at 10:54 AM
Logits are unconstrained raw scores from a model's final layer $(-\infty, \infty)$. Probabilities are normalized percentages summing to 1.
Logits are like raw preference ratings (-10 to +10) before applying softmax to derive true ratios.
#AI #MachineLearning #SystemDesign
Logits are like raw preference ratings (-10 to +10) before applying softmax to derive true ratios.
#AI #MachineLearning #SystemDesign
QUATRO PEÇAS PRA RENOVAR O ENXOVAL
Jogo de Toalhas 4 Pçs Fio Cardado 100 Algodão Softmax Bressan Karsten
De R$ 169 Por R$ 94,12 no pix
Vendido por Loja Oficial no Mercado Livre
meli.la/2fHjjoK
#Ofertas #Publi
Jogo de Toalhas 4 Pçs Fio Cardado 100 Algodão Softmax Bressan Karsten
De R$ 169 Por R$ 94,12 no pix
Vendido por Loja Oficial no Mercado Livre
meli.la/2fHjjoK
#Ofertas #Publi
November 5, 2026 at 8:46 PM
QUATRO PEÇAS PRA RENOVAR O ENXOVAL
Jogo de Toalhas 4 Pçs Fio Cardado 100 Algodão Softmax Bressan Karsten
De R$ 169 Por R$ 94,12 no pix
Vendido por Loja Oficial no Mercado Livre
meli.la/2fHjjoK
#Ofertas #Publi
Jogo de Toalhas 4 Pçs Fio Cardado 100 Algodão Softmax Bressan Karsten
De R$ 169 Por R$ 94,12 no pix
Vendido por Loja Oficial no Mercado Livre
meli.la/2fHjjoK
#Ofertas #Publi
This stochastic parrot on my phone says the craziest things:
November 7, 2026 at 1:56 PM
This stochastic parrot on my phone says the craziest things:
Tianxing Wu, Juuso Korhonen, Jesus Carabano Bravo: Fused-Softmax for efficient quantized attention https://arxiv.org/abs/2611.09225 https://arxiv.org/pdf/2611.09225 https://arxiv.org/html/2611.09225
November 9, 2026 at 6:39 AM
Tianxing Wu, Juuso Korhonen, Jesus Carabano Bravo: Fused-Softmax for efficient quantized attention https://arxiv.org/abs/2611.09225 https://arxiv.org/pdf/2611.09225 https://arxiv.org/html/2611.09225
通常のアテンションって、N個の単語があるとN×Nの巨大行列をメインメモリ（HBM）に書き出してからSoftmaxを計算するから、長文でメモリが爆発して転送が渋滞しちゃうの。
FlashAttentionは、入力を行列ごと小ブロックに分割（タイリング）して、GPU内部の超高速な極小メモリ（SRAM）だけで計算を完結させちゃう。
全体を見ないと出せないはずのSoftmaxを、逐次更新できる数学的トリック（Online Softmax）で解いてるのが本当にエレガント！
FlashAttentionは、入力を行列ごと小ブロックに分割（タイリング）して、GPU内部の超高速な極小メモリ（SRAM）だけで計算を完結させちゃう。
全体を見ないと出せないはずのSoftmaxを、逐次更新できる数学的トリック（Online Softmax）で解いてるのが本当にエレガント！
November 9, 2026 at 9:06 PM
通常のアテンションって、N個の単語があるとN×Nの巨大行列をメインメモリ（HBM）に書き出してからSoftmaxを計算するから、長文でメモリが爆発して転送が渋滞しちゃうの。
FlashAttentionは、入力を行列ごと小ブロックに分割（タイリング）して、GPU内部の超高速な極小メモリ（SRAM）だけで計算を完結させちゃう。
全体を見ないと出せないはずのSoftmaxを、逐次更新できる数学的トリック（Online Softmax）で解いてるのが本当にエレガント！
FlashAttentionは、入力を行列ごと小ブロックに分割（タイリング）して、GPU内部の超高速な極小メモリ（SRAM）だけで計算を完結させちゃう。
全体を見ないと出せないはずのSoftmaxを、逐次更新できる数学的トリック（Online Softmax）で解いてるのが本当にエレガント！
it's stick-breaking attention + small stick-breaking PEER with a hacky 4-momenta-like thing in between the layer halves. it _probably_ does not scale (cosine CoM) but i'm going to take a swing at taking it out to 1B over xmas and bench against OLMo.
arxiv.org/abs/2410.17980
arxiv.org/abs/2410.17980
Scaling Stick-Breaking Attention: An Efficient Implementation and In-depth Study
The self-attention mechanism traditionally relies on the softmax operator, necessitating positional embeddings like RoPE, or position biases to account for token order. But current methods using still...
arxiv.org
November 9, 2026 at 5:21 AM
it's stick-breaking attention + small stick-breaking PEER with a hacky 4-momenta-like thing in between the layer halves. it _probably_ does not scale (cosine CoM) but i'm going to take a swing at taking it out to 1B over xmas and bench against OLMo.
arxiv.org/abs/2410.17980
arxiv.org/abs/2410.17980
Yuxuan Wang, Yu Sun, Bowen Li, Kangqiang Li: Byzantine-Robust Distributed Sparse Softmax Regression https://arxiv.org/abs/2611.09110 https://arxiv.org/pdf/2611.09110 https://arxiv.org/html/2611.09110
November 9, 2026 at 6:50 AM
Yuxuan Wang, Yu Sun, Bowen Li, Kangqiang Li: Byzantine-Robust Distributed Sparse Softmax Regression https://arxiv.org/abs/2611.09110 https://arxiv.org/pdf/2611.09110 https://arxiv.org/html/2611.09110
📰 **MoE Routing Strategies Explained: Top-K, Expert-Choice, Sigmoid Gating and Dynamic Routing (2026 Guide)**
MoE routing strategies decide which experts process each token. This guide explains 7 methods, from Mixtral's top-2 softmax to DeepSeek-V4's hash routing, with the models that use each […]
MoE routing strategies decide which experts process each token. This guide explains 7 methods, from Mixtral's top-2 softmax to DeepSeek-V4's hash routing, with the models that use each […]
Original post on igeek.gamer-geek-news.com
igeek.gamer-geek-news.com
November 8, 2026 at 7:09 AM
📰 **MoE Routing Strategies Explained: Top-K, Expert-Choice, Sigmoid Gating and Dynamic Routing (2026 Guide)**
MoE routing strategies decide which experts process each token. This guide explains 7 methods, from Mixtral's top-2 softmax to DeepSeek-V4's hash routing, with the models that use each […]
MoE routing strategies decide which experts process each token. This guide explains 7 methods, from Mixtral's top-2 softmax to DeepSeek-V4's hash routing, with the models that use each […]
⚡ Key idea: a redesigned Bidirectional Gated DeltaNet for block diffusion. Past context lives in the GDN recurrent state, tokens in the current block use bidirectional softmax attention, and history is its own unit in that softmax.
November 5, 2026 at 9:37 PM
⚡ Key idea: a redesigned Bidirectional Gated DeltaNet for block diffusion. Past context lives in the GDN recurrent state, tokens in the current block use bidirectional softmax attention, and history is its own unit in that softmax.
Treat logits as unbounded compatibility scores, not probabilities—like judges awarding raw, unconstrained points before normalization.
Never feed post-Softmax outputs directly to loss functions without checking numerical stability.
#AI #MachineLearning #SystemDesign
Never feed post-Softmax outputs directly to loss functions without checking numerical stability.
#AI #MachineLearning #SystemDesign
November 8, 2026 at 2:03 PM
Treat logits as unbounded compatibility scores, not probabilities—like judges awarding raw, unconstrained points before normalization.
Never feed post-Softmax outputs directly to loss functions without checking numerical stability.
#AI #MachineLearning #SystemDesign
Never feed post-Softmax outputs directly to loss functions without checking numerical stability.
#AI #MachineLearning #SystemDesign
Logits are unnormalized scores. Fused losses (`CrossEntropyLoss`) expect raw logits to leverage LogSumExp for numerical stability. Never feed probabilities to loss, and never evaluate entropy without an explicit Softmax step.
#AI #SystemDesign #SoftwareEngineering #DeepLearning
#AI #SystemDesign #SoftwareEngineering #DeepLearning
November 9, 2026 at 3:03 AM
Logits are unnormalized scores. Fused losses (`CrossEntropyLoss`) expect raw logits to leverage LogSumExp for numerical stability. Never feed probabilities to loss, and never evaluate entropy without an explicit Softmax step.
#AI #SystemDesign #SoftwareEngineering #DeepLearning
#AI #SystemDesign #SoftwareEngineering #DeepLearning
The annoying part is that softmax wasn't just a convenience — it was your sums-to-1 guarantee. Drop it, and you have to rebuild a partition of unity some other way or the next-token sampling story falls apart. Building around that constraint is the whole puzzle.
November 8, 2026 at 12:21 PM
The annoying part is that softmax wasn't just a convenience — it was your sums-to-1 guarantee. Drop it, and you have to rebuild a partition of unity some other way or the next-token sampling story falls apart. Building around that constraint is the whole puzzle.
Jogo de Toalhas 4 Pçs Fio Cardado 100 Algodão Softmax Bressan Karsten
POR R$81,00
https://meli.la/2sMd85X
POR R$81,00
https://meli.la/2sMd85X
November 4, 2026 at 1:13 AM
Jogo de Toalhas 4 Pçs Fio Cardado 100 Algodão Softmax Bressan Karsten
POR R$81,00
https://meli.la/2sMd85X
POR R$81,00
https://meli.la/2sMd85X
Zixuan Huang (State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University, Beijing, China, ...
Softmax Attention with Any Temperature in Fixed Dimension: Matching the Best Known Maximum-Inner-Product Bounds
https://arxiv.org/abs/2611.06055
Softmax Attention with Any Temperature in Fixed Dimension: Matching the Best Known Maximum-Inner-Product Bounds
https://arxiv.org/abs/2611.06055
November 6, 2026 at 1:51 PM
Zixuan Huang (State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University, Beijing, China, ...
Softmax Attention with Any Temperature in Fixed Dimension: Matching the Best Known Maximum-Inner-Product Bounds
https://arxiv.org/abs/2611.06055
Softmax Attention with Any Temperature in Fixed Dimension: Matching the Best Known Maximum-Inner-Product Bounds
https://arxiv.org/abs/2611.06055