Sparsity is all you need?
November 26, 2024 at 12:08 AM
Sparsity is all you need?
Handed in a first draft of my PhD thesis. 🥳
September 27, 2026 at 11:44 AM
Handed in a first draft of my PhD thesis. 🥳
I gotta say, I don't love the part of election night where TV broadcasters speculate wildly about a statistically laughable sparsity of data
April 29, 2025 at 1:48 AM
I gotta say, I don't love the part of election night where TV broadcasters speculate wildly about a statistically laughable sparsity of data
I have learned that in machine learning sparsity is measured as 'fraction of zeros', as in sparsity=0.9 means very sparse. Whereas in neuroscience sparsity is measured as 'fraction of non-zeros', so sparsity=0.1 would be very sparse
not sure what to do with this discovery but there you go
not sure what to do with this discovery but there you go
November 26, 2025 at 8:11 PM
I have learned that in machine learning sparsity is measured as 'fraction of zeros', as in sparsity=0.9 means very sparse. Whereas in neuroscience sparsity is measured as 'fraction of non-zeros', so sparsity=0.1 would be very sparse
not sure what to do with this discovery but there you go
not sure what to do with this discovery but there you go
I posted this last night cause I kind of wanted to bury it. I got cold feet about putting it out there.
embedding-space.github.io/sparse-netwo...
The subject is WHY neural networks work, and I think the answer I offer is kind of interesting. Maybe even a little correct, possibly.
embedding-space.github.io/sparse-netwo...
The subject is WHY neural networks work, and I think the answer I offer is kind of interesting. Maybe even a little correct, possibly.
October 8, 2025 at 1:57 PM
I posted this last night cause I kind of wanted to bury it. I got cold feet about putting it out there.
embedding-space.github.io/sparse-netwo...
The subject is WHY neural networks work, and I think the answer I offer is kind of interesting. Maybe even a little correct, possibly.
embedding-space.github.io/sparse-netwo...
The subject is WHY neural networks work, and I think the answer I offer is kind of interesting. Maybe even a little correct, possibly.
Epic grunt model :) (sorry for the sparsity of posts, I've been busy and I've also been trying to improve my rigging and texturing skills!)
September 5, 2026 at 4:37 AM
Epic grunt model :) (sorry for the sparsity of posts, I've been busy and I've also been trying to improve my rigging and texturing skills!)
Due to sparsity, Kimi K3 has fewer active parameters (104 billion) than GPT-3 (175 billion, same as total parameters)
August 2, 2026 at 2:41 AM
Due to sparsity, Kimi K3 has fewer active parameters (104 billion) than GPT-3 (175 billion, same as total parameters)
So happy to talk about our work on sparsity at #useR2025 today! Slides for anyone interested
emilhvitfeldt.com/talk/2025-08...
emilhvitfeldt.com/talk/2025-08...
sparsity support in tidymodels, faster and less memory hungry models
Showcasing better sparsity support in tidymodels
emilhvitfeldt.com
August 10, 2025 at 7:09 PM
So happy to talk about our work on sparsity at #useR2025 today! Slides for anyone interested
emilhvitfeldt.com/talk/2025-08...
emilhvitfeldt.com/talk/2025-08...
December 18, 2025 at 9:28 AM
"By systematically contaminating inputs with NaN and observing which outputs become NaN, the method reconstructs conservative sparsity patterns that eliminate a major source of false negatives."
absolutely incredible, and also someone needs to stop these people
arxiv.org/abs/2507.23186
absolutely incredible, and also someone needs to stop these people
arxiv.org/abs/2507.23186
NaN-Propagation: A Novel Method for Sparsity Detection in Black-Box Computational Functions
When numerically evaluating a function's gradient, sparsity detection can enable substantial computational speedups through Jacobian coloring and compression. However, sparsity detection techniques fo...
arxiv.org
August 13, 2025 at 8:26 PM
"By systematically contaminating inputs with NaN and observing which outputs become NaN, the method reconstructs conservative sparsity patterns that eliminate a major source of false negatives."
absolutely incredible, and also someone needs to stop these people
arxiv.org/abs/2507.23186
absolutely incredible, and also someone needs to stop these people
arxiv.org/abs/2507.23186
Slides from yesterdays talk "Don't be dense, embracing sparsity in tidymodels" are live if you are interested!
emilhvitfeldt.github.io/talk-slc-spa...
emilhvitfeldt.github.io/talk-slc-spa...
July 23, 2025 at 11:20 PM
Slides from yesterdays talk "Don't be dense, embracing sparsity in tidymodels" are live if you are interested!
emilhvitfeldt.github.io/talk-slc-spa...
emilhvitfeldt.github.io/talk-slc-spa...
They keep pushing the envelope on sparsity too, really impressive
September 10, 2026 at 4:34 PM
They keep pushing the envelope on sparsity too, really impressive
this is going to be controversial:
LLMs are neurosymbolic models. the structure of the representation space (specifically, sparsity + orthogonalization) creates an explicit albeit opaque learned ontology. leakages from that learned ontology are processed by things which look like PCA.
LLMs are neurosymbolic models. the structure of the representation space (specifically, sparsity + orthogonalization) creates an explicit albeit opaque learned ontology. leakages from that learned ontology are processed by things which look like PCA.
October 13, 2025 at 8:13 PM
this is going to be controversial:
LLMs are neurosymbolic models. the structure of the representation space (specifically, sparsity + orthogonalization) creates an explicit albeit opaque learned ontology. leakages from that learned ontology are processed by things which look like PCA.
LLMs are neurosymbolic models. the structure of the representation space (specifically, sparsity + orthogonalization) creates an explicit albeit opaque learned ontology. leakages from that learned ontology are processed by things which look like PCA.
How do we make LLMs faster and lighter? Don’t force the GPU to adapt to sparsity. Reshape the sparsity to fit the GPU!
Our latest work with NVIDIA introduces new CUDA kernels & data formats for faster inference and training of sparse transformer language models:
Blog: pub.sakana.ai/sparser-fast...
Our latest work with NVIDIA introduces new CUDA kernels & data formats for faster inference and training of sparse transformer language models:
Blog: pub.sakana.ai/sparser-fast...
Excited to share Sakana AI’s new #ICML2026 paper in collaboration with NVIDIA: "Sparser, Faster, Lighter Transformer Language Models" arxiv.org/abs/2603.23198
This work introduces new open-source GPU kernels and data formats for faster inference and training of sparse transformer LLMs:
🧵 Thread 👇
This work introduces new open-source GPU kernels and data formats for faster inference and training of sparse transformer LLMs:
🧵 Thread 👇
May 9, 2026 at 2:52 AM
How do we make LLMs faster and lighter? Don’t force the GPU to adapt to sparsity. Reshape the sparsity to fit the GPU!
Our latest work with NVIDIA introduces new CUDA kernels & data formats for faster inference and training of sparse transformer language models:
Blog: pub.sakana.ai/sparser-fast...
Our latest work with NVIDIA introduces new CUDA kernels & data formats for faster inference and training of sparse transformer language models:
Blog: pub.sakana.ai/sparser-fast...
Super excited to share this one!! Meta-learning sparsity and learning rate gives rise to brain-like gradients of complementary learning systems. So complementary learning systems emerge organically through behavior optimization, and it's not just two of them!!
Excited to share a new preprint w/ @annaschapiro.bsky.social! Why are there gradients of plasticity and sparsity along the neocortex–hippocampus hierarchy? We show that brain-like organization of these properties emerges in ANNs that meta-learn layer-wise plasticity and sparsity. bit.ly/4kB1yg5
A gradient of complementary learning systems emerges through meta-learning
Long-term learning and memory in the primate brain rely on a series of hierarchically organized subsystems extending from early sensory neocortical areas to the hippocampus. The components differ in t...
bit.ly
July 16, 2025 at 4:18 PM
Super excited to share this one!! Meta-learning sparsity and learning rate gives rise to brain-like gradients of complementary learning systems. So complementary learning systems emerge organically through behavior optimization, and it's not just two of them!!
They present a sparsity "scaling law", indicating that more sparsity leads to efficiency gains. They don't attach any numbers to the law directly, but state relative efficiency improvements compared to the 48x sparsity they do use that seem consistent across scales.
July 21, 2025 at 10:36 PM
They present a sparsity "scaling law", indicating that more sparsity leads to efficiency gains. They don't attach any numbers to the law directly, but state relative efficiency improvements compared to the 48x sparsity they do use that seem consistent across scales.
April 29, 2025 at 6:21 AM
It's that time of year when I get withdrawal symptoms from the sparsity of micros, so the Metalampra italica this morning in Ashurst, New Forest was welcome, as was the Acrocercops brongniardella earlier this week. Brick & Clifden Nonpareil nfy during the week #teammoth
September 26, 2026 at 9:38 AM
It's that time of year when I get withdrawal symptoms from the sparsity of micros, so the Metalampra italica this morning in Ashurst, New Forest was welcome, as was the Acrocercops brongniardella earlier this week. Brick & Clifden Nonpareil nfy during the week #teammoth
Right — even then you would see folks visiting their families in the area come to see the sight. By contrast this is just _unreal_ sparsity.
June 30, 2026 at 6:57 PM
Right — even then you would see folks visiting their families in the area come to see the sight. By contrast this is just _unreal_ sparsity.
I'm beyond excited to share with you all that tidymodels now have sparsity support.
We now support sparse data during the whole process, generate data sparsely in recipes steps when your model supports spare data structures. and you don't have to change anything!
www.tidyverse.org/blog/2025/03...
We now support sparse data during the whole process, generate data sparsely in recipes steps when your model supports spare data structures. and you don't have to change anything!
www.tidyverse.org/blog/2025/03...
Improved sparsity support in tidymodels
The tidymodels ecosystem now fully supports sparse data as input, output, and in creation.
www.tidyverse.org
March 19, 2025 at 6:02 PM
I'm beyond excited to share with you all that tidymodels now have sparsity support.
We now support sparse data during the whole process, generate data sparsely in recipes steps when your model supports spare data structures. and you don't have to change anything!
www.tidyverse.org/blog/2025/03...
We now support sparse data during the whole process, generate data sparsely in recipes steps when your model supports spare data structures. and you don't have to change anything!
www.tidyverse.org/blog/2025/03...
Another nano gem from my amazing student
Piotr Nawrot!
A repo & notebook on sparse attention for efficient LLM inference: github.com/PiotrNawrot/...
This will also feature in my #NeurIPS 2024 tutorial "Dynamic Sparsity in ML" with André Martins: dynamic-sparsity.github.io Stay tuned!
Piotr Nawrot!
A repo & notebook on sparse attention for efficient LLM inference: github.com/PiotrNawrot/...
This will also feature in my #NeurIPS 2024 tutorial "Dynamic Sparsity in ML" with André Martins: dynamic-sparsity.github.io Stay tuned!
November 20, 2024 at 12:51 PM
Another nano gem from my amazing student
Piotr Nawrot!
A repo & notebook on sparse attention for efficient LLM inference: github.com/PiotrNawrot/...
This will also feature in my #NeurIPS 2024 tutorial "Dynamic Sparsity in ML" with André Martins: dynamic-sparsity.github.io Stay tuned!
Piotr Nawrot!
A repo & notebook on sparse attention for efficient LLM inference: github.com/PiotrNawrot/...
This will also feature in my #NeurIPS 2024 tutorial "Dynamic Sparsity in ML" with André Martins: dynamic-sparsity.github.io Stay tuned!
Excited to share a new preprint w/ @annaschapiro.bsky.social! Why are there gradients of plasticity and sparsity along the neocortex–hippocampus hierarchy? We show that brain-like organization of these properties emerges in ANNs that meta-learn layer-wise plasticity and sparsity. bit.ly/4kB1yg5
A gradient of complementary learning systems emerges through meta-learning
Long-term learning and memory in the primate brain rely on a series of hierarchically organized subsystems extending from early sensory neocortical areas to the hippocampus. The components differ in t...
bit.ly
July 16, 2025 at 4:15 PM
Excited to share a new preprint w/ @annaschapiro.bsky.social! Why are there gradients of plasticity and sparsity along the neocortex–hippocampus hierarchy? We show that brain-like organization of these properties emerges in ANNs that meta-learn layer-wise plasticity and sparsity. bit.ly/4kB1yg5
oh wow, academics: pay attention
this “blog” is such an amazing way to announce your work. i imagine it’s all LLM generated, but wow, so incredibly clear
reminds me of one of those Financial Times infographic stories
pumpkin-co.github.io/SparsityAndC...
this “blog” is such an amazing way to announce your work. i imagine it’s all LLM generated, but wow, so incredibly clear
reminds me of one of those Financial Times infographic stories
pumpkin-co.github.io/SparsityAndC...
When Does Sparsity Mitigate the Curse of Depth in LLMs
Revealing sparsity as an intrinsic variance regulator that unlocks effective depth utilization in Large Language Models.
pumpkin-co.github.io
March 18, 2026 at 1:18 AM
oh wow, academics: pay attention
this “blog” is such an amazing way to announce your work. i imagine it’s all LLM generated, but wow, so incredibly clear
reminds me of one of those Financial Times infographic stories
pumpkin-co.github.io/SparsityAndC...
this “blog” is such an amazing way to announce your work. i imagine it’s all LLM generated, but wow, so incredibly clear
reminds me of one of those Financial Times infographic stories
pumpkin-co.github.io/SparsityAndC...
Opinion - Trump’s threats should remind us of Canada’s underpopulation risk: In a tariff battle, our sparsity would be the cause of a number of negative side effects
Trump’s threats should remind us of Canada’s underpopulation risk
In a tariff battle, our sparsity would be the cause of a number of negative side effects
www.theglobeandmail.com
December 9, 2024 at 1:03 PM
Opinion - Trump’s threats should remind us of Canada’s underpopulation risk: In a tariff battle, our sparsity would be the cause of a number of negative side effects
Self-recommending, despite the sparsity of Maimonides references.
Sofia Serrano, Zander Brumbaugh, Noah A. Smith
Language Models: A Guide for the Perplexed. (arXiv:2311.17301v1 [cs.CL])
http://arxiv.org/abs/2311.17301
Language Models: A Guide for the Perplexed. (arXiv:2311.17301v1 [cs.CL])
http://arxiv.org/abs/2311.17301
November 30, 2023 at 5:32 PM
Self-recommending, despite the sparsity of Maimonides references.