July 2, 2026 at 1:12 PM
April 5, 2025 at 11:33 PM
imo "central planning via backpropagation with multiple betas" might actually work
Years later I'm still haunted by the crookedtimber commenter who, when asked to explain how acquiring necessities in his ideal postcapitalist utopia would work, explained that you and your neighbors would meet to democratically pick what groceries to order and I'm sorry that's a vision of hell.
December 9, 2024 at 2:27 AM
imo "central planning via backpropagation with multiple betas" might actually work
🧵New paper: "Lost in Backpropagation: The LM Head is a Gradient Bottleneck"
The output layer of LLMs destroys 95-99% of your training signal during backpropagation, and this significantly slows down pretraining 👇
The output layer of LLMs destroys 95-99% of your training signal during backpropagation, and this significantly slows down pretraining 👇
March 12, 2026 at 7:51 PM
🧵New paper: "Lost in Backpropagation: The LM Head is a Gradient Bottleneck"
The output layer of LLMs destroys 95-99% of your training signal during backpropagation, and this significantly slows down pretraining 👇
The output layer of LLMs destroys 95-99% of your training signal during backpropagation, and this significantly slows down pretraining 👇
backpropagation was invented in 1676
March 16, 2026 at 1:33 AM
backpropagation was invented in 1676
“I’m shocked, shocked, to find out backpropagation is going on in here”
February 22, 2026 at 6:52 PM
“I’m shocked, shocked, to find out backpropagation is going on in here”
Remember how I said backprop will die soon? @sakanaai.bsky.social has begun that process, and this isn't the only promising avenue that's doing it:
Augmented Lagrangian Predictive Coding: training 1000-layer networks without backpropagation
A local alternative to backpropagation. PC-ALM trains residual MLPs up to 1000 layers, nearly matching backprop's performance despite using only layer-local dynamics. PC-ALM equips each layer with a f...
pub.sakana.ai
September 14, 2026 at 3:32 PM
Remember how I said backprop will die soon? @sakanaai.bsky.social has begun that process, and this isn't the only promising avenue that's doing it:
me: watch 2 hour video about backpropagation in basic python with basic math at 2x speed
brain: great. now it’s time for your soul to leave your body
brain: great. now it’s time for your soul to leave your body
June 15, 2025 at 9:23 PM
me: watch 2 hour video about backpropagation in basic python with basic math at 2x speed
brain: great. now it’s time for your soul to leave your body
brain: great. now it’s time for your soul to leave your body
Pay attention. Here’s “the Godfather of Artificial Intelligence,” Geoffrey Hinton, accepting the 2024 Nobel Prize. Hinton co-developed the 1980s backpropagation algorithm, which laid the foundation for A.I.
Hinton left Google in 2023, so he could speak freely about the risks A.I. poses to humanity:
Hinton left Google in 2023, so he could speak freely about the risks A.I. poses to humanity:
August 9, 2025 at 7:58 PM
Pay attention. Here’s “the Godfather of Artificial Intelligence,” Geoffrey Hinton, accepting the 2024 Nobel Prize. Hinton co-developed the 1980s backpropagation algorithm, which laid the foundation for A.I.
Hinton left Google in 2023, so he could speak freely about the risks A.I. poses to humanity:
Hinton left Google in 2023, so he could speak freely about the risks A.I. poses to humanity:
Backpropagation can seem intimidating until you understand the idea that makes it work. Nikhil Dasari breaks down the core concept behind backpropagation in a beginner-friendly way.
Backpropagation Explained for Beginners (Part 2): There Has to Be a Better Way | Towards Data Science
The idea that makes backpropagation possible.
towardsdatascience.com
September 24, 2026 at 9:02 PM
Backpropagation can seem intimidating until you understand the idea that makes it work. Nikhil Dasari breaks down the core concept behind backpropagation in a beginner-friendly way.
Well you see, I spent 90 minutes reading how training math works and what a transformer is and what backpropagation is. It seems like spending 90 minutes of time acquiring a child-like understanding of the world's most controversial technology would be helpful if I wanted to have opinions on it
September 16, 2026 at 5:05 PM
Well you see, I spent 90 minutes reading how training math works and what a transformer is and what backpropagation is. It seems like spending 90 minutes of time acquiring a child-like understanding of the world's most controversial technology would be helpful if I wanted to have opinions on it
My go to would be 3blue1brown. youtube.com/playlist?lis...
Neural networks - YouTube
Learn the basics of neural networks and backpropagation, one of the most important algorithms for the modern world.
youtube.com
September 24, 2026 at 3:33 AM
My go to would be 3blue1brown. youtube.com/playlist?lis...
so-called "forward thinkers" during backpropagation
June 24, 2026 at 8:10 PM
so-called "forward thinkers" during backpropagation
quote me one such person who actually understands what backpropagation is
April 13, 2025 at 4:33 AM
quote me one such person who actually understands what backpropagation is
backpropagation and gradient descent are old ideas, too, arguably dating back to the 1960s and 1970s. but academia and industry were hypnotized by symbolic logic, expert systems, etc, despite their obvious and apparent limitations
March 16, 2026 at 1:21 AM
backpropagation and gradient descent are old ideas, too, arguably dating back to the 1960s and 1970s. but academia and industry were hypnotized by symbolic logic, expert systems, etc, despite their obvious and apparent limitations
there's something itching in my head about spooky signal-passing on a randomized substrate using the only separable signal being a plausibly biological alternative to backpropagation but it isn't actually fitting together
for any given model, the distance between concepts with very low co-occurrence within the dataset will be determined by the (randomized) initialization weights, meaning that the coactivation of "random" numbers with the concept "owl" will covary.
This result is crazy. You finetune a teacher model to love owls. Then you have the teacher model generate a string of numbers, and finetune the student model on those numbers. The student model will then also love owls.
alignment.anthropic.com/2025/sublimi...
alignment.anthropic.com/2025/sublimi...
July 22, 2025 at 7:56 PM
there's something itching in my head about spooky signal-passing on a randomized substrate using the only separable signal being a plausibly biological alternative to backpropagation but it isn't actually fitting together
Dynamic programming alternatives to dynamic programming for optimal control. Replace your Bellman equation with backpropagation. www.argmin.net/p/twirling-t...
Twirling Towards Freedom
Avoiding dynamic programming with dynamic programming
www.argmin.net
November 21, 2024 at 3:37 PM
Dynamic programming alternatives to dynamic programming for optimal control. Replace your Bellman equation with backpropagation. www.argmin.net/p/twirling-t...
Fascinating. Change was coming - with backpropagation.
Two influential scientists considered multilayer perceptron networks (i.e., digital neural networks) in 1969 and wrote: "Our intuitive judgment that the extension Is sterile".
This was the greatest misjudgment in the history of modern technology. Who wrote it?
quoteinvestigator.com/2026/07/24/p...
This was the greatest misjudgment in the history of modern technology. Who wrote it?
quoteinvestigator.com/2026/07/24/p...
July 24, 2026 at 1:25 PM
Fascinating. Change was coming - with backpropagation.
fucking like ten people explain how backpropagation works in the comments jesus christ that's not the level of abstraction she cares about at all you guys why
June 9, 2025 at 3:48 AM
fucking like ten people explain how backpropagation works in the comments jesus christ that's not the level of abstraction she cares about at all you guys why
During backpropagation through the Language Model (LM) head, approximately 95% to 99% of the logit-gradient norm resides in directions that are effectively projected away.
March 16, 2026 at 5:07 AM
During backpropagation through the Language Model (LM) head, approximately 95% to 99% of the logit-gradient norm resides in directions that are effectively projected away.
Researchers developed a 100x faster AI training method, reducing energy use by replacing backpropagation with a probabilistic approach.
Backtracking on Backpropagation
Researchers developed a 100x faster AI training method, reducing energy use by replacing backpropagation with a probabilistic approach.
www.hackster.io
March 9, 2025 at 12:27 PM
Researchers developed a 100x faster AI training method, reducing energy use by replacing backpropagation with a probabilistic approach.
Illustration for my ♀️ #durgetash fic "Backpropagation"
Read: archiveofourown.org/works/78350211 (an English translation, the link to the original is in the notes)
Read: archiveofourown.org/works/78350211 (an English translation, the link to the original is in the notes)
January 26, 2026 at 12:20 PM
Illustration for my ♀️ #durgetash fic "Backpropagation"
Read: archiveofourown.org/works/78350211 (an English translation, the link to the original is in the notes)
Read: archiveofourown.org/works/78350211 (an English translation, the link to the original is in the notes)