--> i need to make unprincipled changes to a hyperparameter (the seed is a hyperparameter)
An illustrated guide to never learning anything
December 25, 2024 at 12:55 AM
--> i need to make unprincipled changes to a hyperparameter (the seed is a hyperparameter)
tuning a hyperparameter labeled "racism" on my model and looking at the federal procurement system like a contestant on the Price is Right.
July 24, 2025 at 4:57 AM
tuning a hyperparameter labeled "racism" on my model and looking at the federal procurement system like a contestant on the Price is Right.
Please, please put your hyperparameter tuning procedure into the paper. For your method and the baselines
July 3, 2025 at 4:13 AM
Please, please put your hyperparameter tuning procedure into the paper. For your method and the baselines
god, you people will put a hyperparameter on just about anything, won’t you?
February 16, 2026 at 7:19 AM
god, you people will put a hyperparameter on just about anything, won’t you?
The paper claims that optimization theory for adaptive methods actually predicts most of what we know about hyperparameter scaling in LLM pretraining.
Paper: Deriving Hyperparameter Scaling Laws via Modern Optimization Theory ( arxiv.org/abs/2603.15958 )
Paper: Deriving Hyperparameter Scaling Laws via Modern Optimization Theory ( arxiv.org/abs/2603.15958 )
March 24, 2026 at 4:47 AM
The paper claims that optimization theory for adaptive methods actually predicts most of what we know about hyperparameter scaling in LLM pretraining.
Paper: Deriving Hyperparameter Scaling Laws via Modern Optimization Theory ( arxiv.org/abs/2603.15958 )
Paper: Deriving Hyperparameter Scaling Laws via Modern Optimization Theory ( arxiv.org/abs/2603.15958 )
My computer is running a hyperparameter sweep. My room is getting Very Warm
December 18, 2025 at 2:25 AM
My computer is running a hyperparameter sweep. My room is getting Very Warm
warming by house by gpu and a nice thicc hyperparameter search this weekend
March 1, 2025 at 3:30 AM
warming by house by gpu and a nice thicc hyperparameter search this weekend
And for the love of the great Flying Spaghetti Monster DO NOT TUNE ON YOUR SEED. Just don't.
November 23, 2024 at 5:34 PM
And for the love of the great Flying Spaghetti Monster DO NOT TUNE ON YOUR SEED. Just don't.
just kind of weird that you can do a hyperparameter ablation and get totally different types of artifacts caused by nothing more than hyperparameteriation
July 20, 2026 at 6:25 AM
just kind of weird that you can do a hyperparameter ablation and get totally different types of artifacts caused by nothing more than hyperparameteriation
New NanoGPT training speed record: 3.28 FineWeb val loss in 4.66 minutes
Previous record: 5.03 minutes
Changelog:
- FlexAttention blocksize warmup
- hyperparameter tweaks
Previous record: 5.03 minutes
Changelog:
- FlexAttention blocksize warmup
- hyperparameter tweaks
November 25, 2024 at 1:53 AM
New NanoGPT training speed record: 3.28 FineWeb val loss in 4.66 minutes
Previous record: 5.03 minutes
Changelog:
- FlexAttention blocksize warmup
- hyperparameter tweaks
Previous record: 5.03 minutes
Changelog:
- FlexAttention blocksize warmup
- hyperparameter tweaks
i use AI for only 3 THINGS:
3.0
Temperature
Hyperparameter
Insanity
Naming
God’s
Secrets
3.0
Temperature
Hyperparameter
Insanity
Naming
God’s
Secrets
January 28, 2025 at 9:03 PM
i use AI for only 3 THINGS:
3.0
Temperature
Hyperparameter
Insanity
Naming
God’s
Secrets
3.0
Temperature
Hyperparameter
Insanity
Naming
God’s
Secrets
accidentally forgetting to commit my hyperparameter change because I've sleep four hours in the past week and costing my company my monthly salary in compute credits
October 20, 2025 at 10:01 PM
accidentally forgetting to commit my hyperparameter change because I've sleep four hours in the past week and costing my company my monthly salary in compute credits
March 28, 2025 at 8:07 PM
My take: Engineering *is* important. Ideas are cheap. Novelty is overrated. "Hyperparameter tuning" is a cynically dismissive term for something that is in fact the real, difficult problem.
Sometimes I’ll see a paper that implements an idea I’ve had but implements it poorly for engineering reasons. But then it’s hard to write a follow up paper because it’s not “novel” anymore
October 9, 2025 at 2:13 PM
My take: Engineering *is* important. Ideas are cheap. Novelty is overrated. "Hyperparameter tuning" is a cynically dismissive term for something that is in fact the real, difficult problem.
do the neural net hyperparameter fractals count as ai art
February 24, 2026 at 11:08 PM
do the neural net hyperparameter fractals count as ai art
slowly slipping away from making graphs for the sake of understanding
towards making graphs for the sake of how cool it is to have lots of graphs
towards making graphs for the sake of how cool it is to have lots of graphs
July 27, 2026 at 1:03 AM
slowly slipping away from making graphs for the sake of understanding
towards making graphs for the sake of how cool it is to have lots of graphs
towards making graphs for the sake of how cool it is to have lots of graphs
Elon, yelling to his Grok team "intensify the Fetterman hyperparameter... more... more..! MORE!!!"
July 6, 2025 at 1:40 AM
Elon, yelling to his Grok team "intensify the Fetterman hyperparameter... more... more..! MORE!!!"
New blog post: Automatic Hyperparameter Tuning in Practice.
Part I was about the challenges and main components of hyperparameter tuning (samplers, …). Part II is about the practical application of this technique to RL with the @optuna.bsky.social and SB3 libraries.
araffin.github.io/post/optuna/
Part I was about the challenges and main components of hyperparameter tuning (samplers, …). Part II is about the practical application of this technique to RL with the @optuna.bsky.social and SB3 libraries.
araffin.github.io/post/optuna/
Automatic Hyperparameter Tuning - In Practice (Part 2) | Antonin Raffin | Homepage
This is the second (and last) post on automatic hyperparameter optimization. In the first part, I introduced the challenges and main components of hyperparameter tuning (samplers, pruners, objective f...
araffin.github.io
April 28, 2025 at 9:50 AM
New blog post: Automatic Hyperparameter Tuning in Practice.
Part I was about the challenges and main components of hyperparameter tuning (samplers, …). Part II is about the practical application of this technique to RL with the @optuna.bsky.social and SB3 libraries.
araffin.github.io/post/optuna/
Part I was about the challenges and main components of hyperparameter tuning (samplers, …). Part II is about the practical application of this technique to RL with the @optuna.bsky.social and SB3 libraries.
araffin.github.io/post/optuna/
Doing hyperparameter sweeps has a very “throwing spaghetti at the wall” feel to it
November 20, 2024 at 11:59 PM
Doing hyperparameter sweeps has a very “throwing spaghetti at the wall” feel to it
*"hyperparameter"
January 29, 2026 at 11:21 PM
*"hyperparameter"
how are chinese labs cutting their dependence on NVIDIA? like this:
run experiments on tiny models, transfer hyperparameters (result of experiments) to a far larger model for the yolo run
bsky.app/profile/timk...
run experiments on tiny models, transfer hyperparameters (result of experiments) to a far larger model for the yolo run
bsky.app/profile/timk...
August 31, 2025 at 12:01 PM
how are chinese labs cutting their dependence on NVIDIA? like this:
run experiments on tiny models, transfer hyperparameters (result of experiments) to a far larger model for the yolo run
bsky.app/profile/timk...
run experiments on tiny models, transfer hyperparameters (result of experiments) to a far larger model for the yolo run
bsky.app/profile/timk...
Occasionally you'll run into an ML empirical nihilist, someone who claims that no result is valid because alternative hyperparameter settings could maybe lead to different results. How do you argue with this person?
March 31, 2025 at 3:15 AM
Occasionally you'll run into an ML empirical nihilist, someone who claims that no result is valid because alternative hyperparameter settings could maybe lead to different results. How do you argue with this person?
I've just refactored the automatic hyperparameter tuning of the RL Zoo. You can now use the @optuna.bsky.social journal backend and easily train an agent with tuned hyperparameters using the new `--trial-id` argument.
github.com/DLR-RM/rl-ba...
github.com/DLR-RM/rl-ba...
GitHub - DLR-RM/rl-baselines3-zoo: A training framework for Stable Baselines3 reinforcement learning agents, with hyperparameter optimization and pre-trained agents included.
A training framework for Stable Baselines3 reinforcement learning agents, with hyperparameter optimization and pre-trained agents included. - DLR-RM/rl-baselines3-zoo
github.com
March 19, 2025 at 6:50 PM
I've just refactored the automatic hyperparameter tuning of the RL Zoo. You can now use the @optuna.bsky.social journal backend and easily train an agent with tuned hyperparameters using the new `--trial-id` argument.
github.com/DLR-RM/rl-ba...
github.com/DLR-RM/rl-ba...
I recently gave a talk on hyperparameter tuning in #tidymodels at Posit Day for University of Wisconsin. Nothing new in the slides but wanted to share regardless
emilhvitfeldt.com/talk/2025-11...
emilhvitfeldt.com/talk/2025-11...
Hyperparameter Fine-Tuning in tidymodels
A view into the different kinds of hyperparameter tuning methods out there
emilhvitfeldt.com
November 17, 2025 at 6:25 PM
I recently gave a talk on hyperparameter tuning in #tidymodels at Posit Day for University of Wisconsin. Nothing new in the slides but wanted to share regardless
emilhvitfeldt.com/talk/2025-11...
emilhvitfeldt.com/talk/2025-11...