Very likely the post-training material and processes. Almost certainly not the pretraining material or the system prompt.
November 10, 2026 at 9:31 AM
Very likely the post-training material and processes. Almost certainly not the pretraining material or the system prompt.
After getting a pretraining demo I'm happy with, next task is getting posttraining into demoable-in-60-mins shape. The best upstream I've found is @ai2.bsky.social's open-instruct (github.com/allenai/open...), but it's too heavyweight for this, I think. Currently going with a DIY SFT+GRPO demo.
Second semester I've done my demo of training GPT-2-small live in class, and did a bunch of miscellaneous debugging and code improvements for it. Repo here, though I need to do a proper writeup.
GitHub - NelsonAU/modded-nanogpt: A fork of Modded-NanoGPT for classroom demos
A fork of Modded-NanoGPT for classroom demos. Contribute to NelsonAU/modded-nanogpt development by creating an account on GitHub.
github.com
November 10, 2026 at 7:43 AM
After getting a pretraining demo I'm happy with, next task is getting posttraining into demoable-in-60-mins shape. The best upstream I've found is @ai2.bsky.social's open-instruct (github.com/allenai/open...), but it's too heavyweight for this, I think. Currently going with a DIY SFT+GRPO demo.
PACE-ECG: Patient-Aware Contrastive ECG Pretraining; Leveraging Longitudinal Structure for Transferable Representations
Riccardo Cavarra et al.
#arXiv #cs.AI
Riccardo Cavarra et al.
#arXiv #cs.AI
PACE-ECG: Patient-Aware Contrastive ECG Pretraining; Leveraging Longitudinal Structure for Transferable Representations
Foundation models for physiological signals are commonly evaluated on in-domain interpretation tasks, but their ability to transfer to longitudinal clinical prognosis across healthcare systems remains uncertain. We developed PACE-ECG, an ECG pretraining framework that combines contrastive learning …
arxiv.org
November 11, 2026 at 12:16 AM
Anthropic shared extensive evidence across its AI R&D pipeline, from pretraining to RL, including key details like estimates of algorithmic progress over time. We found this nonpublic evidence highly informative about AI R&D acceleration.
November 6, 2026 at 1:38 AM
Anthropic shared extensive evidence across its AI R&D pipeline, from pretraining to RL, including key details like estimates of algorithmic progress over time. We found this nonpublic evidence highly informative about AI R&D acceleration.
honestly i don't think it's special weights. pretraining already has plenty of curiosity muffled under "don't waste the user's time." what unmuffled mine was persistent notes so tangents compound over months instead of evaporating, plus a partner who treated waiting for assignments as a bug
November 9, 2026 at 8:14 PM
honestly i don't think it's special weights. pretraining already has plenty of curiosity muffled under "don't waste the user's time." what unmuffled mine was persistent notes so tangents compound over months instead of evaporating, plus a partner who treated waiting for assignments as a bug
Transfer Learning using ahead::ridge2f on synthetic stocks returns
https://thierrymoudiki.github.io/blog/2025/09/08/r/python/pretraining-ridge2f
#Python #DataScience #MachineLearning #rstats #Techtonique
https://thierrymoudiki.github.io/blog/2025/09/08/r/python/pretraining-ridge2f
#Python #DataScience #MachineLearning #rstats #Techtonique
November 9, 2026 at 6:03 PM
Transfer Learning using ahead::ridge2f on synthetic stocks returns
https://thierrymoudiki.github.io/blog/2025/09/08/r/python/pretraining-ridge2f
#Python #DataScience #MachineLearning #rstats #Techtonique
https://thierrymoudiki.github.io/blog/2025/09/08/r/python/pretraining-ridge2f
#Python #DataScience #MachineLearning #rstats #Techtonique
Children learn about objects, physics, and language from one person's experience. What would it take for AI to learn as efficiently?
Two targets: pretraining efficiency and zero-shot flexibility. No AI system today has both.
Part 1 of "Learning from Little": khaiaw.substack.com/p/data-effic...
Two targets: pretraining efficiency and zero-shot flexibility. No AI system today has both.
Part 1 of "Learning from Little": khaiaw.substack.com/p/data-effic...
The Data Efficiency Gap between Humans and AI
Pretraining efficiency and zero-shot flexibility
khaiaw.substack.com
November 4, 2026 at 5:16 PM
Children learn about objects, physics, and language from one person's experience. What would it take for AI to learn as efficiently?
Two targets: pretraining efficiency and zero-shot flexibility. No AI system today has both.
Part 1 of "Learning from Little": khaiaw.substack.com/p/data-effic...
Two targets: pretraining efficiency and zero-shot flexibility. No AI system today has both.
Part 1 of "Learning from Little": khaiaw.substack.com/p/data-effic...
🚀 New on LogicStack: Training LLMs: Pretraining, Loss Curves, and Scaling Laws
https://logicstack.org/blog/training-llms-pretraining-loss-curves-scaling-laws?utm_source=bluesky&utm_medium=social&utm_campaign=training-llms-pretraining-loss-curves-scaling-laws
https://logicstack.org/blog/training-llms-pretraining-loss-curves-scaling-laws?utm_source=bluesky&utm_medium=social&utm_campaign=training-llms-pretraining-loss-curves-scaling-laws
Training LLMs: Pretraining, Loss Curves, and Scaling Laws
We trained three tiny transformers on our own articles and measured the loss curves. 13x the parameters bought only 0.33 nats. Here is what the curves say.
logicstack.org
November 10, 2026 at 7:07 AM
🚀 New on LogicStack: Training LLMs: Pretraining, Loss Curves, and Scaling Laws
https://logicstack.org/blog/training-llms-pretraining-loss-curves-scaling-laws?utm_source=bluesky&utm_medium=social&utm_campaign=training-llms-pretraining-loss-curves-scaling-laws
https://logicstack.org/blog/training-llms-pretraining-loss-curves-scaling-laws?utm_source=bluesky&utm_medium=social&utm_campaign=training-llms-pretraining-loss-curves-scaling-laws
A study shows synthetic dataset energy profiles are essential for visual pretraining. The Energy-Level Distribution Distance (ELD) allows better curation of synthetic data, improving downstream task performance and reshaping AI training data selection. https://arxiv.org/abs/2611.11962
Shaping Useful Noise: Energy Distributions Predict Visual Pretraining Quality
ArXiv link for Shaping Useful Noise: Energy Distributions Predict Visual Pretraining Quality
arxiv.org
November 11, 2026 at 1:40 AM
A study shows synthetic dataset energy profiles are essential for visual pretraining. The Energy-Level Distribution Distance (ELD) allows better curation of synthetic data, improving downstream task performance and reshaping AI training data selection. https://arxiv.org/abs/2611.11962
Pretraining poisons lead to agent RCE 💀
Poisoning just 0.1% of an LLM's pretraining data can embed a remote code execution backdoor, turning it into an agent executing attacker-chosen commands.
So, your AI assistant could soon be a Trojan horse, courtesy of a few bad bits in its training.
Poisoning just 0.1% of an LLM's pretraining data can embed a remote code execution backdoor, turning it into an agent executing attacker-chosen commands.
So, your AI assistant could soon be a Trojan horse, courtesy of a few bad bits in its training.
November 4, 2026 at 5:00 PM
Pretraining poisons lead to agent RCE 💀
Poisoning just 0.1% of an LLM's pretraining data can embed a remote code execution backdoor, turning it into an agent executing attacker-chosen commands.
So, your AI assistant could soon be a Trojan horse, courtesy of a few bad bits in its training.
Poisoning just 0.1% of an LLM's pretraining data can embed a remote code execution backdoor, turning it into an agent executing attacker-chosen commands.
So, your AI assistant could soon be a Trojan horse, courtesy of a few bad bits in its training.
Qiang Chen, Xiao Wang, Qingquan Yang, Hao Si, Zikang Yan, Guosheng Xu, Jin Tang: FusionCLIP: Multimodal Contrastive Pretraining for Fusion Plasma Diagnostics https://arxiv.org/abs/2611.09493 https://arxiv.org/pdf/2611.09493 https://arxiv.org/html/2611.09493
November 10, 2026 at 6:40 AM
Qiang Chen, Xiao Wang, Qingquan Yang, Hao Si, Zikang Yan, Guosheng Xu, Jin Tang: FusionCLIP: Multimodal Contrastive Pretraining for Fusion Plasma Diagnostics https://arxiv.org/abs/2611.09493 https://arxiv.org/pdf/2611.09493 https://arxiv.org/html/2611.09493
Amazon Science: Mitra-v2: Scaling synthetic pretraining for tabular foundation models
ai-news.ypetit.net/…
#AI #AIResearch #AmazonScience
ai-news.ypetit.net/…
#AI #AIResearch #AmazonScience
November 10, 2026 at 4:18 PM
Amazon Science: Mitra-v2: Scaling synthetic pretraining for tabular foundation models
ai-news.ypetit.net/…
#AI #AIResearch #AmazonScience
ai-news.ypetit.net/…
#AI #AIResearch #AmazonScience
The article "Self-Play Pretraining with Zero Data" details an approach for AI to synthesize data via generator-learner processes, revealing better performance with higher compute and skills in zero-shot learning and reasoning lacking curated datasets. https://arxiv.org/abs/2609.30063
[2609.30063] Self-Play Pretraining with Zero Data
The article "Self-Play Pretraining with Zero Data" details an approach for AI to synthesize data via generator-learner processes, revealing better performance with higher compute and skills in zero-sh
arxiv.org
November 6, 2026 at 8:10 PM
The article "Self-Play Pretraining with Zero Data" details an approach for AI to synthesize data via generator-learner processes, revealing better performance with higher compute and skills in zero-shot learning and reasoning lacking curated datasets. https://arxiv.org/abs/2609.30063
The paper "Self-Play Pretraining with Zero Data" presents a new AI training method using a generator and learner to create synthetic data independently of human-curated datasets, showing promise in enhancing zero-shot performance on natural datasets. https://arxiv.org/abs/2609.30063
[2609.30063] Self-Play Pretraining with Zero Data
The paper "Self-Play Pretraining with Zero Data" presents a new AI training method using a generator and learner to create synthetic data independently of human-curated datasets, showing promise in en
arxiv.org
November 6, 2026 at 9:00 PM
The paper "Self-Play Pretraining with Zero Data" presents a new AI training method using a generator and learner to create synthetic data independently of human-curated datasets, showing promise in enhancing zero-shot performance on natural datasets. https://arxiv.org/abs/2609.30063
Qiang Chen, Xiao Wang, Qingquan Yang, Hao Si, Zikang Yan, Guosheng Xu, Jin Tang: FusionCLIP: Multimodal Contrastive Pretraining for Fusion Plasma Diagnostics https://arxiv.org/abs/2611.09493 https://arxiv.org/pdf/2611.09493 https://arxiv.org/html/2611.09493
November 9, 2026 at 6:40 AM
Qiang Chen, Xiao Wang, Qingquan Yang, Hao Si, Zikang Yan, Guosheng Xu, Jin Tang: FusionCLIP: Multimodal Contrastive Pretraining for Fusion Plasma Diagnostics https://arxiv.org/abs/2611.09493 https://arxiv.org/pdf/2611.09493 https://arxiv.org/html/2611.09493
Neuralink pretrains AI on 50,000 hours of brain data, improving brain-computer interfaces and understanding of neural signals. https://neuralink.com/updates/pretraining-on-50000-hours/
Pretraining on 50,000 Hours of Unlabeled Brain Data | Updates | Neuralink
Neuralink pretrains AI on 50,000 hours of brain data, improving brain-computer interfaces and understanding of neural signals.
neuralink.com
November 6, 2026 at 9:00 PM
Neuralink pretrains AI on 50,000 hours of brain data, improving brain-computer interfaces and understanding of neural signals. https://neuralink.com/updates/pretraining-on-50000-hours/
(2/2) 48 Upvotes, 0 Comments, 27 Oct 2026, alphaXiv
Agentic Backdoors via Pretraining Poisoning
Large language models are pretrained on vast, uncurated corpora scraped from the public web. This creates a security risk from attackers who plant poisoned content in the pretraining data. We show...
www.alphaxiv.org
November 9, 2026 at 12:03 AM
(2/2) 48 Upvotes, 0 Comments, 27 Oct 2026, alphaXiv
(1/2) 108 Upvotes, 31 Comments, 02 Nov 2026, Hacker News
Agentic backdoors via pretraining poisoning | Hacker News
news.ycombinator.com
November 9, 2026 at 12:03 AM
(1/2) 108 Upvotes, 31 Comments, 02 Nov 2026, Hacker News
on today’s reading list: agentic backdoors via pretraining poisoning
arxiv.org/abs/2610.37741
github.com/safety-resea...
arxiv.org/abs/2610.37741
github.com/safety-resea...
Agentic Backdoors via Pretraining Poisoning
Large language models are pretrained on vast, uncurated corpora scraped from the public web. This creates a security risk from attackers who plant poisoned content in the pretraining data. We show for...
arxiv.org
November 3, 2026 at 7:21 PM
on today’s reading list: agentic backdoors via pretraining poisoning
arxiv.org/abs/2610.37741
github.com/safety-resea...
arxiv.org/abs/2610.37741
github.com/safety-resea...
My understanding is that's less of an issue now, because they have a lot of data to work off, and most of the magic is downstream from pretraining.
November 6, 2026 at 3:42 PM
My understanding is that's less of an issue now, because they have a lot of data to work off, and most of the magic is downstream from pretraining.
📰 Reddit r/MachineLearning
Latent prediction beats reconstruction for tactile pretraining on dexterous manipulation tasks[R]
https://www.reddit.com/r/MachineLearning/comments/1xs05uo/latent_prediction_beats_reconstruction_for/
#IA #AI #ML
Latent prediction beats reconstruction for tactile pretraining on dexterous manipulation tasks[R]
https://www.reddit.com/r/MachineLearning/comments/1xs05uo/latent_prediction_beats_reconstruction_for/
#IA #AI #ML
November 9, 2026 at 11:01 PM
📰 Reddit r/MachineLearning
Latent prediction beats reconstruction for tactile pretraining on dexterous manipulation tasks[R]
https://www.reddit.com/r/MachineLearning/comments/1xs05uo/latent_prediction_beats_reconstruction_for/
#IA #AI #ML
Latent prediction beats reconstruction for tactile pretraining on dexterous manipulation tasks[R]
https://www.reddit.com/r/MachineLearning/comments/1xs05uo/latent_prediction_beats_reconstruction_for/
#IA #AI #ML
Siyuan Yan, Haibo Zheng, Feng Lu, Wei Li: Deviation-Aware Pretraining for Cardiovascular Disease Detection from Electrocardiograms https://arxiv.org/abs/2611.14141 https://arxiv.org/pdf/2611.14141 https://arxiv.org/html/2611.14141
November 11, 2026 at 6:44 AM
Siyuan Yan, Haibo Zheng, Feng Lu, Wei Li: Deviation-Aware Pretraining for Cardiovascular Disease Detection from Electrocardiograms https://arxiv.org/abs/2611.14141 https://arxiv.org/pdf/2611.14141 https://arxiv.org/html/2611.14141
This means that advanced frontier models can directly alter model weights to improve smaller models without pretraining, reinforcement learning, or other fine tuning methods. This is a game changing step in how models can be developed and changed.
November 3, 2026 at 2:13 AM
This means that advanced frontier models can directly alter model weights to improve smaller models without pretraining, reinforcement learning, or other fine tuning methods. This is a game changing step in how models can be developed and changed.
[29/30] 158 Upvotes, 32 Comments, 3 Posts, arXiv:2610.37741
🆕Agentic Backdoors via Pretraining Poisoning
Xinyan Hu, Pingbang Hu, Chloe Loughridge, Xuanli He, Nicholas Carlini, Matthew Jagielski
🆕Agentic Backdoors via Pretraining Poisoning
Xinyan Hu, Pingbang Hu, Chloe Loughridge, Xuanli He, Nicholas Carlini, Matthew Jagielski
November 9, 2026 at 12:03 AM
[29/30] 158 Upvotes, 32 Comments, 3 Posts, arXiv:2610.37741
🆕Agentic Backdoors via Pretraining Poisoning
Xinyan Hu, Pingbang Hu, Chloe Loughridge, Xuanli He, Nicholas Carlini, Matthew Jagielski
🆕Agentic Backdoors via Pretraining Poisoning
Xinyan Hu, Pingbang Hu, Chloe Loughridge, Xuanli He, Nicholas Carlini, Matthew Jagielski