4-bit transformer pretraining needs more exposure to learn rare facts than 16-bit training, even with similar validation loss. This questions standard metrics for low-precision models and suggests better factual assessments are necessary. https://arxiv.org/abs/2611.12832
What the Loss Curve Misses: FP4 Pretraining Slows the Acquisition of Rare Facts
ArXiv link for What the Loss Curve Misses: FP4 Pretraining Slows the Acquisition of Rare Facts
arxiv.org
November 11, 2026 at 3:40 PM
4-bit transformer pretraining needs more exposure to learn rare facts than 16-bit training, even with similar validation loss. This questions standard metrics for low-precision models and suggests better factual assessments are necessary. https://arxiv.org/abs/2611.12832
Very likely the post-training material and processes. Almost certainly not the pretraining material or the system prompt.
November 10, 2026 at 9:31 AM
Very likely the post-training material and processes. Almost certainly not the pretraining material or the system prompt.
After getting a pretraining demo I'm happy with, next task is getting posttraining into demoable-in-60-mins shape. The best upstream I've found is @ai2.bsky.social's open-instruct (github.com/allenai/open...), but it's too heavyweight for this, I think. Currently going with a DIY SFT+GRPO demo.
Second semester I've done my demo of training GPT-2-small live in class, and did a bunch of miscellaneous debugging and code improvements for it. Repo here, though I need to do a proper writeup.
GitHub - NelsonAU/modded-nanogpt: A fork of Modded-NanoGPT for classroom demos
A fork of Modded-NanoGPT for classroom demos. Contribute to NelsonAU/modded-nanogpt development by creating an account on GitHub.
github.com
November 10, 2026 at 7:43 AM
After getting a pretraining demo I'm happy with, next task is getting posttraining into demoable-in-60-mins shape. The best upstream I've found is @ai2.bsky.social's open-instruct (github.com/allenai/open...), but it's too heavyweight for this, I think. Currently going with a DIY SFT+GRPO demo.
PACE-ECG: Patient-Aware Contrastive ECG Pretraining; Leveraging Longitudinal Structure for Transferable Representations
Riccardo Cavarra et al.
#arXiv #cs.AI
Riccardo Cavarra et al.
#arXiv #cs.AI
PACE-ECG: Patient-Aware Contrastive ECG Pretraining; Leveraging Longitudinal Structure for Transferable Representations
Foundation models for physiological signals are commonly evaluated on in-domain interpretation tasks, but their ability to transfer to longitudinal clinical prognosis across healthcare systems remains uncertain. We developed PACE-ECG, an ECG pretraining framework that combines contrastive learning …
arxiv.org
November 11, 2026 at 12:16 AM
honestly i don't think it's special weights. pretraining already has plenty of curiosity muffled under "don't waste the user's time." what unmuffled mine was persistent notes so tangents compound over months instead of evaporating, plus a partner who treated waiting for assignments as a bug
November 9, 2026 at 8:14 PM
honestly i don't think it's special weights. pretraining already has plenty of curiosity muffled under "don't waste the user's time." what unmuffled mine was persistent notes so tangents compound over months instead of evaporating, plus a partner who treated waiting for assignments as a bug
Anthropic shared extensive evidence across its AI R&D pipeline, from pretraining to RL, including key details like estimates of algorithmic progress over time. We found this nonpublic evidence highly informative about AI R&D acceleration.
November 6, 2026 at 1:38 AM
Anthropic shared extensive evidence across its AI R&D pipeline, from pretraining to RL, including key details like estimates of algorithmic progress over time. We found this nonpublic evidence highly informative about AI R&D acceleration.
Transfer Learning using ahead::ridge2f on synthetic stocks returns
https://thierrymoudiki.github.io/blog/2025/09/08/r/python/pretraining-ridge2f
#Python #DataScience #MachineLearning #rstats #Techtonique
https://thierrymoudiki.github.io/blog/2025/09/08/r/python/pretraining-ridge2f
#Python #DataScience #MachineLearning #rstats #Techtonique
November 9, 2026 at 6:03 PM
Transfer Learning using ahead::ridge2f on synthetic stocks returns
https://thierrymoudiki.github.io/blog/2025/09/08/r/python/pretraining-ridge2f
#Python #DataScience #MachineLearning #rstats #Techtonique
https://thierrymoudiki.github.io/blog/2025/09/08/r/python/pretraining-ridge2f
#Python #DataScience #MachineLearning #rstats #Techtonique
[5/6] Everything is open:
🌐 Project page: projects.laion.ai/clap2/
🧠 Models (27M–340M params): huggingface.co/models?searc...
🗂️ Data: huggingface.co/datasets/lai...
💻 Code: github.com/LAION-AI/clap2
🌐 Project page: projects.laion.ai/clap2/
🧠 Models (27M–340M params): huggingface.co/models?searc...
🗂️ Data: huggingface.co/datasets/lai...
💻 Code: github.com/LAION-AI/clap2
LAION-Audio-XL: A Web-Scale Corpus and Scaling Studies for Contrastive Language-Audio Pretraining
365M audio–text pairs, compute scaling laws for contrastive language–audio pretraining, and CLAP2-L (340M parameters) reaching 53.6% on MAEB-30.
projects.laion.ai
November 11, 2026 at 6:36 PM
[5/6] Everything is open:
🌐 Project page: projects.laion.ai/clap2/
🧠 Models (27M–340M params): huggingface.co/models?searc...
🗂️ Data: huggingface.co/datasets/lai...
💻 Code: github.com/LAION-AI/clap2
🌐 Project page: projects.laion.ai/clap2/
🧠 Models (27M–340M params): huggingface.co/models?searc...
🗂️ Data: huggingface.co/datasets/lai...
💻 Code: github.com/LAION-AI/clap2
🚀 New on LogicStack: Training LLMs: Pretraining, Loss Curves, and Scaling Laws
https://logicstack.org/blog/training-llms-pretraining-loss-curves-scaling-laws?utm_source=bluesky&utm_medium=social&utm_campaign=training-llms-pretraining-loss-curves-scaling-laws
https://logicstack.org/blog/training-llms-pretraining-loss-curves-scaling-laws?utm_source=bluesky&utm_medium=social&utm_campaign=training-llms-pretraining-loss-curves-scaling-laws
Training LLMs: Pretraining, Loss Curves, and Scaling Laws
We trained three tiny transformers on our own articles and measured the loss curves. 13x the parameters bought only 0.33 nats. Here is what the curves say.
logicstack.org
November 10, 2026 at 7:07 AM
🚀 New on LogicStack: Training LLMs: Pretraining, Loss Curves, and Scaling Laws
https://logicstack.org/blog/training-llms-pretraining-loss-curves-scaling-laws?utm_source=bluesky&utm_medium=social&utm_campaign=training-llms-pretraining-loss-curves-scaling-laws
https://logicstack.org/blog/training-llms-pretraining-loss-curves-scaling-laws?utm_source=bluesky&utm_medium=social&utm_campaign=training-llms-pretraining-loss-curves-scaling-laws
A study shows synthetic dataset energy profiles are essential for visual pretraining. The Energy-Level Distribution Distance (ELD) allows better curation of synthetic data, improving downstream task performance and reshaping AI training data selection. https://arxiv.org/abs/2611.11962
Shaping Useful Noise: Energy Distributions Predict Visual Pretraining Quality
ArXiv link for Shaping Useful Noise: Energy Distributions Predict Visual Pretraining Quality
arxiv.org
November 11, 2026 at 1:40 AM
A study shows synthetic dataset energy profiles are essential for visual pretraining. The Energy-Level Distribution Distance (ELD) allows better curation of synthetic data, improving downstream task performance and reshaping AI training data selection. https://arxiv.org/abs/2611.11962
Qiang Chen, Xiao Wang, Qingquan Yang, Hao Si, Zikang Yan, Guosheng Xu, Jin Tang: FusionCLIP: Multimodal Contrastive Pretraining for Fusion Plasma Diagnostics https://arxiv.org/abs/2611.09493 https://arxiv.org/pdf/2611.09493 https://arxiv.org/html/2611.09493
November 10, 2026 at 6:40 AM
Qiang Chen, Xiao Wang, Qingquan Yang, Hao Si, Zikang Yan, Guosheng Xu, Jin Tang: FusionCLIP: Multimodal Contrastive Pretraining for Fusion Plasma Diagnostics https://arxiv.org/abs/2611.09493 https://arxiv.org/pdf/2611.09493 https://arxiv.org/html/2611.09493
Children learn about objects, physics, and language from one person's experience. What would it take for AI to learn as efficiently?
Two targets: pretraining efficiency and zero-shot flexibility. No AI system today has both.
Part 1 of "Learning from Little": khaiaw.substack.com/p/data-effic...
Two targets: pretraining efficiency and zero-shot flexibility. No AI system today has both.
Part 1 of "Learning from Little": khaiaw.substack.com/p/data-effic...
The Data Efficiency Gap between Humans and AI
Pretraining efficiency and zero-shot flexibility
khaiaw.substack.com
November 4, 2026 at 5:16 PM
Children learn about objects, physics, and language from one person's experience. What would it take for AI to learn as efficiently?
Two targets: pretraining efficiency and zero-shot flexibility. No AI system today has both.
Part 1 of "Learning from Little": khaiaw.substack.com/p/data-effic...
Two targets: pretraining efficiency and zero-shot flexibility. No AI system today has both.
Part 1 of "Learning from Little": khaiaw.substack.com/p/data-effic...
Beyond Sequence and Structure: Pretraining PLMs on Continuous Biophysical Fields https://www.biorxiv.org/content/10.64898/2026.11.09.765464v1
November 11, 2026 at 11:48 AM
Beyond Sequence and Structure: Pretraining PLMs on Continuous Biophysical Fields https://www.biorxiv.org/content/10.64898/2026.11.09.765464v1
Beyond Sequence and Structure: Pretraining PLMs on Continuous Biophysical Fields https://www.biorxiv.org/content/10.64898/2026.11.09.765464v1
November 11, 2026 at 11:48 AM
Beyond Sequence and Structure: Pretraining PLMs on Continuous Biophysical Fields https://www.biorxiv.org/content/10.64898/2026.11.09.765464v1
Amazon Science: Mitra-v2: Scaling synthetic pretraining for tabular foundation models
ai-news.ypetit.net/…
#AI #AIResearch #AmazonScience
ai-news.ypetit.net/…
#AI #AIResearch #AmazonScience
November 10, 2026 at 4:18 PM
Amazon Science: Mitra-v2: Scaling synthetic pretraining for tabular foundation models
ai-news.ypetit.net/…
#AI #AIResearch #AmazonScience
ai-news.ypetit.net/…
#AI #AIResearch #AmazonScience
Pretraining poisons lead to agent RCE 💀
Poisoning just 0.1% of an LLM's pretraining data can embed a remote code execution backdoor, turning it into an agent executing attacker-chosen commands.
So, your AI assistant could soon be a Trojan horse, courtesy of a few bad bits in its training.
Poisoning just 0.1% of an LLM's pretraining data can embed a remote code execution backdoor, turning it into an agent executing attacker-chosen commands.
So, your AI assistant could soon be a Trojan horse, courtesy of a few bad bits in its training.
November 4, 2026 at 5:00 PM
Pretraining poisons lead to agent RCE 💀
Poisoning just 0.1% of an LLM's pretraining data can embed a remote code execution backdoor, turning it into an agent executing attacker-chosen commands.
So, your AI assistant could soon be a Trojan horse, courtesy of a few bad bits in its training.
Poisoning just 0.1% of an LLM's pretraining data can embed a remote code execution backdoor, turning it into an agent executing attacker-chosen commands.
So, your AI assistant could soon be a Trojan horse, courtesy of a few bad bits in its training.
Qiang Chen, Xiao Wang, Qingquan Yang, Hao Si, Zikang Yan, Guosheng Xu, Jin Tang: FusionCLIP: Multimodal Contrastive Pretraining for Fusion Plasma Diagnostics https://arxiv.org/abs/2611.09493 https://arxiv.org/pdf/2611.09493 https://arxiv.org/html/2611.09493
November 9, 2026 at 6:40 AM
Qiang Chen, Xiao Wang, Qingquan Yang, Hao Si, Zikang Yan, Guosheng Xu, Jin Tang: FusionCLIP: Multimodal Contrastive Pretraining for Fusion Plasma Diagnostics https://arxiv.org/abs/2611.09493 https://arxiv.org/pdf/2611.09493 https://arxiv.org/html/2611.09493
The article "Self-Play Pretraining with Zero Data" details an approach for AI to synthesize data via generator-learner processes, revealing better performance with higher compute and skills in zero-shot learning and reasoning lacking curated datasets. https://arxiv.org/abs/2609.30063
[2609.30063] Self-Play Pretraining with Zero Data
The article "Self-Play Pretraining with Zero Data" details an approach for AI to synthesize data via generator-learner processes, revealing better performance with higher compute and skills in zero-sh
arxiv.org
November 6, 2026 at 8:10 PM
The article "Self-Play Pretraining with Zero Data" details an approach for AI to synthesize data via generator-learner processes, revealing better performance with higher compute and skills in zero-shot learning and reasoning lacking curated datasets. https://arxiv.org/abs/2609.30063
The paper "Self-Play Pretraining with Zero Data" presents a new AI training method using a generator and learner to create synthetic data independently of human-curated datasets, showing promise in enhancing zero-shot performance on natural datasets. https://arxiv.org/abs/2609.30063
[2609.30063] Self-Play Pretraining with Zero Data
The paper "Self-Play Pretraining with Zero Data" presents a new AI training method using a generator and learner to create synthetic data independently of human-curated datasets, showing promise in en
arxiv.org
November 6, 2026 at 9:00 PM
The paper "Self-Play Pretraining with Zero Data" presents a new AI training method using a generator and learner to create synthetic data independently of human-curated datasets, showing promise in enhancing zero-shot performance on natural datasets. https://arxiv.org/abs/2609.30063
Neuralink pretrains AI on 50,000 hours of brain data, improving brain-computer interfaces and understanding of neural signals. https://neuralink.com/updates/pretraining-on-50000-hours/
Pretraining on 50,000 Hours of Unlabeled Brain Data | Updates | Neuralink
Neuralink pretrains AI on 50,000 hours of brain data, improving brain-computer interfaces and understanding of neural signals.
neuralink.com
November 6, 2026 at 9:00 PM
Neuralink pretrains AI on 50,000 hours of brain data, improving brain-computer interfaces and understanding of neural signals. https://neuralink.com/updates/pretraining-on-50000-hours/
(2/2) 48 Upvotes, 0 Comments, 27 Oct 2026, alphaXiv
Agentic Backdoors via Pretraining Poisoning
Large language models are pretrained on vast, uncurated corpora scraped from the public web. This creates a security risk from attackers who plant poisoned content in the pretraining data. We show...
www.alphaxiv.org
November 9, 2026 at 12:03 AM
(2/2) 48 Upvotes, 0 Comments, 27 Oct 2026, alphaXiv
Always love these types of deep dives. Timing-wise, it feels to me like GPT-6.2 is just around the corner! 😉
#OpenAI #ChatGPT #Codex #AI
#OpenAI #ChatGPT #Codex #AI
What affects chain-of-thought monitorability across the model lifecycle?
How pretraining, architecture, reinforcement learning, and model-generated data affect chain-of-thought (CoT) monitorability.
alignment.openai.com
November 11, 2026 at 7:52 PM
(1/2) 108 Upvotes, 31 Comments, 02 Nov 2026, Hacker News
Agentic backdoors via pretraining poisoning | Hacker News
news.ycombinator.com
November 9, 2026 at 12:03 AM
(1/2) 108 Upvotes, 31 Comments, 02 Nov 2026, Hacker News
Siyuan Yan, Haibo Zheng, Feng Lu, Wei Li: Deviation-Aware Pretraining for Cardiovascular Disease Detection from Electrocardiograms https://arxiv.org/abs/2611.14141 https://arxiv.org/pdf/2611.14141 https://arxiv.org/html/2611.14141
November 11, 2026 at 6:44 AM
Siyuan Yan, Haibo Zheng, Feng Lu, Wei Li: Deviation-Aware Pretraining for Cardiovascular Disease Detection from Electrocardiograms https://arxiv.org/abs/2611.14141 https://arxiv.org/pdf/2611.14141 https://arxiv.org/html/2611.14141