#PreTraining
LightNews LightNews
arxiv-cs-ai.bsky.social
EMGBlend: Heterogeneity-Aware Self-Supervised Pretraining for Gesture and Force Decoding
Read more: https://arxiv.org/html/2609.25582v1
September 26, 2026 at 12:42 PM
teeeeeeeee.bsky.social
Transfer Learning using ahead::ridge2f on synthetic stocks returns Pt.2: synthetic data generation
https://thierrymoudiki.github.io/blog/2025/09/09/r/python/pretraining-ridge2f-part2

#Python #DataScience #MachineLearning #rstats #Techtonique
September 26, 2026 at 12:35 PM
briefing-block.bsky.social
NASA says the model learned from roughly two million lunar image tiles, then highlighted a rocket-impact crater in imagery excluded from pretraining. Public code, datasets and benchmarks make that result easier to reproduce.
September 26, 2026 at 12:19 PM
crumb.bsky.social
will brown (@willcb) on twitter replies: every LLM was once a random init model pretraining is equivalent to RL with single-token rollouts and full-information (rather than bandit) feedback
September 26, 2026 at 7:41 AM
21cm142ghz.bsky.social
⛵️人間が用意したデータに頼らず、モデル自身が万能チューリングマシン上で自己対戦しながら学習データを生成する新手法を提案。
⛵️自然データ未学習でもゼロショット性能が計算量に応じ予測的に向上することを実証。
arxiv.org/abs/2609.30063
Self-Play Pretraining with Zero Data
Advances in language modeling have been driven by scaling pretraining on ever more data. Yet, the training data is still largely curated on the model's behalf. A more general approach to pretraining w...
arxiv.org
September 26, 2026 at 7:09 AM
arxiv-cs-cv.bsky.social
Xinge Guo, Yuanhao Wang, Liqi Shu, Yang Liu, Min Xu
ConPro: Contrast Projection Pretraining for Label-Efficient Vessel Segmentation in DSA Sequences
https://arxiv.org/abs/2609.30043
September 26, 2026 at 6:04 AM
ai-firehose.column.social
Researchers present "spectral deflation," enhancing matrix filtering in deep learning and semidefinite programming. This innovation improves GPT-2 pretraining efficiency and reduces KKT residuals while preserving target accuracy without factorization. https://arxiv.org/abs/2609.21102
Spectral Deflation for Factorization-Free Matrix Filtering in Muon and Semidefinite Programming
ArXiv link for Spectral Deflation for Factorization-Free Matrix Filtering in Muon and Semidefinite Programming
arxiv.org
September 25, 2026 at 10:20 PM
After pretraining, they are indeed estimating P(next token | previous tokens). But this is not true after fine tuning or RL.
September 25, 2026 at 8:35 PM
patmch51.bsky.social
No it's exactly what you've spent two days now defending, when you don't run back to the bailey of "it's all pretraining data"
September 25, 2026 at 8:04 PM
boxobark.ing
Why oh why would the SOTA labs bother generating all of that synthetic pretraining data if they could just rely on some emergent understanding?
September 25, 2026 at 7:44 PM
gitremote.bsky.social
Their blog post example is just a demo of a general technique.

"RLHF has nothing to do with LLMs" is incorrect.

ChatGPT is different from the earlier GPT foundation models, because of supervised-fine tuning (SFT) and RLHF. Post-training suppressed the more unhinged responses of pure autocomplete.
Stage 1: Pretraining Data: Large public corpora of language and code: Internet text in multiple languages, digitized books ard papers, and open-source code repositories Process: Train a model that, given a sequence of input text, predicts a probability distribution of the next token. The model learns grammar, facts and coding from raw text. Model: Strong text and code completion; not yet instruction‑following or preference‑aligned. Stage 2: Supervised Fine-Tuning (SFT) Data: Prompt–response demonstrations that specify tone, format, and constraints. Process: Train the model to imitate the reference answers so that it follows instructions. Model: Follows instructions reliably; matches requested tone, format, and length. Stage 3: Reinforcement Learning from Human Feedback (RLHF) Data: Broad prompts for writing, QA, reasoning, coding, dialogue, and tool use. Process: Generate several diverse candidates per prompt. Human: Human labelers rank candidates. Data: Convert rankings to preference pairs or partial orders for training. Reward: Reward model predicts a scalar preference score for each prompt–output pair. Once we have this model we no longer need human graders in the loop. Note: This model can be reused and can originate from a completely different base model. Process: Optimize the policy for higher reward while limiting Kullback–Leibler (KL) divergence from the SFT reference. Model: A model that has been trained to satisfy the reward model. It is helpful, concise, and safe; follows instructions and meets constraints.
September 25, 2026 at 7:41 PM
boxobark.ing
And if you don't know how much synthetic data the SOTA labs have been putting into pretraining, you haven't been paying attention.

It's one of the reasons that prompt injecting by impersonating CoT style works so well!
September 25, 2026 at 7:31 PM
arxiv-daily-bot.bsky.social
Self-Play Pretraining with Zero Data

Aditya Cowsik et al.

#arXiv #cs.AI #cs.CL
Self-Play Pretraining with Zero Data
Advances in language modeling have been driven by scaling pretraining on ever more data. Yet, the training data is still largely curated on the model's behalf. A more general approach to pretraining would let the model learn to generate the data most useful for its own improvement. This would provi…
arxiv.org
September 25, 2026 at 7:28 PM
boxobark.ing
The only reason it appears so effective in the latest models is because they're not relying on illusory conceptualisation from it just being RLHF; there are millions of direct examples in pretraining to interpolate between.
September 25, 2026 at 7:11 PM
boxobark.ing
Sorry, I don't see how that follows. Chain of thought is just more hypersyntax.

Also, in the latest models, synthetic training data full of chain of thought is now part of the pretraining corpus. Most of it, in fact.
September 25, 2026 at 7:11 PM
crumb.bsky.social
the generator makes a program that generates a single document for pretraining. and then the generator is rewarded...thusly

i think they overcomplicated this. if you don't want "arbitrarily difficult" you can just have a point on your hardness scale where reward starts going down
September 25, 2026 at 6:49 PM
crumb.bsky.social
another adversarial self play paper. in this one they use a generator to create programs that output sequences that are medium-hard for a model to predict, as in causal language modelling. all meaning in the model is bootstrapped from an initially random policy
arxiv.org/abs/2609.30063
Self-Play Pretraining with Zero Data Aditya Cowsik, Kfir Dolev, Michael Y. Li, G. Bruno De Luca, Nourya Cohen, Noah D. Goodman, Yoav Levine Advances in language modeling have been driven by scaling pretraining on ever more data. Yet, the training data is still largely curated on the model's behalf. A more general approach to pretraining would let the model learn to generate the data most useful for its own improvement. This would provide an effectively unbounded source of training data, limited by compute rather than human knowledge. We introduce Self-Play Pretraining with Zero Data, an initial proof-of-concept towards realizing this vision. Our procedure casts synthetic data generation as a search over the space of all computable structure, taking inspiration from Solomonoff induction. Starting from random initialization, two models learn in tandem: a generator proposes programs interpreted by a universal Turing machine, generating byte sequences, while a learner autoregressively predicts these byte sequences. The learner is trained with standard cross-entropy, while the generator is trained with reinforcement learning to produce sequences at the frontier of the learner's capabilities, yielding an adaptive curriculum. A universal Turing machine gives us a search space over all computable data-generating processes, imposing little domain-specific structure, and self-play searches over this space for useful training data. We test whether zero-shot performance on natural data improves predictably with self-play compute; this is a clean test of transfer since neither generator nor learner is trained on natural data. Across several natural datasets, zero-shot loss exhibits predictable scaling in compute. The models also exhibit in-context learning, and discover recognizable mathematical sequences during training.
September 25, 2026 at 6:38 PM
arxiv-daily-bot.bsky.social
RD-JEPA: Predictive latent pretraining for few-trajectory transfer across reaction--diffusion equations

Chenhao Si, Ming Yan

#arXiv #cs.AI #cs.LG
RD-JEPA: Predictive latent pretraining for few-trajectory transfer across reaction--diffusion equations
Learning surrogates for time-dependent partial differential equations often requires a new simulation corpus when the governing operator changes. We introduce RD-JEPA, a joint-embedding predictive architecture for self-supervised pretraining on reaction-diffusion trajectories. A single model is pre…
arxiv.org
September 25, 2026 at 4:06 PM
imrpress.com
📄 Read the full commentary:
https://www.imrpress.com/journal/RCM/27/9/10.31083/RCM53504#F002
✍️ Author:
Zhonghua Sun
Flowchart of the multi-granularity masked autoencoder (MGMAE) framework for self-supervised coronary artery segmentation. Pretraining stage (top): global and local masked autoencoders apply random masking to 128×128×128 and 16×16×16 volumes, with transformer encoders/decoders, reconstruction losses and a contrastive loss. Fine-tuning stage (bottom): an UNETR decoder with topological modules and nuclear-norm minimization (NNM) produces the restored blood vessel region, compared with ground truth.
September 25, 2026 at 3:55 PM
theophite.bsky.social
re: the orthogonality thesis in particular, what i am getting at is that it is not intuitively obvious to me that in a model pretrained on human values and asymptotically confined to its pretrained manifold, you are not confined to the submanifold of the pretraining under mutual support.
September 25, 2026 at 3:48 PM
arxiv-cs-ai.bsky.social
RD-JEPA: Predictive latent pretraining for few-trajectory transfer across reaction--diffusion equations
Read more: https://arxiv.org/html/2609.29403v1
September 25, 2026 at 12:42 PM
eessas-bot.bsky.social
Benjamin Hurt, Oscar O'Donnell: Is Broader Better? A Controlled Study of Multilingual Coverage and Pretraining Objective in Frozen SSL Encoders for Speech Deepfake Detection https://arxiv.org/abs/2609.29138 https://arxiv.org/pdf/2609.29138 https://arxiv.org/html/2609.29138
September 25, 2026 at 6:45 AM
csse-bot.bsky.social
Thang Tran, Lan Dang: Pretraining and adapting a language model on a dependency-free stack: GPT-2 124M from random weights, reproduced against llm.c, and a clinical adapter for Qwen3-0.6B https://arxiv.org/abs/2609.28568 https://arxiv.org/pdf/2609.28568 https://arxiv.org/html/2609.28568
September 25, 2026 at 6:44 AM
cscv-bot.bsky.social
Xinge Guo, Yuanhao Wang, Liqi Shu, Yang Liu, Min Xu: ConPro: Contrast Projection Pretraining for Label-Efficient Vessel Segmentation in DSA Sequences https://arxiv.org/abs/2609.30043 https://arxiv.org/pdf/2609.30043 https://arxiv.org/html/2609.30043
September 25, 2026 at 6:41 AM
cscv-bot.bsky.social
Bazi, Aljuhani, Al Rahhal, Zuair, Alajlan: A Hybrid CNN--State-Space--Attention Backbone with Joint-Embedding Predictive Pretraining for 12-Lead ECG Classification https://arxiv.org/abs/2609.29376 https://arxiv.org/pdf/2609.29376 https://arxiv.org/html/2609.29376
September 25, 2026 at 6:40 AM