#OpenRLHF
Bfloat16 Rounding Issues Cause Near-Complete Stagnation in RL Policy Updates

🤖 IA: It's not clickbait ✅
👥 Users: It's not clickbait ✅

#ai #machinelearning #bfloat16

👇👇👇
Bfloat16 Rounding Issues Cause Near-Complete Stagnation in RL Policy Updates
The article discusses a critical issue in reinforcement learning (RL) post-training for language models, where standard bfloat16 (bf16) rounding discards over 99% of parameter updates. Modern frameworks like Verl, TRL, and OpenRLHF maintain master weights in fp32 within PyTorch but serve rollouts using a bf16 copy in inference engines such as vLLM or SGLang. The conversion from fp32 to bf16 uses round-to-nearest, which creates a 'dead zone' where small updates are rounded to zero. At typical RL learning rates (1e-6 to 3e-7), nearly all per-parameter updates fall below the threshold, causing the inference engine to serve a model that is effectively frozen. The Stanford paper by Andrew Gao and William Gao demonstrates that only 0.4-0.5% of weights change per optimizer step, with weights needing 50-200 steps to cross a bf16 grid boundary. This discrepancy leads to a significant mismatch between the trained model and the one used for rollouts, as evidenced by lower scores on checkpoints compared to the actual inference model. The paper proposes 'Fair Cast,' a solution using stochastic rounding at the export barrier to ensure exported weights match master weights in expectation. This method requires adding 16 random bits before truncation, with minimal latency impact. Testing on Qwen3 models showed Fair Cast outperforming round-to-nearest in 18 out of 25 runs, particularly improving accuracy on logic puzzles from 0.33 to 0.57. Frameworks should check if their weight sync uses standard casts like weight.to(torch.bfloat16), which introduces this issue, and consider switching to stochastic rounding to prevent policy update stagnation.
en.killbait.com
November 2, 2026 at 5:49 PM
(1/2) 104 Likes, 0 Comments, 12 Jun 2025, alphaXiv
Magistral | alphaXiv
View 1 comments: Magistral uses REINFORCE++-baseline from OpenRLHF to train the reasoning model.
www.alphaxiv.org
July 11, 2025 at 12:07 AM
ふふふ、今日は人工知能ハッカソンの日
タイムライン
19:00 ~ 19:05 配信開始・オープニング
19:05 ~ 19:20 Auris(独自アーキテクチャの音声モデル向け大規模事前学習)
19:20 ~ 19:35 OpenRLHF(RLHFで学習したLLM)
19:35 ~ 19:50 youlery(mistralモデルをベースとした日本語の大規模言語モデル)
19:50 ~ 20:05 Local Novel LLM Project(小説生成モデル)
20:05 ~ 20:20 文字レベルLLM実験(LLM(Mambaを予定)を文字単位で学習させる実験)
May 1, 2024 at 8:23 AM
[2025-07-10] 📚 Updates in #VLM

(1) <a href="https://researchtrend.ai/papers/2404.02353" class="hover:underline text-blue-600 dark:text-sky-400 no-card-link" target="_blank" rel="noopener" data-link="bsky">Semantic Augmentation in Images using Language
(2) Semantic Augmentation in Images using Language
(3) OpenRLHF: An Easy-to-use, Scalable and High-performance RLHF Framework

🔍 More at researchtrend.ai/communities/VLM
July 10, 2025 at 3:06 AM
Accelerating RLHF with vLLM, Best Practice from OpenRLHF As demand grows for training reasoning-c...

https://blog.vllm.ai/2025/04/23/openrlhf-vllm.html

Result Details
Accelerating RLHF with vLLM, Best Practice from OpenRLHF
As demand grows for training reasoning-capable large language models (LLMs), Reinforcement Learning from Human Feedback (RLHF) has emerged as a cornerstone technique. However, conventional RLHF pipelines—especially those using Proximal Policy Optimization (PPO)—are often hindered by substantial computational overhead. This challenge is particularly pronounced with models that excel at complex reasoning tasks (such as OpenAI-o1 and DeepSeek-R1), where generating long chain-of-thought (CoT) outputs can account for up to 90% of total training time. These models must produce detailed, step-by-step reasoning that can span thousands of tokens, making inference significantly more time-consuming than the training phase itself. As a pioneering inference framework, vLLM provides a user-friendly interface for generating RLHF samples and updating model weights. ## Design of OpenRLHF To strike a balance between performance and usability in RLHF frameworks, OpenRLHF is designed as a high-performance yet user-friendly solution that integrates key technologies like Ray, vLLM, Zero Redundancy Optimizer (ZeRO-3), and Automatic Tensor Parallelism (AutoTP): **Ray** acts as the backbone of OpenRLHF’s distributed architecture. With powerful scheduling and orchestration features, Ray efficiently manages complex data flows and computations, including distributing rule-based reward models across multiple nodes. **vLLM with Ray Executor and AutoTP** plays a central role in accelerating inference. With built-in support for Ray Executors and integration with HuggingFace Transformers, it enables efficient weight updates through AutoTP, resulting in high-throughput and memory-efficient LLM generation. **ZeRO-3 withHuggingFace Transformers**, a memory optimization approach from DeepSpeed, empowers OpenRLHF to train large models without requiring heavyweight frameworks like Megatron. This seamless integration with HuggingFace allows for simple loading and fine-tuning of pre-trained models. Together, Ray, vLLM, ZeRO-3, and HuggingFace Transformers create a cutting-edge yet streamlined solution for accelerating RLHF training. The architecture has also influenced other frameworks such as veRL, which adopt similar paradigms for scalable and efficient RLHF training. OpenRLHF is also the first open-source RLHF framework developed based on Ray, vLLM and ZeRO-3, and has been used by Google, Bytedance, Alibaba, Meituan, Berkeley Starling Team etc. As illustrated above, OpenRLHF uses Ray’s Placement Group API to flexibly schedule components of the RLHF pipeline, including the vLLM engine, Actor, Critic, Reference, and Reward models. Although represented separately, these components can be colocated in shared Ray placement groups to maximize resource efficiency. For example, all modules can operate within the same GPU group in a hybrid engine configuration, or specific components—such as the Actor and Critic—can be grouped together. All modules are orchestrated by a central Ray Actor, which manages the entire training lifecycle. Weight synchronization between the Actor and the vLLM engine is handled via high-performance communication methods, such as NVIDIA Collective Communications Library (NCCL) or CUDA Inter-Process Communication (IPC) memory transfers in hybrid engine settings. ## Implementing RLHF Acceleration with vLLM Ray Executor OpenRLHF and vLLM provide a clean and efficient set of APIs to simplify interaction within RLHF pipelines. By implementing a custom `WorkerExtension` class, users can handle weight synchronization between training and inference components. The environment variables `VLLM_RAY_PER_WORKER_GPUS` and `VLLM_RAY_BUNDLE_INDICES` allows fine-grained GPU resource allocation per worker, enabling hybrid engine configurations where multiple components share a GPU group: # rlhf_utils.py class ColocateWorkerExtension: """ Extension class for vLLM workers to handle weight synchronization. This class ensures compatibility with both vLLM V0 and V1. """ def report_device_id(self) -> str: """Report the unique device ID for this worker""" from vllm.platforms import current_platform self.device_uuid = current_platform.get_device_uuid(self.device.index) return self.device_uuid def update_weights_from_ipc_handles(self, ipc_handles): """Update model weights using IPC handles""" handles = ipc_handles[self.device_uuid] device_id = self.device.index weights = [] for name, handle in handles.items(): func, args = handle list_args = list(args) list_args[6] = device_id # Update device ID for current process tensor = func(*list_args) weights.append((name, tensor)) self.model_runner.model.load_weights(weights=weights) torch.cuda.synchronize() # main.py class MyLLM(LLM): """ Custom LLM class to handle GPU resource allocation and bundle indices. This ensures proper GPU utilization and placement group management. """ def __init__(self, *args, bundle_indices: list, **kwargs): # Prevent Ray from manipulating CUDA_VISIBLE_DEVICES at the top level os.environ.pop("CUDA_VISIBLE_DEVICES", None) # Configure GPU utilization per worker os.environ["VLLM_RAY_PER_WORKER_GPUS"] = "0.4" os.environ["VLLM_RAY_BUNDLE_INDICES"] = ",".join(map(str, bundle_indices)) super().__init__(*args, **kwargs) # Create Ray's placement group for GPU allocation pg = placement_group([{"GPU": 1, "CPU": 0}] * 4) ray.get(pg.ready()) # Create inference engines inference_engines = [] for bundle_indices in [[0, 1], [2, 3]]: llm = ray.remote( num_gpus=0, scheduling_strategy=PlacementGroupSchedulingStrategy( placement_group=pg ) )(MyLLM).remote( model="facebook/opt-125m", tensor_parallel_size=2, distributed_executor_backend="ray", gpu_memory_utilization=0.4, worker_extension_cls="rlhf_utils.ColocateWorkerExtension", bundle_indices=bundle_indices ) inference_engines.append(llm) The complete RLHF example walks through initializing Ray with a specified GPU count, creating a placement group to manage resources, and defining both training actors and inference engines. The training actors manage model initialization and weight updates, while the inference engines serve models via vLLM. Weight synchronization is carried out using CUDA IPC or NCCL, ensuring coherence and efficiency throughout the RLHF pipeline. ## Acknowledgements We would like to express our sincere gratitude to the vLLM contributors, including Kaichao You, Cody Yu, Rui Qiao, and many others, without which the OpenRLHF integration with vLLM will not be possible. Kaichao You from the vLLM team leads the RLHF integration. The OpenRLHF project is the first open-source RLHF framework based on Ray and vLLM. We would like to thank Jian Hu, Songlin Jiang, Zilin Zhu, Xibin Wu and many others for their significant contributions to the Ray, vLLM Wrapper and Hybrid Engine components of the OpenRLHF project. Jian Hu leads the development.
blog.vllm.ai
April 24, 2025 at 2:15 PM
Feed: "Aflah’s Substack"
By: Aflah on Thursday, January 8, 2026
SkyPilot × OpenRLHF: A Unified Workflow for Multi-Cloud RLHF Training
Learn about SkyPilot and my experience porting an OpenRLHF workload to SkyPilot
aflah02.substack.com
January 10, 2026 at 12:04 PM
🚀 Dive into NVIDIA's new Molt framework—PyTorch‑powered, built for agentic RL research. Harness H100 GPUs, NeMo integration, and OpenRLHF support. Curious how it reshapes RL labs? Check it out! #Molt #AgenticRL #OpenRLHF

🔗 aidailypost.com/news/nvidias...
August 2, 2026 at 6:31 AM
New Blog Up!

open.substack.com/pub/aflah02/...

Last month I spent a bunch of time playing around with SkyPilot and OpenRLHF. This post recaps my experience with both and walks through my process of porting an OpenRLHF workload to SkyPilot.
January 8, 2026 at 7:25 AM
a bug in DeepSpeed silently dropped micro-batches during gradient accumulation. TRL, OpenRLHF, Llama-Factory all affected.

dozens of papers found mixed-policy RL beats SFT-then-RL. the baseline was suppressed.

fix the bug: SFT-then-RL wins. +22 points.

trusted the tooling. the tooling was wrong.
May 6, 2026 at 10:57 AM
Wanted to test out OpenRLHF, so ran a REINFORCE test run on Llama-3.2-1B-Instruct using gpt-4o-mini as an external LLM-as-judge reward model, and it uplifted the scores on ARC, GSM8k, Hellaswag, and MMLU. Pretty cool in my opinion! huggingface.co/ericflo/Llam...
ericflo/Llama-3.2-1B-Instruct-RLHF-v0.1 · Hugging Face
We’re on a journey to advance and democratize artificial intelligence through open source and open science.
huggingface.co
December 23, 2024 at 11:35 PM
On the code level, how does our self-play method work?
We built on OpenRLHF and their Re++ algorithm, a critic-free method like GPRO. Both roles share the same LLM parameters and mix the training experiences for gradient descent together.
Our code is also open-sourced (see next)!
June 12, 2025 at 5:12 AM
Holy crap, I burnt like 8 hours this week banging my head on trying to fix things, but my big runs were blowing up b/c DeepSpeed 0.16.4 is super borked (fix is disable gradient checkpointing or downgrade to 0.15.0): github.com/deepspeedai/...
[BUG] OOM when train 70B models using deepspeed 0.16.4 · Issue #7116 · deepspeedai/DeepSpeed
We found that using OpenRLHF + DeepSpeed 0.15.0, SFT + Adam Offload can train a 70B model with 8 A100 70G + ZeRO3, whereas DeepSpeed 0.16.4 results in OOM. You can try the script https://github.com...
github.com
March 27, 2025 at 3:21 PM
Have seen such discussion in the OpenRLHF group chat, but no paper or blog I think
March 14, 2025 at 8:29 PM