#DeepSpeed
The 2026 PyTorch Foundation Contributor Awards honored 15 winners across nine categories for contributions to PyTorch, vLLM, Ray, and DeepSpeed. Congratulations to the winners and 11 nominees.

Meet the honorees at https://pytorch.org/blog/2026-pytorch-foundation-contributor-awards/
November 16, 2026 at 8:47 PM
DEEPSPEED V0.20.0: ZERO-3 MASTER WEIGHT REUSE, AUTOTP FROZEN MODELS, HYBRID ENGINE GRAPH CAPTURE OPT-IN
November 14, 2026 at 2:15 PM
The source, dated, on The Deploy Log:
https://deployedbyai.com/log/7bd7e718dc16/deepspeed-v0-20-0/?k=br

Follow for one line every time AI actually ships, as it ships, 24/7.
DeepSpeed v0.20.0
DeepSpeed, 12 Nov 2026. Sourced to github.com.
deployedbyai.com
November 14, 2026 at 2:15 PM
All 146 session recordings from #PyTorchCon North America 2026 are on YouTube, from keynotes to technical talks on training, inference, and kernel engineering. The conference covered PyTorch, vLLM, DeepSpeed, Ray, Helion, and Safetensors.

https://www.youtube.com/playlist?list=PLVqjs87Mef-U
November 12, 2026 at 6:17 PM
🎯 Training and fine-tuning: #Unsloth, #PEFT, #TRL, #Axolotl, #LitGPT, #DeepSpeed and #torchtune
November 7, 2026 at 11:00 AM
Training 100B+ LLMs breaks standard DDP.

Our SRE guide to Megatron-DeepSpeed 3D Parallelism covers TP/PP/DP topology math, ZeRO-2 vs ZeRO-3 trade-offs, NCCL RoCEv2 tuning, and Slurm scripts:

www.servermo.com/howto/setup-...

#DeepSpeed #MegatronLM #AI #MLOps
November 6, 2026 at 9:02 AM
DeepSpeed virtual office hours on October 27 cover features and benchmark results presented at PyTorch Conference and the proposed Q4 roadmap.

9 AM PT / 12 PM ET
https://zoom-lfx.platform.linuxfoundation.org/meeting/93902569995?password=7d9c4fc9-3efa-4715-88f0-df8a6deb008b
October 26, 2026 at 4:50 PM
#PyTorchCon North America 2026 starts today in San Jose! Can't be here in person? Stream the keynotes live:
https://events.linuxfoundation.org/pytorch-conference-north-america/program/keynote-livestream/

Talks and workshops across PyTorch, vLLM, DeepSpeed, Ray, Helion, and Safetensors.
October 20, 2026 at 4:07 PM
Stop by the PyTorch Foundation booth in the Community Expo and tell us what you're building. Ask about PyTorch, vLLM, DeepSpeed, Ray, Helion, or Safetensors, or share what you're seeing across the open source AI stack. #PyTorchCon
October 20, 2026 at 7:21 PM
#後で読む 用メモです→
フレームワークが動くのを待つなFSDPを書け リリース8日後に397B MoEを学習した基盤を全公開
フレームワークが動くのを待つな、FSDPを書け — リリース8日後に397B MoEを学習した基盤を全公開 - Qiita
はじめに:フレームワークが動けば、たしかに簡単。動けば。 LLMのファインチューニングというと、verl・ms-swift・axolotl・DeepSpeedといった学習フレームワークを使うのが定石とされています。 しかし、日本の共有計算資源(Slurmクラスタ、固定され...
qiita.com
October 17, 2026 at 9:54 PM
Think DeepSpeed is only good for ZeRO and its usability, not training throughput?

At #PyTorchCon North America 2026, Masahiro Tanaka will cover tensor, sequence, and expert parallelism in DeepSpeed and walk through benchmarks.

https://hubs.la/Q04v4SL60
October 6, 2026 at 10:43 PM
Open source. Supports ZeRO-3 DeepSpeed, PyTorch FSDP, and runs on Azure, LambdaLabs, and FluidStack. GitHub Actions CI/CD docs on the listing. www.everydev.ai/tools/higgs...
Higgsfield - Open Source Distributed ML Training | EveryDev.ai
Higgsfield is an open-source GPU workload manager and machine learning framework built for training massive neural networks — from billions to…
www.everydev.ai
October 1, 2026 at 2:01 PM
📝 Summary:

Higgsfield is an open-source GPU orchestration framework that manages multi-node, fault-tolerant training for trillion-parameter models by coordinating resources, supporting deepspeed/PyTorch sharding, and integrating with GitHub for CI/CD, all aimed at streamlined distributed LLM (1/2)
September 19, 2026 at 1:17 PM
The first-ever PyTorch Day Japan comes to Tokyo Dec. 10, co-hosted by PyTorch Foundation, Hugging Face, IBM, and Mitsubishi Electric.

Explore PyTorch, vLLM, DeepSpeed, Ray, Helion, and Safetensors. CFP open through Sept. 27.

https://pytorch.org/blog/pytorch-day-japan-2026-comes-to-tokyo/
September 18, 2026 at 12:36 AM
ZeRO vs FSDP2, one harness, two interconnects: on 4x A100 (NVLink) FSDP2 reshard beats ZeRO-3 by 1.83x. On 4x L4 (PCIe) ZeRO-2/3 died in NCCL watchdog timeouts, revived by one allocator env var. Verbatim errors + failure taxonomy:

implicit-none.com/en/deepspeed...
DeepSpeed ZeRO vs PyTorch FSDP2, Measured: 1.8x on NVLink, Barely Finishing on PCIe (Plus How to Actually Get GPUs on GKE)
DeepSpeed ZeRO 0-3 vs PyTorch FSDP2 on one shared harness, one model (Qwen2.5-3B), two interconnects. On A100 (NVLink) FSDP2 beats ZeRO-3 by 1.8x; on L4 (PCIe) ZeRO-2/3 die with NCCL watchdog timeouts...
implicit-none.com
September 16, 2026 at 7:21 AM
Sharding a small model costs more than it saves.

Qwen2.5-0.5B over 20 steps on 2x H100: 15.76 s on a single card, 35.19 s with ZeRO-2, 49.99 s with ZeRO-3.

ZeRO earns its place at 7B, not at 0.5B.

https://massedcompute.com/deepspeed-multi-gpu-training-cloud/
September 13, 2026 at 3:05 PM
At PyTorch Conference North America 2026, connect with people working across PyTorch, vLLM, DeepSpeed, Ray, Helion, Safetensors, and the AI stack.

At #PyTorchCon NA 2025, Richard Liaw spoke about “how important Ray can be for the stack.”

Register by Sept. 4: https://hubs.la/Q04tDf0j0

#PyTorchCon
September 3, 2026 at 6:31 PM
Ray comes to the PyTorch Foundation thenewstack.io/ray-c... via @thenewstack | The Ray project will join existing projects like PyTorch itself, the vLLM inference engine and deep learning optimization library DeepSpeed
Ray Comes to the PyTorch Foundation
The Ray project will join existing projects like PyTorch itself, the vLLM inference engine and deep learning optimization library DeepSpeed.
thenewstack.io
October 23, 2025 at 1:00 PM
if you don't follow me on twitter where i've been liveblogging this saga basically i got fed up with deepspeed (an existing framework for doing this) after trying to get it working for two days, gave up this afternoon and just wrote it myself
November 22, 2024 at 7:04 AM
Implementing DeepSpeed For Scalable Transformers: Superior Coaching With Gradient Checkpointing And Parallelism

On this superior DeepSpeed tutorial, we offer a hands-on walkthrough of cutting-edge optimization strategies for coaching massive language fashions effectively. By combining ZeRO…
Implementing DeepSpeed For Scalable Transformers: Superior Coaching With Gradient Checkpointing And Parallelism
On this superior DeepSpeed tutorial, we offer a hands-on walkthrough of cutting-edge optimization strategies for coaching massive language fashions effectively. By combining ZeRO optimization, mixed-precision coaching, gradient accumulation, and superior DeepSpeed configurations, the tutorial demonstrates the best way to maximize GPU reminiscence utilization, cut back coaching overhead, and allow scaling of transformer fashions in resource-constrained environments, similar to Colab. Alongside mannequin creation and coaching, it additionally covers efficiency monitoring, inference optimization, checkpointing, and benchmarking totally different ZeRO levels, offering practitioners with each theoretical insights and sensible code to speed up mannequin improvement.
nextbusiness24.com
September 7, 2025 at 2:47 AM
Implementing DeepSpeed for Scalable Transformers: Advanced Training with Gradient Checkpointing and Parallelism

In this advanced DeepSpeed tutorial, we provide a hands-on walkthrough of cutting-edge optimization techniques for training large language models efficiently. By combining ZeRO…
Implementing DeepSpeed for Scalable Transformers: Advanced Training with Gradient Checkpointing and Parallelism
In this advanced DeepSpeed tutorial, we provide a hands-on walkthrough of cutting-edge optimization techniques for training large language models efficiently. By combining ZeRO optimization, mixed-precision training, gradient accumulation, and advanced DeepSpeed configurations, the tutorial demonstrates how to maximize GPU memory utilization, reduce training overhead, and enable scaling of transformer models in resource-constrained environments, such as Colab. Alongside model creation and training, it also covers performance monitoring, inference optimization, checkpointing, and benchmarking different ZeRO stages, providing practitioners with both theoretical insights and practical code to accelerate model development.
nexttech-news.com
September 7, 2025 at 12:10 AM
ZenFlow: A New DeepSpeed Extension Designed As A Stall-Free Offloading Engine For Massive Language Mannequin (LLM) Coaching

The DeepSpeed workforce unveiled ZenFlow, a brand new offloading engine designed to beat a significant bottleneck in giant language mannequin (LLM) coaching: CPU-induced GPU…
ZenFlow: A New DeepSpeed Extension Designed As A Stall-Free Offloading Engine For Massive Language Mannequin (LLM) Coaching
The DeepSpeed workforce unveiled ZenFlow, a brand new offloading engine designed to beat a significant bottleneck in giant language mannequin (LLM) coaching: CPU-induced GPU stalls. Whereas offloading optimizers and gradients to CPU reminiscence reduces GPU reminiscence stress, conventional frameworks like ZeRO-Offload and ZeRO-Infinity usually depart costly GPUs idle for many of every coaching step—ready on sluggish CPU updates and PCIe transfers. For instance, fine-tuning Llama 2-7B on 4× A100 GPUs with full offloading can balloon step time from 0.5s to over 7s, a 14× slowdown. ZenFlow eliminates these stalls by decoupling GPU and CPU computation with importance-aware pipelining, delivering as much as 5× end-to-end speedup over ZeRO-Offload and decreasing GPU stalls by greater than 85%.
nextbusiness24.com
August 21, 2025 at 4:43 AM
ZenFlow: A New DeepSpeed Extension Designed as a Stall-Free Offloading Engine for Large Language Model (LLM) Training

The DeepSpeed team unveiled ZenFlow, a new offloading engine designed to overcome a major bottleneck in large language model (LLM) training: CPU-induced GPU stalls. While…
ZenFlow: A New DeepSpeed Extension Designed as a Stall-Free Offloading Engine for Large Language Model (LLM) Training
The DeepSpeed team unveiled ZenFlow, a new offloading engine designed to overcome a major bottleneck in large language model (LLM) training: CPU-induced GPU stalls. While offloading optimizers and gradients to CPU memory reduces GPU memory pressure, traditional frameworks like ZeRO-Offload and ZeRO-Infinity often leave expensive GPUs idle for most of each training step—waiting on slow CPU updates and PCIe transfers.
nexttech-news.com
August 21, 2025 at 4:32 AM
What is the best library to train LLMs? There are many options and hard to understand the details... #MLSky
- GPT-NeoX
- Megatron-LM
- Megatron-Deepspeed
- TinyLlama
- Others
February 15, 2024 at 5:04 PM