Awesome to see people reproducing the accessible LeRobot zero-shot sim2real project! Trained for just 90 min in ManiSkill and deployed directly in real. Sim2real is not easy, but very rewarding when it works
Original post by Jianwei Zhang on LinkedIn www.linkedin.com/posts/jianwe...
Original post by Jianwei Zhang on LinkedIn www.linkedin.com/posts/jianwe...
July 4, 2025 at 6:31 PM
Awesome to see people reproducing the accessible LeRobot zero-shot sim2real project! Trained for just 90 min in ManiSkill and deployed directly in real. Sim2real is not easy, but very rewarding when it works
Original post by Jianwei Zhang on LinkedIn www.linkedin.com/posts/jianwe...
Original post by Jianwei Zhang on LinkedIn www.linkedin.com/posts/jianwe...
What "distribution shift"? What do you mean "sim2real gap"?
November 22, 2024 at 1:33 PM
What "distribution shift"? What do you mean "sim2real gap"?
Code/tutorial released for🤗LeRobot zero-shot visual sim2real Reinforcement Learning!
Train your SO100 robot in ManiSkill's fast simulator+renderer💨 and deploy a cube picking model zero-shot. Just in time for the LeRobot Hackathon, happy hacking!
github.com/StoneT2000/l...
Train your SO100 robot in ManiSkill's fast simulator+renderer💨 and deploy a cube picking model zero-shot. Just in time for the LeRobot Hackathon, happy hacking!
github.com/StoneT2000/l...
GitHub - StoneT2000/lerobot-sim2real: lerobot sim2real code
lerobot sim2real code. Contribute to StoneT2000/lerobot-sim2real development by creating an account on GitHub.
github.com
June 13, 2025 at 10:53 PM
Code/tutorial released for🤗LeRobot zero-shot visual sim2real Reinforcement Learning!
Train your SO100 robot in ManiSkill's fast simulator+renderer💨 and deploy a cube picking model zero-shot. Just in time for the LeRobot Hackathon, happy hacking!
github.com/StoneT2000/l...
Train your SO100 robot in ManiSkill's fast simulator+renderer💨 and deploy a cube picking model zero-shot. Just in time for the LeRobot Hackathon, happy hacking!
github.com/StoneT2000/l...
might be the first time I made it personally on the trending developers list on github! One of the contributing factors being the lerobot sim2real code
July 19, 2025 at 3:12 AM
might be the first time I made it personally on the trending developers list on github! One of the contributing factors being the lerobot sim2real code
*This robotics issue called "Sim2Real" seems more general than one might think, as in, "How come my real-life Real romance varies radically from this cute Sim romance in this 2hr romcom movie" #Sim2Real
June 29, 2025 at 10:35 AM
*This robotics issue called "Sim2Real" seems more general than one might think, as in, "How come my real-life Real romance varies radically from this cute Sim romance in this 2hr romcom movie" #Sim2Real
Max Verstappen is another evidence that Sim2Real is working
May 17, 2026 at 9:11 AM
Max Verstappen is another evidence that Sim2Real is working
Personal update: I’m elated to have joined NVIDIA as a researcher in Seattle where I will be working on human-inspired sim2real perception in robotics. Grateful for the amazing opportunity and excited to see where this journey goes!
November 21, 2025 at 9:57 PM
Personal update: I’m elated to have joined NVIDIA as a researcher in Seattle where I will be working on human-inspired sim2real perception in robotics. Grateful for the amazing opportunity and excited to see where this journey goes!
The robot behaviors shown below are trained without any teleop, sim2real, genai, or motion planning. Simply show the robot a few examples of doing the task yourself, and our new method, called Point Policy, spits out a robot-compatible policy!
February 28, 2025 at 7:09 PM
The robot behaviors shown below are trained without any teleop, sim2real, genai, or motion planning. Simply show the robot a few examples of doing the task yourself, and our new method, called Point Policy, spits out a robot-compatible policy!
sneak peek of what Xander and I will show next week at RSS 2025 for the maniskill demo session:
Zero shot visual sim2real (basic) manipulation, one camera and <1 hour of RL in sim with SO100
~3 seconds to pick random color cubes, could be faster but goal is accessibility and low cost!
Zero shot visual sim2real (basic) manipulation, one camera and <1 hour of RL in sim with SO100
~3 seconds to pick random color cubes, could be faster but goal is accessibility and low cost!
June 11, 2025 at 5:56 PM
sneak peek of what Xander and I will show next week at RSS 2025 for the maniskill demo session:
Zero shot visual sim2real (basic) manipulation, one camera and <1 hour of RL in sim with SO100
~3 seconds to pick random color cubes, could be faster but goal is accessibility and low cost!
Zero shot visual sim2real (basic) manipulation, one camera and <1 hour of RL in sim with SO100
~3 seconds to pick random color cubes, could be faster but goal is accessibility and low cost!
“Sim2Real” sounds like a 90s boy band
May 30, 2026 at 10:24 PM
“Sim2Real” sounds like a 90s boy band
*You're allowing ten zillion simulated robots to train themselves in LLM and then shoehorning the results into an existent metal device #Sim2Real
*That's a pretty far cry from "traditional control systems"
*That's a pretty far cry from "traditional control systems"
Traditional control systems can already handle this sort of problem really well, with well over a century of research and real-world application. Heck, more general neural networks have even been applied to the problem too. Why shoehorn LLMs there?
July 1, 2025 at 6:34 AM
*You're allowing ten zillion simulated robots to train themselves in LLM and then shoehorning the results into an existent metal device #Sim2Real
*That's a pretty far cry from "traditional control systems"
*That's a pretty far cry from "traditional control systems"
Currently trying to do a kind of api setup where you can really write code like this to get easy sim2real real environment gym interfaces where the action/observation space is the same as the sim env, controllers are aligned etc. hoping to make sim2real more accessible
March 14, 2025 at 10:20 PM
Currently trying to do a kind of api setup where you can really write code like this to get easy sim2real real environment gym interfaces where the action/observation space is the same as the sim env, controllers are aligned etc. hoping to make sim2real more accessible
i second this! At least for those working on robot simulation there’s a lot of small details that go into making sim fast, rl+sim fast, making sim working in the first place, sim2real and real2sim!
I think we need an AMA series for Robotics / Embodied AI with an optional anonymous setting. Will be both fun and informative to new community members to absorb folk knowledge.
Does everyone in your community agree on some folk knowledge that isn’t published anywhere? Put it in a paper! It’s a pretty valuable contribution
November 28, 2024 at 6:28 AM
i second this! At least for those working on robot simulation there’s a lot of small details that go into making sim fast, rl+sim fast, making sim working in the first place, sim2real and real2sim!
Monday, September 28 at 11:30am: TTIC Colloquium presents Osbert Bastani of UPenn with a talk titled "Scaling Sim2Real Reinforcement Learning." Please join us in Room 530, 5th floor.
September 25, 2026 at 5:15 PM
Monday, September 28 at 11:30am: TTIC Colloquium presents Osbert Bastani of UPenn with a talk titled "Scaling Sim2Real Reinforcement Learning." Please join us in Room 530, 5th floor.
The sim2real demo had some mixed success, hampered primarily by the lighting conditions of the outdoors.
At least it worked sometimes! Hindsight says that despite the weather, low-cost nature, only 1 hour of training, anything working is a miracle
At least it worked sometimes! Hindsight says that despite the weather, low-cost nature, only 1 hour of training, anything working is a miracle
June 22, 2025 at 4:29 PM
The sim2real demo had some mixed success, hampered primarily by the lighting conditions of the outdoors.
At least it worked sometimes! Hindsight says that despite the weather, low-cost nature, only 1 hour of training, anything working is a miracle
At least it worked sometimes! Hindsight says that despite the weather, low-cost nature, only 1 hour of training, anything working is a miracle
After a busy start to the year, I have a new lecture to share. Generalist models have been making good progress, but #sim2real just works. In this lecture, I explain domain randomization, simulation limitations, and connections to generalization.
April 20, 2026 at 2:06 PM
After a busy start to the year, I have a new lecture to share. Generalist models have been making good progress, but #sim2real just works. In this lecture, I explain domain randomization, simulation limitations, and connections to generalization.
NVIDIA Isaac Labを使って深層強化学習でオリジナル設計ヒューマノイドロボットをSim2Realで歩かせる
#ヒューマノイド #ロボット #深層強化学習 #IsaacLab #sim2real
#ヒューマノイド #ロボット #深層強化学習 #IsaacLab #sim2real
NVIDIA Isaac Labを使って深層強化学習でオリジナル設計ヒューマノイドロボットをSim2Realで歩かせる - Qiita
はじめに 自作の小型二足歩行ヒューマノイドロボット「スピロ」を、深層強化学習で歩かせた話です。 昨年はパラメータから足先軌道を生成する方式で歩かせていましたが、今年は歩容そのものを強化学習で獲得させ、学習済みポリシーをマイコン単体で動かすところまで持っていきました。この...
qiita.com
September 24, 2026 at 6:20 PM
How would you address sim2real challenges?
November 21, 2024 at 11:15 PM
How would you address sim2real challenges?
We’ve been investigating how sim, while wrong, can be useful for real-world robotic RL! In our #NeurIPS2024 work, we theoretically showed how naive sim2real transfer can be inefficient, but if you *learn to explore* in sim, this transfers to the real world! We show this works on real robots! 🧵(1/6)
December 6, 2024 at 12:46 AM
We’ve been investigating how sim, while wrong, can be useful for real-world robotic RL! In our #NeurIPS2024 work, we theoretically showed how naive sim2real transfer can be inefficient, but if you *learn to explore* in sim, this transfers to the real world! We show this works on real robots! 🧵(1/6)
Almost all robotics startups are betting on learning from large supervised learning datasets collected by teleoperation. The odds of success for this strategy are small.
Like the Sim2Real bubble, this bubble might not burst for years. At-least it's keeping the roboticists employed.
Like the Sim2Real bubble, this bubble might not burst for years. At-least it's keeping the roboticists employed.
January 1, 2025 at 3:04 PM
Almost all robotics startups are betting on learning from large supervised learning datasets collected by teleoperation. The odds of success for this strategy are small.
Like the Sim2Real bubble, this bubble might not burst for years. At-least it's keeping the roboticists employed.
Like the Sim2Real bubble, this bubble might not burst for years. At-least it's keeping the roboticists employed.
Deep policy gradients are used to train the largest #LLM models and are the main #reinforcementlearning algorithm in sim2real transfer. In my next set of lectures on #robotlearning I cover the basics of policy gradients to the methods used to train #AlphaStar (in part 2).
March 3, 2025 at 8:00 PM
Deep policy gradients are used to train the largest #LLM models and are the main #reinforcementlearning algorithm in sim2real transfer. In my next set of lectures on #robotlearning I cover the basics of policy gradients to the methods used to train #AlphaStar (in part 2).
This is what you do when you set allow_sliding=true in your Habitat simulator. If you want to run test with sim2real transfer, you might want to consider wearing a helmet !
July 4, 2025 at 5:20 PM
This is what you do when you set allow_sliding=true in your Habitat simulator. If you want to run test with sim2real transfer, you might want to consider wearing a helmet !
direct sim2real of a RGB dextrous manipulation policy trained via RL + distillation! Just from the videos you can see a big advantage of RL over pure imitation learning is learning fast behaviors+solve precise tasks more easily
parallelrendering is doing a lot of wonders for people's research!
parallelrendering is doing a lot of wonders for people's research!
February 6, 2025 at 11:24 PM
direct sim2real of a RGB dextrous manipulation policy trained via RL + distillation! Just from the videos you can see a big advantage of RL over pure imitation learning is learning fast behaviors+solve precise tasks more easily
parallelrendering is doing a lot of wonders for people's research!
parallelrendering is doing a lot of wonders for people's research!