Estimated reading time: 5 minutes Introduction Embodied AI agents are increasingly being called upon to interpret complex, multimodal instructions and act robustly in dynamic environments. ThinkAct, presented by researchers from Nvidia and National Taiwan University, offers a breakthrough for vision-language-action (VLA) reasoning, introducing reinforced visual latent planning to bridge high-level multimodal reasoning and low-level robot control. Typical VLA models map raw visual and language inputs directly to actions through end-to-end training, which limits reasoning, long-term planning, and adaptability.

Estimated reading time: 5 minutes Introduction Embodied AI agents are increasingly being called upon to interpret complex, multimodal instructions and act robustly in dynamic environments. ThinkAct, presented by researchers from Nvidia and National Taiwan University, offers a breakthrough for vision-language-action (VLA) reasoning, introducing reinforced visual latent planning to bridge high-level multimodal reasoning and low-level robot control. Typical VLA models map raw visual and language inputs directly to actions through end-to-end training, which limits reasoning, long-term planning, and adaptability.

NVIDIA, we współpracy z Narodowym Uniwersytetem Tajwanu, prezentuje ThinkAct – rewolucyjne podejście do sterowania robotami, które łączy rozumowanie na wysokim poziomie z precyzyjnym działaniem. System ten, oparty na wzmocnieniu wizualnego planowania latentnego, umożliwia agentom AI skuteczne wykonywanie złożonych zadań w dynamicznych środowiskach, otwierając nowe perspektywy dla robotyki. To znaczący krok w kierunku generalistycznych robotów zdolnych do samodzielnej adaptacji i korekty błędów.

Introduction Embodied AI agents are increasingly being called upon to interpret complex, multimodal instructions and act robustly in dynamic environments. ThinkAct, presented by researchers from Nvidia and National Taiwan University, offers a breakthrough for vision-language-action (VLA) reasoning, introducing reinforced visual latent planning to bridge high-level multimodal reasoning and low-level robot control. Typical VLA models map […] The post NVIDIA AI Presents ThinkAct: Vision-Language-Action Reasoning via Reinforced Visual Latent Planning appeared first on MarkTechPost.

机器人操控在动态真实环境中面临挑战，包括物体、光照及接触动力学的变化，以及仿真与现实的鸿沟。NVIDIA 的最新研究成果正通过多维度创新，着力解决这些瓶颈。 ThinkAct 框架整合了高级推理与低级动作执行。其核心在于利用多模态大语言模型（MLLM）通过强化学习生成物理上可行的推理计划，再将计划压缩为潜在轨迹，指导独立的动作模型执行。此“先思考，后行动”的模式，使机器人能在复杂环境中完成长期任务并具备自纠错能力，已在机器人操控和具身推理基准测试中展现出优异的少样本部署和长时序操控能力。 为克服真实世界数据收集的成本与限制，Sim-and-Real Policy Co-training 方法利用仿真与少量真实世界演示数据进行协同训练。通过最优传输（OT）技术对齐模拟与真实世界的观测与动作，学习共享的潜在空间，从而生成能有效泛化至真实场景的操控策略。该框架在物体堆叠、装箱等任务中表现出色，仅需少量演示即可完成复杂任务。 RobotSmith 利用视觉-语言模型（VLM）的能力，自动设计适用于特定任务的机器人工具。它通过 VLM 协作生成工具几何形状，并结合仿真进行任务规划与优化，实现了从推、舀到包裹等多样化工具设计，并成功应用于制作煎饼等长时序任务，展现了其在复杂操作中的潜力。 此外，NVIDIA Cosmos Cookbook 作为一个开源资源库，提供利用 Cosmos 开放世界基础模型（WFMs）生成高质量合成数据集的“食谱”，进一步弥合了仿真与现实的差距，加速机器人和自动驾驶系统的开发与部署。 查看消息来源