NVIDIA unveils suite of tools to accelerate humanoid robotics

Developers in the field of humanoid robotics will now have access to a suite of new tools and services from NVIDIA, as announced at the SIGGRAPH 2024 conference. These include NVIDIA NIM Microservices for robotics simulation in Isaac Lab and Isaac Sim, the OSMO Robot Cloud Compute Orchestration Service, Teleoperated Data Capture Workflow, and several other offerings.

Jensen Huang, founder and CEO of NVIDIA, spoke at the conference, highlighting the company's commitment to advancing humanoid robotics. "The next wave of AI is robotics and one of the most exciting developments is humanoid robots," said Huang. "We're advancing the entire NVIDIA robotics stack, opening access for worldwide humanoid developers and companies to use the platforms, acceleration libraries and AI models best suited for their needs."

NVIDIA NIM microservices include pre-built containers powered by NVIDIA inference software, designed to reduce deployment times significantly. Two new AI microservices, MimicGen and Robocasa, are set to enhance simulation workflows for generative physical AI in NVIDIA Isaac Sim. MimicGen generates synthetic motion data from spatial computing devices like Apple Vision Pro. Robocasa creates robot tasks and simulation-ready environments in OpenUSD, a framework for 3D world development and collaboration.

In addition to NIM, NVIDIA has introduced OSMO, a cloud-native managed service. OSMO aims to simplify and accelerate the training and simulation workflows for robots, cutting development cycle times from months to under a week. Tasks that OSMO manages include generating synthetic data, training models, conducting reinforcement learning, and implementing software-in-the-loop testing at scale for various robotic applications.

Data capture remains a crucial aspect of developing humanoid robots, and NVIDIA showcased an AI- and Omniverse-enabled teleoperation reference workflow at the conference. Researchers can now generate large amounts of synthetic motion and perception data from a minimal amount of teleoperated human demonstrations. This process utilises Apple Vision Pro to capture a few teleoperated demonstrations, which are then simulated in NVIDIA Isaac Sim using the MimicGen NIM microservice to produce synthetic datasets.

The synthetic and real data collected is used to train the Project GR00T humanoid foundation model. This approach allows developers to save time and reduce costs. They employ the Robocasa NIM microservice in Isaac Lab to generate new experiences to retrain the model. Throughout this workflow, NVIDIA OSMO handles the assignment of computing jobs across various resources, further optimising the process.

Alex Gu, CEO of Fourier, a company that develops general-purpose robot platforms, expressed optimism about the new tools. "Developing humanoid robots is extremely complex, requiring an incredible amount of real data, tediously captured from the real world," said Gu. "NVIDIA's new simulation and generative AI developer tools will help bootstrap and accelerate our model development workflows."

NVIDIA is also expanding access to its humanoid developer technologies through three main computing platforms: NVIDIA AI supercomputers for model training, NVIDIA Isaac Sim built on Omniverse for robot skill refinement in simulated environments, and NVIDIA Jetson Thor humanoid robot computers to run these models. A new NVIDIA Humanoid Robot Developer Program will offer early access to these new tools, along with the latest updates to NVIDIA Isaac Sim, NVIDIA Isaac Lab, Jetson Thor, and Project GR00T.

Several companies have already joined the early-access program, including 1x, Boston Dynamics, ByteDance Research, Field AI, Figure, Fourier, Galbot, LimX Dynamics, Mentee, Neura Robotics, RobotEra, and Skild AI. Aaron Saunders, CTO of Boston Dynamics, commented on the collaboration: "Boston Dynamics and NVIDIA have a long history of close collaboration to push the boundaries of what's possible in robotics. We're really excited to see the fruits of this work accelerating the industry at large, and the early-access program is a fantastic way to access best-in-class technology."

Developers can join the NVIDIA Humanoid Robot Developer Program immediately to access NVIDIA OSMO and Isaac Lab. NVIDIA NIM microservices will become available shortly thereafter.