Senior Research Scientist, Multimodal Foundation Models And Robotics

Details

We are now looking for a Senior Research Scientist focused on Multimodal Foundation Models and Robotics at NVIDIA to build humanoid robot foundation models and systems within the Generalist Embodied Agent Research (GEAR) group. The mission is to build general-purpose embodied agents that learn to explore and master complex skills across both virtual and physical environments.

Responsibilities

• Design and implement novel AI algorithms and models for general-purpose humanoid robots and embodied agents.
• Develop large-scale AI training and inference methods for foundation models.
• Optimize and deploy AI models in physical simulation and on robot hardware.
• Collaborate with research and engineering teams across NVIDIA to transfer research into products and services.

Requirements

• A Ph.D. in Computer Science/Engineering, Electrical Engineering, or equivalent research experience.
• 5 years of relevant work/research experience in one or both of these fields:
  • Multimodal Foundation Models:
    • Hands-on training experience and publications in topics like LLMs, large vision-language models, video generative models, diffusion algorithms, or action-based transformers.
    • Proficiency in rapid prototyping and model training frameworks such as PyTorch, Jax, Tensorflow; Python required; C++ and CUDA skills are a plus.
    • Experience working with large-scale machine learning/AI systems and compute infrastructure.
  • Robotics:
    • Experience in robot learning, e.g., reinforcement learning, imitation learning, classical control methods.
    • Strong programming skills in Python, C++, ROS, and machine learning frameworks like PyTorch.
    • Deep understanding of robot kinematics, dynamics, and sensors.
    • Ability to safely operate robot hardware, lab equipment, and tools.
    • Knowledge of control methods such as PID, model predictive control, and whole-body control.
    • Familiarity with physics simulation frameworks like MuJoCo and Isaac Sim.
    • Experience in robot hardware design and hands-on building.

Benefits

• Competitive base salary ranging from 184,000 USD to 356,500 USD depending on location, experience, and internal comparisons.
• Eligibility for equity and additional benefits.
• Opportunity to be part of cutting-edge research on general-purpose robots and embodied agents.
• Diverse and inclusive work environment.

Additional resources and projects include Eureka, VIMA, Voyager, MineDojo, MimicPlay, Prismer, and Project GR00T.