Principal Engineer for AI Software Resiliency

at Nvidia

📍 Santa Clara, United States

$272,000-419,800 per year

SENIOR
✅ On-site

SCRAPED

Used Tools & Technologies

Not specified

Required Skills & Competences ?

Software Development @ 4 Distributed Systems @ 7 Leadership @ 4 Scoping @ 4 QA @ 4 PyTorch @ 4

Details

We are now looking for a Principal Engineer for AI Resiliency. At NVIDIA, we are pushing the boundaries of what’s possible in AI. We are seeking a Principal Software Engineer to lead the development of AI software resiliency for the most powerful AI supercomputers in the world. As a lead focused on AI Software Resiliency, you will play a pivotal role in defining and implementing critical resiliency features for AI supercomputers at a scale of 100,000+ GPUs. Your expertise will be crucial in driving down cluster downtime towards zero, ensuring that our AI systems remain robust and reliable at all times.

Responsibilities

  • AI Resiliency Expertise: Serve as a trusted authority on AI software resiliency, guiding the architecture, modeling, and scoping of resiliency features to support frontier model training at scale.
  • Feature Development: Lead the execution and development of software resiliency features, including fast checkpoint-recovery, automatic error detection, error isolation, SDC detection and mitigation, and straggler/hang detection.
  • Software Engineering Leadership: Drive engineering excellence by contributing to large software codebases, ensuring high code quality, rigorous testing, and solving complex challenges. Lead others by example, fostering a culture of collaboration, innovation, and continuous improvement.
  • Cross-Team Collaboration: Work closely with multiple teams and stakeholders across NVIDIA to align on mission requirements, provide regular updates, and ensure the successful integration of resiliency features into AI frameworks like PyTorch and JAX/XLA.
  • Customer Engagement: Collaborate directly with major customers to embed AI resilience features into their AI frameworks, ensuring seamless integration and optimal performance.
  • Product Delivery: Partner effectively with TPMs, PMs, and QA teams to ensure the timely and successful launch of resiliency features.

Requirements

  • Master’s or Ph.D. in Computer Science, Electrical Engineering, Computer Engineering, or a related field from a reputed institution, or equivalent experience.
  • A minimum of 10 years of experience in systems architecture or related fields, with a deep understanding of distributed systems and large-scale AI infrastructure.
  • At least 10 years of hands-on experience in software development for distributed systems and 5 years in developing AI frameworks such as PyTorch or JAX/XLA.
  • Proven track record of working effectively across multiple engineering fields and communicate complex technical concepts to a diverse set of collaborators.

Benefits

NVIDIA is recognized as one of the world’s most desirable technology employers, home to some of the most forward-thinking and hardworking people in the world. If you’re creative, autonomous, and driven by innovation, we want to hear from you! The base salary range is 272,000 USD - 419,750 USD. You will also be eligible for equity and benefits.