Senior System Software Engineer, DGX Cloud AI Infrastructure

at Nvidia

USD 224,000-431,200 per year

SENIOR

✅ On-site

Tech Stack
Tag name is followed by "@" symbol and proficiency level value. About proficiency levels:

1-2 — basic awareness. Minimal hands-on experience, and a rudimentary understanding of the technology's purpose;

3-6 — daily use. Comfortable and regular usage, capable of handling common tasks and challenges related to the technology;

7-9 — you are an expert, you can teach others, you know all the pitfalls and tricks;

10 — exceptional knowledge, comprehensive understanding, and adeptness in all aspects of the technology, including advanced problem-solving. Think twice before claiming or demanding such level.

AI @ 4 API CI/CD @ 4 GPU @ 4 JAX @ 4 Kubernetes @ 7 LLM @ 4 Machine Learning @ 4 Mentoring Networking @ 7 Performance Analysis @ 4 PyTorch @ 4 Slurm @ 7 TensorFlow @ 4

Details

NVIDIA’s DGX Cloud team builds the infrastructure behind large-scale AI training for NVIDIA’s own training workloads and for its customers. The team builds services, tools, and open-source software that make machine learning workloads more efficient and resilient.

The Recovery and Resilience team is seeking a Senior Software Engineer to own AI infrastructure services, tools, and open-source projects supporting resilient and efficient training from design through production. The role includes mentoring engineers and helping determine how the team operates and delivers its systems.

Responsibilities

Drive the design, development, launch, and operation of services, tools, and open-source projects.
Plan and assign team work, mentor junior engineers, and review designs.
Build resilient AI infrastructure on Kubernetes and Slurm for NVIDIA’s internal and customer training workloads, including supercomputers at major cloud providers and on-premises clusters.
Build APIs connecting to NVIDIA’s resiliency systems, including telemetry, break/fix automation, checkpointing, and execution systems.
Build a job submission layer that hides infrastructure failures from users and automatically restarts long-running training jobs.
Build agentic tools that help customers improve training efficiency by finding bottlenecks, diagnosing failures, and recommending fixes.
Drive open-source initiatives by releasing components for customers and the community and collaborating with them to improve those components.
Establish operational practices for services, including support, monitoring, alerting, CI/CD, integration test gates, and on-call plans.

Requirements

Bachelor’s, master’s, or PhD in Computer Science or a related field, or equivalent experience, plus 12 or more years of experience.
Proven experience taking large systems from design through production.
Strong experience with Kubernetes and/or Slurm at scale.
Hands-on experience with PyTorch, TensorFlow, JAX, or Ray.
Strong understanding of NVIDIA GPUs and networking, including their failure modes.
Experience operating production services, including CI/CD, monitoring, alerting, and on-call responsibilities.

Preferred Qualifications

Hands-on experience with fault tolerance or checkpointing for large-scale distributed training on GPU supercomputers.
Demonstrated experience building LLM-based agents or AI tools for performance analysis or troubleshooting.
Experience building external services or maintaining open-source projects in machine learning infrastructure.

Compensation and Benefits

The expected base salary ranges are $224,000–$356,500 for Level 5 and $272,000–$431,250 for Level 6. Compensation is determined based on location, experience, and other objective job-related factors. The position also includes eligibility for equity and benefits.

Applications will be accepted at least until November 10, 2026. This posting is for an existing vacancy. NVIDIA uses AI tools in its recruiting processes and is an equal opportunity employer.