Senior Software Engineer, Resilience Engineering - DGX Cloud

at Nvidia

USD 184,000-356,500 per year

SENIOR

✅ On-site

Tech Stack
Tag name is followed by "@" symbol and proficiency level value. About proficiency levels:

1-2 — basic awareness. Minimal hands-on experience, and a rudimentary understanding of the technology's purpose;

3-6 — daily use. Comfortable and regular usage, capable of handling common tasks and challenges related to the technology;

7-9 — you are an expert, you can teach others, you know all the pitfalls and tricks;

10 — exceptional knowledge, comprehensive understanding, and adeptness in all aspects of the technology, including advanced problem-solving. Think twice before claiming or demanding such level.

AI @ 6 GPU @ 6 Go @ 7 Grafana @ 6 HPC @ 6 Leadership @ 7 Observability @ 6 OpenTelemetry @ 6 Prometheus @ 6 Python @ 7 SRE @ 4 Technical Leadership @ 7

Details

NVIDIA is seeking a seasoned software engineer to join its DGX Cloud Resilience Engineering team. The role focuses on defining reliability strategy, improving operational practices, and building resilient systems for large-scale, 24/7 environments.

Responsibilities

Build an organization-wide reliability strategy and guide the maturation of operational practices.
Establish and maintain a rigorous Service Level Objective (SLO) program across teams.
Lead incident response for high-severity incidents, ensuring low-drama and high-signal resolution.
Build and improve production code daily, enhancing the data platform and related tooling.
Implement chaos engineering, failure injection, and resilience testing.
Improve operational standards through hands-on technical leadership and strong engineering practices.

Requirements

Deep, hands-on experience running large-scale production systems with a proven track record.
Detailed understanding of failure modes in large systems, including cascading dependencies and retry storms.
Strong software engineering skills with current, hands-on experience in Go, Python, or similar programming languages.
Proven experience establishing and maintaining an SLO program with operational rigor.
Practical experience in reliability engineering, chaos engineering, and failure injection.
Ability to influence across team boundaries through credibility and expertise.
At least 10 years of industry experience with a Bachelor's or Master's degree, or equivalent experience operating systems at scale.

Preferred Qualifications

Experience in a world-class reliability function, such as Google SRE or Meta production engineering.
Expertise operating GPU, HPC, or AI training infrastructure and managing its failure modes.
Track record of measurable reliability improvements within an organization.
Proficiency with modern observability and operational tools, including Prometheus, OpenTelemetry, Grafana, PagerDuty, and Rootly.

Compensation and Benefits

The expected base salary ranges are:

USD 184,000–287,500 for Level 4
USD 224,000–356,500 for Level 5

The role also includes eligibility for equity and benefits. NVIDIA offers a comprehensive benefits package.

Applications will be accepted at least until November 16, 2026. This posting is for an existing vacancy. NVIDIA uses AI tools in its recruiting processes and is an equal opportunity employer.