Senior System Architect, Infrastructure Reliability

Details

NVIDIA is seeking a Senior System Architect to solve a complex challenge in accelerated computing: failure attribution at scale. As EDA or equivalent workloads scale across thousands of heterogeneous nodes, a single failure can cause significant resource waste. The engineer will develop and build an automated framework that ingests telemetry from CPU and GPU clusters to identify the root cause of job failures in real time, distinguishing between hardware faults, infrastructure instability, and software defects.

Responsibilities

• Build scalable failure attribution frameworks, including a high-fidelity "flight recorder" for EDA jobs that captures CPU, GPU, and fabric state at the moment of failure.
• Build automated diagnostics that correlate GPU XID errors, PCIe bus failures, and CUDA memory exceptions with system-level events such as OOM kills and NUMA-related hangs.
• Implement low-overhead distributed logging and tracing mechanisms using tracing tools or custom agents across multi-node Slurm or Kubernetes clusters.
• Develop heuristics and machine-learning models to classify failures as hardware faults, software bugs, or environment issues, reducing Mean Time to Identify (MTTI) for R&D teams.
• Work with hardware and infrastructure teams to define signals of impending failure and enable proactive job migration or checkpointing before crashes occur.

Requirements

• Bachelor's, master's, or doctoral degree in Computer Science or Electrical Engineering, or equivalent experience, with 6 or more years of systems programming experience.
• Experience building automated Root Cause Analysis (RCA) pipelines for HPC or cloud-scale environments.
• Expert knowledge of x86/ARM node-level metrics, including IPC (Instructions Per Cycle), cache contention, NUMA imbalance, and hardware interrupts.
• Strong C++ and Python programming skills, including the ability to build high-performance daemons that monitor system health without affecting workload performance.
• Familiarity with cluster resource managers such as Slurm, LSF, or Kubernetes, including job lifecycle and signal propagation management.

Preferred Qualifications

• Expert knowledge of the Linux kernel and error-reporting interfaces such as /dev/mcelog, dmesg, and journald.
• Understanding of how the Linux kernel handles hardware exceptions and memory faults.
• Deep experience with NVIDIA DCGM (Data Center GPU Manager) and NVIDIA Management Library (NVML) for monitoring device health and capturing state dumps.
• Experience with non-intrusive application-health monitoring and syscall-level failure-pattern analysis.
• Experience with checkpoint/restore technologies such as CRIU and their application in long-running EDA flows.

Compensation and Benefits

• Base salary range: $184,000–$287,500 for Level 4.
• Base salary range: $224,000–$356,500 for Level 5.
• Base salary is determined by location, experience, and compensation for similar positions.
• Eligible for equity and benefits.
• This is a hybrid position.
• NVIDIA is an equal opportunity employer committed to an inclusive work environment.