DSX NVCRE Senior Engineer

Details

NVIDIA is seeking a Senior Software Engineer to join its DSX team and build and operate tooling that verifies AI infrastructure is ready for production. The role focuses on cluster readiness and certification software, including the open-source Cluster Readiness Engine, which validates GPU clusters across Kubernetes and Slurm, measures real training and inference performance, and produces trusted evidence for customers and partners.

Responsibilities

• Design and build systems that make cluster validation repeatable and verifiable.
• Develop cluster readiness and certification tooling in Go and Python, including controllers, operators, command-line interfaces, and scheduler integrations for Kubernetes and Slurm.
• Build benchmark and workload catalogs covering communication, compute, storage, diagnostics, training, inference, reliability, and security, with architecture-specific expectations and pass/fail criteria.
• Develop provenance and evidence pipelines that capture execution details, environments, and configurations, producing signed and auditable results.
• Integrate new GPU platforms, schedulers, and fabrics, from single-node bring-up through multi-node production-cluster validation.
• Collaborate with platform, networking, solutions, and cloud partner teams to investigate performance and correctness issues and feed fixes into platforms and reference architectures.
• Contribute to the open-source Cluster Readiness Engine and related projects through design documents, documentation, and code reviews.
• Author design documents, runbooks, and operational guides describing cluster validation and result interpretation.

Requirements

• BS, MS, or Ph.D. in Computer Science, Computer Engineering, Electrical Engineering, or a related technical field, or equivalent experience.
• Eight or more years of software engineering experience building infrastructure or systems software, preferably for large-scale compute, cloud, or HPC environments.
• Strong programming skills in Go and Python, with experience delivering production-quality services, controllers, or command-line tools.
• Deep familiarity with Linux systems, containers, and Kubernetes, including writing operators or controllers against the Kubernetes API.
• Working knowledge of HPC schedulers such as Slurm and GPU cluster fundamentals, including drivers, NCCL, InfiniBand or RoCE fabrics, and GPU health tooling such as DCGM.
• Experience benchmarking or validating distributed systems, with a rigorous approach to measurement, reproducibility, and evidence.
• Ability to lead complex technical investigations to root cause and communicate findings clearly to engineering and leadership audiences.

Preferred Qualifications

• Experience running or tuning large-scale distributed training or inference workloads with PyTorch, Megatron, NeMo, vLLM, SGLang, or TensorRT-LLM.
• Experience with the NVIDIA ecosystem, including DGX systems, GB200 or GB300 platforms, CUDA, NVLink, NVSwitch, and NVIDIA networking.
• Contributions to open-source infrastructure projects, particularly Kubernetes operators, scheduler plugins, or benchmarking frameworks.
• Background in software supply-chain security, attestation, signed artifacts, and audit-grade evidence systems.
• Familiarity with Prometheus, Grafana, OpenTelemetry, GitOps workflows, and infrastructure as code for GPU clusters.

Compensation and Benefits

The base salary range is USD 184,000–287,500 for Level 4 and USD 224,000–356,500 for Level 5. The role is also eligible for equity and benefits.

Applications will be accepted at least until November 1, 2026. NVIDIA is an equal opportunity employer.