Senior Software Engineer, DGX Cloud AI Infrastructure

Details

NVIDIA is looking for a Senior Software Engineer to lead the bring-up, triage, benchmarking, analysis, and optimization of distributed training and inference workloads across NVIDIA GPU platforms at large scale. The role operates at the intersection of deep learning systems, GPU performance, distributed computing, and large-scale operations.

The engineer will set technical direction across communication libraries, model frameworks, and inference and training stacks to ensure large language model workloads run efficiently and reliably. This is a hands-on senior individual-contributor role with technical leadership and mentoring responsibilities.

Responsibilities

• Lead bring-up, validation, and debugging of large-scale AI clusters, infrastructure, and end-to-end workloads.
• Bring up, tune, and benchmark AI pre-training, post-training, and inference workloads using PyTorch, NeMo / Megatron, TensorRT-LLM, and adjacent NVIDIA AI software stacks.
• Profile and optimize workload performance across compute, memory, networking, and communication layers using Nsight Systems, NCCL tests, and custom microbenchmarks.
• Analyze scaling efficiency for distributed LLM workloads using data, tensor, pipeline, and expert parallelism across modern GPU clusters.
• Own root-cause analysis of complex failures, including hangs, performance regressions, and topology sensitivity in large distributed environments.
• Define and build resilience and failure-attribution capabilities for detecting, triaging, and attributing node, fabric, and workload failures across clusters.
• Build repeatable benchmark suites, automation, acceptance criteria, and qualification workflows for new platforms.
• Tune runtime settings, communication parameters, and deployment configurations in partnership with framework, systems, and platform teams.
• Deliver data-driven recommendations based on profiling, benchmark results, and cluster characterization.
• Mentor engineers, drive technical standards, and contribute across the performance and infrastructure organization.

Requirements

• Bachelor’s or Master’s degree in Computer Science or a related technical field, or equivalent experience.
• 8+ years of experience developing software infrastructure for large-scale AI or HPC systems, including technical leadership experience.
• Expertise debugging and triaging AI applications across the full stack, from the application layer to the hardware.
• Deep hands-on experience with NCCL, CUDA-aware distributed execution, and debugging multi-GPU and multi-node workloads at scale.
• Proven experience architecting, debugging, and scaling large-scale distributed systems.
• Expert-level Python and C/C++ programming skills.
• Experience operating workloads in scheduled, containerized cluster environments.
• Excellent analytical, debugging, and communication skills, with the ability to influence across teams.

Preferred Qualifications

• Demonstrated experience debugging and optimizing AI workloads at large scale.
• Deep familiarity with the RDMA software stack, including NCCL, IB verbs, UCX, and libfabric.
• Strong knowledge of GPU cluster fabrics and topology, including NVLink, NVSwitch, PCIe, RoCE, and InfiniBand.
• Experience building acceptance tests, benchmark harnesses, regression gates, or cluster qualification tooling for AI platforms.
• Experience building resilience, fault-detection, or failure-attribution systems for datacenter-scale infrastructure.

Benefits

The role includes eligibility for equity and NVIDIA benefits. NVIDIA is committed to fostering an inclusive work environment and is an equal opportunity employer.

Compensation

The base salary range is USD 184,000–287,500 for Level 4 and USD 224,000–356,500 for Level 5. Salary is determined based on location, experience, and the pay of employees in similar positions. Applications will be accepted at least until June 8, 2026.