Senior Software Engineer, AI Compute Scheduling

Details

NVIDIA's Managed AI Research Superclusters (MARS) team builds and scales the infrastructure, platforms, and tools that enable researchers and engineers to develop next-generation AI/ML systems. The MARS Scheduling team manages the allocation of NVIDIA's AI research compute across large GPU clusters, multiple clouds, and partners. The environment runs Slurm and Kubernetes at large scale and supports frontier-model pretraining, reinforcement learning, agentic workloads, and inference.

The scheduling stack is topology-sensitive on rack-scale systems and includes fair sharing, quotas, preemption, reservations, backfill, and elastic jobs that recover from failure without losing progress. The role owns production components of the scheduling stack and provides opportunities to contribute to open-source and internal platform code.

Responsibilities

• Design and build scheduling for large GPU fleets on Slurm and Kubernetes, including queueing, gang and topology-aware placement on rack-scale NVLink systems, preemption, reservations, and backfill.
• Build the policy layer for fair share, quotas, and allocations across research teams, driven by a central allocation system and enforced consistently across schedulers.
• Run Slurm as a cloud-native service on Kubernetes and extend Kubernetes batch scheduling, including Kueue and its ecosystem, for training, reinforcement learning, sandboxes, and inference.
• Build services around the scheduler, including reservation automation, node-health-aware placement, workflow orchestration across clusters, and APIs for submitting and tracking work.
• Measure and improve goodput, GPU occupancy, waste, fairness, and time to start for large jobs.
• Own reliability for built components, including on-call responsibilities, incident analysis, and fixes that prevent recurring failures.
• Contribute to Slurm, Kubernetes, and related open-source projects while collaborating with platform, storage, networking, and research teams.

Requirements

• Bachelor's or master's degree in Computer Science, Computer Engineering, Electrical Engineering, or a related field, or equivalent experience.
• At least 6 years of experience building production systems software.
• Deep experience with at least one batch or cluster scheduler, such as Slurm, Kueue, Volcano, KAI, YuniKorn, or a scheduler developed by the candidate; ability to explain scheduler internals rather than only configuration.
• Strong programming skills in Go, C, or C++, plus Python for tooling and analysis.
• Hands-on Kubernetes experience, including controllers or operators, CRDs, scheduling, and resource management.
• Solid Linux systems knowledge, including containers such as Docker, Podman, Enroot, or Apptainer; cgroups; networking basics; and debugging under load.
• Demonstrated ownership of a production service and experience making it measurably better.
• Clear written and spoken communication with engineers and researchers.

Preferred Qualifications

• Experience scheduling large-scale AI training workloads involving thousands of GPUs, NVLink, InfiniBand or RoCE topology, and multi-node jobs that must start together.
• Contributions to Slurm, Kubernetes SIG Scheduling, Kueue, Volcano, KAI, or similar open-source schedulers.
• Experience running Slurm on Kubernetes, such as with Slinky, or building Kubernetes operators for batch or HPC workloads.
• Familiarity with reinforcement learning, agentic, or inference workloads, including heterogeneous jobs, sandboxes, elastic scale, and fast restarts.
• Experience with workflow orchestrators such as Flyte, Airflow, Ray, or Kubeflow at scale.

Compensation and Benefits

The expected base salary ranges are USD 184,000–287,500 for Level 4 and USD 224,000–356,500 for Level 5. Base salary is determined by location, experience, and other objective job-related factors. The position also includes eligibility for equity and benefits.