Senior Software Engineer, AI Data

Details

NVIDIA's Managed AI Research Superclusters (MARS) team builds and scales the infrastructure, platforms, and tools that enable researchers and engineers to develop next-generation AI/ML systems. The MARS AI Data team manages petabytes of training data, checkpoints, and datasets across multiple clouds and partners, building the layer between storage and workloads.

This role will help design MARS's data platform on Kubernetes and work directly with researchers training NVIDIA's models. The work will improve training speed, restart times, and storage reliability.

Responsibilities

• Design and build the AI data layer for a multi-cloud GPU fleet, including dataset, checkpoint, and scratch-data placement, caching, tiering, and movement between storage backends.
• Build high-throughput data paths for training and reinforcement learning, including streaming loaders from object storage, distributed caches, and checkpoint save and restore across thousands of GPUs.
• Implement peer-to-peer and local-first checkpoint restoration.
• Build Kubernetes-native APIs, operators, and storage integrations so jobs can declare required data and the platform can mount, cache, and account for it on Slurm and Kubernetes.
• Own quotas, capacity, and data lifecycle across hundreds of research teams, including usage attribution, quota enforcement, retention, stale-data cleanup, and capacity forecasting.
• Build I/O telemetry and access analytics to identify hot and cold data, metadata pressure, and I/O stalls, and use the results to improve data placement and performance.
• Own reliability for the systems built, including on-call responsibilities, incident analysis, and changes that prevent data incidents such as full file systems, runaway metadata load, and lost checkpoints.

Requirements

• Bachelor's or master's degree in Computer Science, Computer Engineering, Electrical Engineering, or a related field, or equivalent experience.
• Five or more years of experience building production systems or data-infrastructure software.
• Deep experience with at least one of the following:
  • Parallel or distributed file systems such as Lustre, WEKA, VAST, GPFS, or Ceph.
  • Object storage at scale, including S3 or compatible systems.
  • Large-scale data pipelines and data lakes.
• Clear understanding of I/O performance, including throughput, latency, metadata cost, caching, and access patterns.
• Strong programming skills in Go, Rust, C, or C++, plus Python for tooling and analysis.
• Track record of owning a production service and making it measurably better.
• Clear written and spoken communication skills with engineers and researchers.

Preferred Qualifications

• Experience supporting large-scale AI training, including data loaders, streaming datasets, and storage behavior across thousands of GPUs.
• Experience with asynchronous or distributed checkpointing, tiered checkpoint storage, and fast restart after failure.
• Experience building or operating a distributed cache or AI-oriented storage system, or using node-local NVMe as a managed tier.
• Experience building quota, accounting, or capacity systems used by multiple teams.
• Knowledge of reinforcement learning and agentic workloads, including their requirements for many small reads, environment snapshots, sandbox images, and fast fan-out.

Compensation and Benefits

The expected base salary ranges are $152,000–$241,500 for Level 3 and $184,000–$287,500 for Level 4, depending on location, experience, and other objective job-related factors. The position is also eligible for equity and benefits.

Applications will be accepted at least until November 13, 2026. This posting is for an existing vacancy. NVIDIA uses AI tools in its recruiting processes and is an equal opportunity employer.