Software Engineer, ML Performance

Details

NVIDIA is seeking a Software Engineer, ML Performance to work on optimizing distributed AI training and inference workloads at scale. The role involves analyzing performance across the full stack, identifying bottlenecks, evaluating automated diagnoses, and partnering with teams to drive product improvements.

Responsibilities

• Analyze the performance of distributed training and inference workloads across the entire stack, including models and frameworks, data loading and checkpointing, compilers and kernels, CUDA and NCCL, hosts, and networks.
• Identify bottlenecks, resolve them or recommend solutions, and provide supporting evidence.
• Define performance metrics tracked across fleets of GPUs and break them down into factors engineers can use for debugging.
• Determine which signals the platform should collect first.
• Drive regression analysis by segmenting workloads across machines, versions, and configurations to isolate causes.
• Develop reusable analysis methods for other engineers.
• Evaluate automated diagnoses for accuracy, misleading results, and missed problems.
• Build labeled cases to improve the reliability of automated analysis.
• Work closely with partners on real workloads, carry findings through to improvements, and provide product feedback for the roadmap.

Requirements

• Bachelor's degree in Computer Science, Computer Engineering, or a related field, or equivalent experience.
• 12+ years of relevant software engineering experience, including hands-on optimization of GPU training or inference workloads in production.
• Demonstrated performance work across multiple layers of the stack, including clear documentation of baselines, evidence, alternative explanations, and validated outcomes.
• Experience diagnosing performance across many jobs, machines, or tenants rather than tuning only a single workload.
• Familiarity with framework, runtime, and compiler interfaces sufficient to determine where signals should be measured.
• Ability to distinguish symptoms from causes, recognize noisy or incomplete measurements and measurement effects, and determine when evidence is insufficient.
• Strong programming and debugging skills, demonstrated through maintainable analysis code, tests, or repeatable investigation procedures, including when using automated development tools.
• Ability to clearly communicate findings, trade-offs, and remaining uncertainty to workload owners.
• Proficiency with Python and CUDA.

Preferred Qualifications

• Experience improving both training and inference operations.
• Deep knowledge of one or more of PyTorch, JAX, Megatron, NCCL, TensorRT-LLM, vLLM, SGLang, or Dynamo.
• Experience across application, system software, driver, or hardware roles, with a full-stack perspective.
• Statistical judgment applied to noisy, multi-node, fleet-scale performance data.
• Experience with counterfactual analysis and workload segmentation for diagnosing regressions.
• Experience building debugging or performance-analysis tools used by other engineers.

Compensation and Benefits

The base salary range is USD 224,000–356,500, depending on location, experience, and the pay of employees in similar positions. The role is also eligible for equity and benefits.

Applications will be accepted at least until November 2, 2026. This posting is for an existing vacancy. NVIDIA uses AI tools in its recruiting processes and is an equal opportunity employer.