Senior High-Performance LLM Training Engineer

at Nvidia

📍 Santa Clara, United States

USD 184,000-356,500 per year

SENIOR

✅ Hybrid

Tech Stack
Tag name is followed by "@" symbol and proficiency level value. About proficiency levels:

1-2 — basic awareness. Minimal hands-on experience, and a rudimentary understanding of the technology's purpose;

3-6 — daily use. Comfortable and regular usage, capable of handling common tasks and challenges related to the technology;

7-9 — you are an expert, you can teach others, you know all the pitfalls and tricks;

10 — exceptional knowledge, comprehensive understanding, and adeptness in all aspects of the technology, including advanced problem-solving. Think twice before claiming or demanding such level.

AI @ 4 CUDA @ 6 Deep Learning @ 7 GPU @ 3 JAX LLM Performance Analysis @ 4 PyTorch Python @ 6

Details

NVIDIA is seeking an experienced engineer specializing in performance analysis and optimization to improve the efficiency of large language model training workloads. The role focuses on optimizing NVIDIA's high-performance LLM software stack, including frameworks such as PyTorch and JAX, for training on thousands of GPUs, while helping shape hardware roadmaps for future GPU generations.

Responsibilities

Understand, analyze, profile, and optimize AI training workloads on innovative hardware and software platforms.
Analyze the overall training performance of GPUs, prioritize performance problems, and solve issues across state-of-the-art neural networks.
Implement production-quality software across multiple layers of NVIDIA's deep learning platform stack, from drivers to deep learning frameworks.
Build and support NVIDIA submissions to the MLPerf Training benchmark suite.
Implement key deep learning training workloads in NVIDIA's proprietary processor and system simulators to enable future architecture studies.
Build tools to automate workload analysis, workload optimization, and other critical workflows.

Requirements

PhD in Computer Science, Electrical Engineering, or Computer Engineering with 5+ years of experience; or an MS degree or equivalent experience with 8+ years of meaningful work experience.
Strong background in deep learning and neural networks, particularly training.
Deep knowledge of computer architecture and familiarity with GPU architecture fundamentals.
Proven experience analyzing and tuning application performance, as well as processor- and system-level performance modeling.
Programming skills in C++, Python, and CUDA.

Benefits

Base salary range of USD 184,000–287,500 for Level 4.
Base salary range of USD 224,000–356,500 for Level 5.
Eligibility for equity and benefits.
Opportunity to collaborate across the hardware and software stack, from GPU architecture to application code, in an environment that encourages innovation.
NVIDIA is committed to fostering a diverse work environment and is an equal opportunity employer.