Software Engineer, Performance Instrumentation

Details

NVIDIA has been transforming computer graphics, PC gaming, and accelerated computing for more than 25 years. Today, NVIDIA is using AI to define the next era of computing, with GPUs powering computers, robots, and self-driving cars.

NVIDIA is seeking a Software Engineer, Performance Instrumentation to join its team in Santa Clara, California. The role focuses on innovation at the intersection of AI and accelerated computing.

Responsibilities

• Define and maintain performance data requirements from sources including profiling libraries, annotation APIs, frameworks, runtimes, communication libraries, schedulers, configurations, and infrastructure telemetry.
• Identify and prove visibility gaps with concrete evidence, demonstrating which diagnoses are not possible with current data.
• Collaborate with relevant teams to address visibility gaps or build solutions where necessary.
• Inform the development of low-overhead instrumentation and annotation APIs from the consumer perspective, ensuring that necessary information is conveyed efficiently and effectively.
• Ensure data quality at ingestion by implementing validation, sanitization, checks, and alerting mechanisms.
• Measure and manage the overhead and observer effects of instrumentation on real workloads.
• Collaborate with multiple partner teams, manage commitments, and ensure continuous progress.

Requirements

• Bachelor's degree in Computer Science, Computer Engineering, or a related field, or equivalent experience.
• At least 8 years of relevant software engineering experience, including hands-on experience with instrumentation or tracing in production machine learning or systems software.
• Proficiency in CUDA.
• Working knowledge of machine learning frameworks such as PyTorch or JAX, and annotation or tracing APIs such as NVTX.
• Experience maintaining data quality in a pipeline, including validation, checks, alerting, and provenance.
• Ability to measure instrumentation overhead and make trade-offs among overhead, fidelity, sampling, and aggregation.
• Experience implementing modifications in code managed by other groups by identifying requirements, reaching consensus on interfaces, and ensuring completion through delivery.
• Strong debugging skills across component boundaries, with the ability to validate changes through code review, tests, and independent measurements, including automated development tools.
• Excellent communication skills, particularly when conveying evidence, uncertainty, and next steps to engineers and data users.
• Proficiency in Python and C++.

Preferred Qualifications

• Experience establishing telemetry conventions or schemas adopted by other teams.
• Background in storage, checkpointing, or data-loading performance.
• Expertise in developing low-overhead event collection.
• Familiarity with network and collective-communication telemetry, including NCCL, NICs, and fabric counters.

Compensation and Benefits

The base salary range is $168,000–$270,250 USD, determined based on location, experience, and the pay of employees in similar positions. The role is also eligible for equity and benefits.

Applications will be accepted at least until November 2, 2026. This posting is for an existing vacancy. NVIDIA uses AI tools in its recruiting processes and is committed to fostering an inclusive work environment and providing equal employment opportunities.