Senior Engineer - AI and HPC Observability

Details

NVIDIA is a pioneer in accelerated computing, known for inventing the GPU and driving breakthroughs in gaming, computer graphics, high-performance computing, and artificial intelligence. Our technology powers everything from generative AI to autonomous systems. Within this mission, the Managed AI Superclusters (MARS) team builds and scales the infrastructure, platforms, and tools that enable researchers and engineers to develop the next generation of AI/ML systems.

Observability is at the heart of this transformation. This role focuses on designing and building the next-generation observability platform for large-scale AI workloads, GPU clusters, and high-performance computing (HPC) environments. The role blends deep technical engineering with large-scale data systems to develop scalable telemetry pipelines, AI-driven insights, and intelligent monitoring across NVIDIA's GPU infrastructure.

Responsibilities

• Design and implement full-stack observability systems covering metrics, logs, traces, and events for GPU-powered AI and HPC workloads.
• Build large-scale telemetry data pipelines leveraging OpenTelemetry, Kafka, Prometheus, and other distributed systems to ingest, process, and analyze massive data streams.
• Develop analytics and anomaly detection frameworks to enable real-time visibility, performance optimization, and predictive insights across multi-tenant environments.
• Architect and tune high-throughput data stores (e.g., TSDBs, columnar databases, OLAP systems) for large-scale observability data.
• Drive self-service analytics capabilities through APIs, dashboards, and recommendation engines that empower developers and operators with actionable insights.
• Collaborate with AI platform, GPU, and cloud infrastructure teams to optimize observability for model training, inference workloads, and HPC performance.
• Leverage machine learning and statistical techniques for correlation, anomaly detection, and intelligent alerting.
• Contribute to performance tuning, scalability, and reliability of observability services across on-prem and cloud environments.

Requirements

• BS or equivalent experience in Computer Science, Computer Engineering, or a related technical field.
• 8+ years of experience in large-scale observability, data engineering, or performance monitoring systems.
• Proven expertise in building and scaling observability stacks (metrics, logs, traces, events) using OpenTelemetry, Prometheus, Grafana, or Thanos.
• Deep understanding of data collection, transformation, and storage at scale; experience with streaming frameworks (Kafka, Flink, Spark) preferred.
• Hands-on experience with Python, Go, and/or Java for backend development and automation.
• Strong knowledge of API design, data modeling, SQL/NoSQL, and data pipeline architecture.
• Experience working with PromQL, time-series databases, and large-scale monitoring systems.
• Familiarity with AI/ML pipelines, GPU-based workloads, and HPC environments.
• Experience with anomaly detection, log analytics, and recommendation systems using ML or statistical techniques.
• Excellent problem-solving, debugging, and performance-tuning skills in distributed systems.

Ways To Stand Out

• Proven experience designing and scaling full-stack observability platforms for large-scale AI, GPU, or HPC environments.
• Hands-on expertise with OpenTelemetry, Prometheus, Kafka, and distributed data pipelines handling high-volume telemetry streams.
• Strong background in data engineering, performance tuning, and time-series data modeling for real-time analytics.
• Demonstrated use of machine learning or statistical techniques for anomaly detection, correlation, or intelligent alerting.
• Deep understanding of API design, self-service observability, and building platforms that empower internal developers and operators.

Compensation & Benefits

Your base salary will be determined based on your location, experience, and the pay of employees in similar positions. The base salary range is 184,000 USD - 287,500 USD for Level 4, and 224,000 USD - 356,500 USD for Level 5. You will also be eligible for equity and benefits (see NVIDIA benefits page).

Applications for this job will be accepted at least until October 24, 2025.

Equal Opportunity

NVIDIA is committed to fostering a diverse work environment and is an equal opportunity employer. We do not discriminate in hiring and promotion practices on the basis of race, religion, color, national origin, gender, gender expression, sexual orientation, age, marital status, veteran status, disability status, or any other characteristic protected by law.