Senior Systems Software Engineer - Accelerated Spark Applications

at Nvidia

USD 184,000-287,500 per year

SENIOR

✅ On-site

Tech Stack
Tag name is followed by "@" symbol and proficiency level value. About proficiency levels:

1-2 — basic awareness. Minimal hands-on experience, and a rudimentary understanding of the technology's purpose;

3-6 — daily use. Comfortable and regular usage, capable of handling common tasks and challenges related to the technology;

7-9 — you are an expert, you can teach others, you know all the pitfalls and tricks;

10 — exceptional knowledge, comprehensive understanding, and adeptness in all aspects of the technology, including advanced problem-solving. Think twice before claiming or demanding such level.

AI @ 4 API AWS CUDA @ 4 Data Pipelines GPU @ 4 Helm Java @ 7 Kubernetes @ 4 Presto @ 6 Python @ 7 Scala @ 7 Software Development @ 7 Spark @ 4

Details

NVIDIA is seeking a Senior Systems Software Engineer for the Apache Spark Acceleration group. The team develops and operates GPU-accelerated Apache Spark services for production data pipelines. The service runs thousands of jobs per hour on Kubernetes and AWS under a customer SLA and packages the cuDF Spark plugin and a CPU/GPU workload router into an API-based service built on Spark Connect, Kyuubi, Helm, and Flux.

Responsibilities

Design, build, and operate a hosted Spark service control plane on Kubernetes, including the Java REST API, Spark Connect runner, Kyuubi, Helm charts, and Flux deployments.
Own code reviews for one or more services end to end and keep them releasable on the release train.
Extend the cuDF Spark runtime and CPU/GPU workload router in Scala and Java, including interoperability with Delta Lake, Apache Iceberg, and Unity Catalog.
Investigate production failures on clusters running more than 1,000 concurrent jobs with customer teams and upstream library teams, and drive fixes to closure.
Instrument metrics, dashboards, and alerts; participate in the on-call rotation; and maintain current runbooks.
Provide recommendations and feedback on infrastructure, continuous integration, and testing strategy.

Requirements

Bachelor's, master's, or PhD in Computer Science, Computer Engineering, or a closely related field, or equivalent experience.
At least 9 years of software development experience.
Strong distributed-systems fundamentals and proficiency in Java or Scala, plus C++ or Python.
Hands-on experience operating services on Kubernetes in on-premises or cloud environments.
Experience owning or reviewing long-lived servers, including REST front ends, proxies or request routers, and JDBC or Thrift services.
Experience supporting enterprise or internal production customers, including on-call support.
Experience using AI tools in software development.

Preferred Qualifications

Experience with Apache Spark internals, Spark Connect, or Kyuubi.
Experience integrating Delta Lake, Apache Iceberg, or Unity Catalog.
GPU data-processing experience with cuDF or CUDA.
Significant contributions to collaborative data-platform projects such as Apache Spark, Velox, Presto, Apache Arrow, or Apache DataFusion.

Compensation and Benefits

The base salary range is USD 184,000 to USD 287,500 per year, determined by location, experience, and compensation for similar positions. The role also includes eligibility for equity and benefits.