Distinguished Engineer, Apache Spark

at Nvidia
📍 Santa Clara, United States
USD 308,000-471,500 per year
SENIOR
✅ On-site

SCRAPED

Used Tools & Technologies

Not specified

Required Skills & Competences ?

Software Development @ 7 Kafka @ 6 Kubernetes @ 8 Scala @ 4 Spark @ 4 Java @ 4 Distributed Systems @ 4 Flink @ 6 AWS @ 4 AWS EMR @ 4 Hadoop @ 4 Databricks @ 4 Hive @ 4 Oracle @ 4 CUDA @ 4 GPU @ 4

Details

NVIDIA is seeking a Distinguished Engineer for the Apache Spark Acceleration group. You will serve as a hands-on architect of the Nvidia Spark Acceleration Group, working with a team of distributed systems engineers (including PMC and committers of Apache Spark, Apache Hadoop, Apache Hive, and Apache Arrow) and engaging with open-source projects such as Apache Spark, RAPIDS, Apache Iceberg, Delta Lake, UCX and more. The goal is to accelerate Spark applications on GPUs without code changes and to enable GPU-accelerated data processing in on-premise and cloud deployments (AWS EMR, Databricks, Google Dataproc, Oracle Cloud Data Flow, Bytedance Volcengine, Tencent Cloud, Cloudera).

Responsibilities

  • Lead the architecture, design and implementation of accelerated Apache Spark and related big-data frameworks.
  • Engage with open source communities (Apache Spark, RAPIDS, Apache Iceberg, Delta Lake, UCX and others) for technical discussion and contribution; help expand NVIDIA's presence in new communities.
  • Work with NVIDIA partners to deploy GPU-enabled data analytics solutions in public cloud and on-premises clusters.
  • Present technical solutions at industry conferences and meetups.
  • Collaborate with distributed systems teams to define large-scale distributed processing solutions.
  • Provide recommendations and feedback on infrastructure, continuous integration and testing strategy.
  • Build, test and optimize CUDA/C++ libraries across different platforms.
  • Build automation and tools to increase productivity of teams developing distributed systems.
  • Mentor members of the engineering team.

Requirements

  • BS, MS, or PhD in Computer Science, Computer Engineering, or closely related field (or equivalent experience).
  • 17+ years of work or research experience in software development.
  • Prior experience delivering complex software projects as a lead architect.
  • Outstanding technical skills in designing and implementing high-quality distributed systems.
  • Excellent programming skills in C++, Java, and/or Scala.
  • 5+ years working experience with key open-source big-data projects as a contributor or committer (examples given: Apache Spark, Apache Hadoop, Apache Flink, Apache Kafka, Apache Hive, Apache Arrow, Delta Lake).
  • Excellent knowledge about distributed-system schedulers: Kubernetes, Hadoop YARN, Apache Spark.
  • Able to delve into new areas quickly and work across teams and geographies.

Preferred / Ways to stand out

  • Experience designing and developing columnar query engines.
  • Committership at major open-source projects (Apache Spark, Apache Hadoop, Apache Flink).
  • Working experience with acceleration libraries and ecosystems: CUDA, RAPIDS, UCX.

Compensation & Benefits

  • Base salary range: 308,000 USD - 471,500 USD (determined based on location, experience, and internal pay equity).
  • Eligibility for equity and company benefits (see NVIDIA benefits pages).

Additional information

  • Location: Santa Clara, CA, United States.
  • Employment type: Full time.
  • Applications accepted at least until August 5, 2025.
  • NVIDIA is an equal opportunity employer committed to fostering a diverse work environment.