Distinguished Engineer, Apache Spark

at Nvidia
📍 Santa Clara, United States
USD 308,000-471,500 per year
SENIOR
✅ On-site

SCRAPED

Used Tools & Technologies

Not specified

Required Skills & Competences ?

Software Development @ 7 Kafka @ 6 Kubernetes @ 8 Scala @ 4 Spark @ 4 Java @ 4 Distributed Systems @ 4 Flink @ 6 AWS @ 4 AWS EMR @ 4 Communication @ 7 Hadoop @ 4 Databricks @ 4 Hive @ 4 Oracle @ 4 CUDA @ 4 GPU @ 4

Details

NVIDIA is seeking a Distinguished Engineer for the Apache Spark Acceleration group. You will serve as a hands-on architect of the NVIDIA Spark Acceleration Group, working with a team of distributed systems engineers including PMC and committers of Apache Spark, Apache Hadoop, Apache Hive, and Apache Arrow. You will engage in open-source projects such as Apache Spark, RAPIDS, Apache Iceberg, Delta Lake, UCX and more, and work with partners to deploy GPU-enabled data analytics solutions in public cloud or on-premises clusters.

Responsibilities

  • Lead the architecture, design and implementation of accelerated Apache Spark and related big-data frameworks.
  • Engage open source communities (including Apache Spark, RAPIDS, Apache Iceberg, Delta Lake and UCX) for technical discussion and contribution, and engage new communities where NVIDIA may not have a strong presence yet.
  • Work with NVIDIA partners to deploy GPU-enabled data analytics solutions in public cloud or on-premises clusters (AWS EMR, Databricks, Google Dataproc, Oracle Cloud Data Flow, Bytedance Volcengine, Tencent Cloud, Cloudera integrations are mentioned).
  • Present technical solutions at industry conferences and meetups.
  • Collaborate with distributed systems teams to define solutions to distributed processing problems and challenges at large scale.
  • Provide recommendations and feedback to teams regarding topics such as infrastructure, continuous integration and testing strategy.
  • Build, test and optimize CUDA/C++ libraries across different platforms.
  • Build automation and tools that increase the productivity of teams developing distributed systems.
  • Mentor members of the engineering team.

Requirements

  • BS, MS, or PhD in Computer Science, Computer Engineering, or closely related field (or equivalent experience).
  • 17+ years of work or research experience in software development.
  • Prior experience delivering complex software projects as a lead architect.
  • Outstanding technical skills in designing and implementing high-quality distributed systems.
  • Excellent programming skills in C++, Java, and/or Scala.
  • Highly motivated with strong interpersonal and communication skills.
  • 5+ years working experience with key open source big-data projects as a contributor or committer (examples listed: Apache Spark, Apache Hadoop, Apache Flink, Apache Kafka, Apache Hive, Apache Arrow, Delta Lake).
  • Excellent knowledge about distributed system schedulers: Kubernetes, Hadoop YARN, Apache Spark.
  • Ability to quickly learn new areas and work with teams across boundaries and geographies.

Ways to stand out

  • Experience designing and developing columnar query engines is a huge plus.
  • Committer status at major open source projects (such as Apache Spark, Apache Hadoop, Apache Flink) is a big plus.
  • Working experience with acceleration libraries (CUDA, RAPIDS, UCX) is helpful.

Compensation & Benefits

  • Base salary range: 308,000 USD - 471,500 USD (will be determined based on location, experience, and pay of employees in similar positions).
  • Eligible for equity and benefits (link to NVIDIA benefits provided in original posting).

Other information

  • Applications for this job will be accepted at least until August 5, 2025.
  • NVIDIA is committed to fostering a diverse work environment and is an equal opportunity employer. The company does not discriminate based on legally protected characteristics.

#deeplearning