Senior Systems Software Engineer, Spark Service - Accelerated Spark

at Nvidia

📍 Santa Clara, United States

$220,000-339,200 per year

SENIOR
✅ On-site

SCRAPED

Used Tools & Technologies

Not specified

Required Skills & Competences ?

Software Development @ 7 Kubernetes @ 4 Python @ 4 Scala @ 4 Spark @ 4 GCP @ 4 Java @ 4 Algorithms @ 4 Machine Learning @ 4 Data Science @ 4 TensorFlow @ 4 Azure @ 4 GCP Dataproc @ 4 gRPC @ 3 Helm @ 4 API @ 4 BI @ 4 Databricks @ 4 PyTorch @ 4 XGBoost @ 4 Spring Boot @ 4

Details

We are seeking expert Senior System Software Engineers adept at Apache Spark to join our team. Data scientists spend a considerable amount of time exploring data and iterating over machine learning (ML) experiments. Every hour of compute required to sort through datasets, extract features and fit ML algorithms impedes an efficient business workflow. NVIDIA believes that data science workflows can benefit tremendously from being accelerated, to enable data scientists to explore many more and larger datasets to drive towards their business goals, faster and more efficiently.

Responsibilities

  • Design and develop a world-class GPU accelerated Apache Spark service.
  • Enable a collection of micro-services to provide the ability to run Spark applications on Kubernetes or other platforms.
  • Implement the REST API and its client libraries to simplify customer adoptions.
  • Customize the open-source projects to meet the project requirements.
  • Deploy and verify the solution on CSPs or on-prem Kubernetes environments.
  • Engage open source communities, including Apache Spark and RAPIDS, for technical discussions and contributions.

Requirements

  • 8+ years of experience in software development.
  • 5+ years hands-on experience with web service design and development.
  • BS/MS/PhD in computer science or a related field or equivalent experience.
  • Experience with REST service frameworks like Spring Boot.
  • Familiarity with the modern data open source ecosystem (Apache Spark, Apache Kyuubi, Apache Zookeeper, gRPC, etc).
  • Experience with Kubernetes and Helm charts and building performance and reliable service APIs.
  • Experience working on public and private cloud platforms and with object storage and distributed file systems.
  • Prior experience supporting enterprise customers.
  • Solid understanding of Python and Scala/Java.
  • Excellence at communicating, presenting and explaining technical topics.

Ways to stand out from the crowd:

  • Working experience with Spark distributions: Databricks, AMS EMR, GCP Dataproc, Azure Synapse Analytics.
  • Contributions to major open source projects such as Apache Spark, Apache Kyuubi, Apache Ranger, Apache Iceberg, and Delta Lake.
  • Development experience of Apache Spark Data Sources and connectors.
  • Development experience with Spark Client interfaces/tools like Jupyter, Zeppelin, Spark Connect, and BI tools.
  • Working knowledge of secrets and encryption management systems and basic ML/DL experience with PyTorch, TensorFlow, Spark ML and XGBoost.

NVIDIA is widely considered to be one of the technology world’s most desirable employers. We have some of the most forward-thinking and hardworking people on the planet working for us. If you're creative, passionate and self-motivated, we want to hear from you! NVIDIA is leading the way in groundbreaking developments in Artificial Intelligence, High-Performance Computing and Visualization. The GPU, our invention, serves as the visual cortex of modern computers and is at the heart of our products and services.