Query Engine Architect - Accelerated Apache Spark

Details

We are seeking an experienced Query Engine Architect to accelerate Apache Spark and related frameworks on GPUs. As Nvidia is leading the world in accelerated computing, we are building the next generation data processing ecosystem. Apache Spark is the most popular distributed data processing engine in data centers. It is used for a wide variety of workloads, from data preparation, feature generation, reporting, analytics, and more. Data scientists spend a considerable amount of time exploring data and iterating over machine learning (ML) experiments. Every hour of compute required to sort through datasets, extract features and fit ML algorithms impedes an efficient business workflow.

At NVIDIA, we are passionate about working on hard problems that have an impact. You will work with the open source community to enable Apache Spark data processing with GPUs. Data workflows can benefit tremendously from being accelerated, enabling data scientists to explore many more and larger datasets to achieve their business goals, faster and more efficiently.

Responsibilities

• Lead the query optimization effort on the RAPIDS Spark team.
• Review each stage of query processing, and identify areas for logical plan and physical plan optimization. Construct optimization of plans taking into account CPU / GPU hardware resources.
• Find opportunities for adaptive query execution that are resource aware, for example, adapting based on CPU or GPU characteristics.
• Identify where operator fusion could drive better performance.
• Review columnar processing engine practices and see how they might apply to GPU based columnar processing.
• Engage open source communities, including Apache Spark and RAPIDS, for technical discussions and contributions.
• Work with Nvidia strategic partners on deploying accelerated data processing solutions in public cloud or on-premise clusters.
• Present technical solutions in industry conferences and meetups.

Requirements

• BS, MS, or PhD in Computer Science, Computer Engineering, or equivalent experience.
• 15+ years of work or research experience in software development.
• 5+ years working with key open source big-data projects as a contributor or committer including Apache Spark, Apache Hadoop, Apache Hive, Apache Flink, Apache Impala, Apache Drill, Apache Calcite, and Substrait.
• Outstanding technical skills in crafting and implementing high-quality distributed systems.
• Deep expertise in database query engines and query optimization.
• Excellent programming skills in C++, Java, and/or Scala.
• Knowledge of distributed system schedulers: Kubernetes, Hadoop YARN, Spark standalone, and/or Mesos.
• Ability to work with multi-functional teams across boundaries and geographies.
• Highly motivated with strong interpersonal skills.

Ways to stand out from the crowd:

• Contributions to major open source projects such as Apache Spark, Apache Hive, Apache Impala, Apache Drill, Substrait, Apache Calcite.
• Working experience with acceleration libraries (CUDA, RAPIDS, UCX).
• Basic ML/DL experience with Spark ML and XGBoost.

We are widely considered to be one of the technology world’s most desirable employers, and as a result have some of the most forward-thinking and hardworking people in the world working for us. If you're passionate, creative, and driven, we’d love to have you join the team. With competitive salaries and a generous benefits package, we are widely considered to be one of the technology world’s most desirable employers. We have some of the most forward-thinking and hardworking people in the world working for us and, due to unprecedented growth, our exclusive engineering teams are rapidly growing. If you’re a creative and autonomous engineer with a real passion for technology, we want to hear from you.