Senior Software Architect - Deep Learning And Hpc Communications

at Nvidia

USD 224,000-431,200 per year

SENIOR

✅ On-site

Tech Stack
Tag name is followed by "@" symbol and proficiency level value. About proficiency levels:

1-2 — basic awareness. Minimal hands-on experience, and a rudimentary understanding of the technology's purpose;

3-6 — daily use. Comfortable and regular usage, capable of handling common tasks and challenges related to the technology;

7-9 — you are an expert, you can teach others, you know all the pitfalls and tricks;

10 — exceptional knowledge, comprehensive understanding, and adeptness in all aspects of the technology, including advanced problem-solving. Think twice before claiming or demanding such level.

AI @ 6 Algorithms @ 6 CUDA @ 4 Communication @ 4 Debugging @ 6 Deep Learning @ 7 GPU HPC @ 7 InfiniBand @ 4 Linux @ 7 MPI @ 4 NCCL @ 4 NVLink @ 4 Networking Parallel Programming @ 4 PyTorch @ 7 System Architecture @ 7 TensorFlow @ 7

Details

NVIDIA is leading groundbreaking developments in Artificial Intelligence, High Performance Computing and Visualization. The GPU serves as the visual cortex of modern computers and is at the heart of NVIDIA products and services. NVIDIA’s GPU Communications Libraries and Networking team builds communication libraries like NCCL, NVSHMEM, and UCX that are crucial for scaling Deep Learning and HPC.

DL and HPC applications have huge compute demands and already run at scales of up to tens of thousands of GPUs. GPUs are connected with high-speed interconnects (e.g. NVLink, PCIe) within a node and with high-speed networking (e.g. InfiniBand, Ethernet) across nodes. Efficient and fast communication between GPUs directly impacts end-to-end application performance, which continues to grow with the increasing scale of next generation systems.

This role is to co-design next-gen data center platforms and scalable communications software.

Responsibilities

Investigate opportunities to improve communication performance by identifying bottlenecks in today’s systems.
Design and implement new communication technologies to accelerate AI and HPC workloads.
Explore innovative solutions in HW and SW for next generation platforms as part of co-design efforts involving GPU, Networking, and SW architects.
Build proofs-of-concept, conduct experiments, and perform quantitative modeling to evaluate and drive new innovations.
Use simulation to explore performance of large GPU clusters (think scales of 100s of 1000s of GPUs).

Requirements

M.S./Ph.D. degree in CS/CE or equivalent experience.
12+ years of relevant experience.
Excellent C/C++ programming and debugging skills.
Experience with parallel programming models (MPI, SHMEM) and at least one communication runtime (MPI, NCCL, NVSHMEM, OpenSHMEM, UCX, UCC).
Deep understanding of operating systems, computer and system architecture.
Solid fundamentals of network architecture, topology, algorithms, and communication scaling relevant to AI and HPC workloads.
Strong experience with Linux.
Ability and flexibility to work and communicate effectively in a multi-national, multi-time-zone corporate environment.

Ways to stand out from the crowd

Expertise in related technology and passion for what you do, including experience with CUDA programming and NVIDIA GPUs. Knowledge of high-performance networks like InfiniBand, RoCE, NVLink, etc.
Experience with Deep Learning Frameworks such as PyTorch and TensorFlow; knowledge of deep learning parallelisms and mapping to the communication subsystem; experience with HPC applications.
Strong collaborative and interpersonal skills and a proven track record of effectively guiding and influencing within a dynamic and multi-functional environment.