Member of Technical Staff (AI Infrastructure Engineer)

Details

We are looking for an AI infrastructure engineer to join the AI Research & Systems team. The role partners closely with Inference and Research teams to build, deploy, and optimize large-scale AI training and inference clusters using Kubernetes, Slurm, Python, C++, PyTorch, and AWS.

Responsibilities

• Design, deploy, and maintain scalable Kubernetes clusters for AI model inference and training workloads.
• Manage and optimize Slurm-based HPC environments for distributed training of large language models.
• Develop robust APIs and orchestration systems for training pipelines and inference services.
• Implement resource scheduling and job management systems across heterogeneous compute environments.
• Benchmark system performance, diagnose bottlenecks, and implement improvements across training and inference infrastructure.
• Build monitoring, alerting, and observability solutions for ML workloads running on Kubernetes and Slurm.
• Respond to system outages and collaborate across teams to maintain high uptime for critical training runs and inference services.
• Optimize cluster utilization and implement autoscaling strategies for dynamic workload demands.

Qualifications

• Strong expertise in Kubernetes administration, including custom resource definitions, operators, and cluster management.
• Hands-on experience with Slurm workload management, including job scheduling, resource allocation, and cluster optimization.
• Experience deploying and managing distributed training systems at scale.
• Deep understanding of container orchestration and distributed systems architecture.
• High-level familiarity with LLM architecture and training processes, including Multi-Head Attention, Multi-Query, Grouped-Query, and distributed training strategies.
• Experience managing GPU clusters and optimizing compute resource utilization.

Required Skills

• Expert-level Kubernetes administration and YAML configuration management.
• Proficiency with Slurm job scheduling, resource management, and cluster configuration.
• Python and C++ programming focused on systems and infrastructure automation.
• Hands-on experience with ML frameworks such as PyTorch in distributed training contexts.
• Strong understanding of networking, storage, and compute resource management for ML workloads.
• Experience developing APIs and managing distributed systems for batch and real-time workloads.
• Solid debugging and monitoring skills, with expertise in observability tools for containerized environments.

Preferred Skills

• Experience with Kubernetes operators and custom controllers for ML workloads.
• Advanced Slurm administration, including multi-cluster federation and advanced scheduling policies.
• Familiarity with GPU cluster management and CUDA optimization.
• Experience with other ML frameworks such as TensorFlow or distributed training libraries.
• Background in HPC environments, parallel computing, and high-performance networking.
• Knowledge of infrastructure as code, including Terraform and Ansible, and GitOps practices.
• Experience with container registries, image optimization, and multi-stage builds for ML workloads.

Required Experience

• Demonstrated experience managing large-scale Kubernetes deployments in production environments.
• Proven track record with Slurm cluster administration and HPC workload management.
• Previous experience in SRE, DevOps, or Platform Engineering focused on ML infrastructure.
• Experience supporting long-running training jobs and high-availability inference services.
• Ideally, 3–5 years of relevant experience in ML systems deployment, with a focus on cluster orchestration and resource management.

Benefits

Full-time U.S. employees receive benefits including equity, health, dental, vision, retirement, fitness, commuter and dependent care accounts, and more. International employees receive benefits tailored to their region of residence. USD salary ranges apply only to U.S.-based positions.