Tech Stack
Tag name is followed by "@" symbol and proficiency level value.
About proficiency levels:
- 1-2 — basic awareness. Minimal hands-on experience, and a rudimentary understanding of the technology's purpose;
- 3-6 — daily use. Comfortable and regular usage, capable of handling common tasks and challenges related to the technology;
- 7-9 — you are an expert, you can teach others, you know all the pitfalls and tricks;
- 10 — exceptional knowledge, comprehensive understanding, and adeptness in all aspects of the technology, including advanced problem-solving. Think twice before claiming or demanding such level.
AI
Ansible @ 6
ArgoCD @ 6
Communication @ 6
Compliance @ 4
Debugging @ 3
GPU @ 3
Go @ 6
IaC @ 6
Kubernetes @ 7
Observability
Planning @ 3
SRE
Security @ 4
Terraform @ 6
- 1-2 — basic awareness. Minimal hands-on experience, and a rudimentary understanding of the technology's purpose;
- 3-6 — daily use. Comfortable and regular usage, capable of handling common tasks and challenges related to the technology;
- 7-9 — you are an expert, you can teach others, you know all the pitfalls and tricks;
- 10 — exceptional knowledge, comprehensive understanding, and adeptness in all aspects of the technology, including advanced problem-solving. Think twice before claiming or demanding such level.
Details
About the Role
We are seeking a highly skilled Senior Infrastructure Engineer to join our US Government Team, focused on designing, building, and operating secure, scalable infrastructure for critical government projects. In this role, you will develop and manage training and inference clusters, as well as highly reliable applications, across bare metal, classified cloud, and hybrid cloud architectures. You will leverage your expertise in Kubernetes and GPU hardware to deliver robust, secure systems that support large-scale AI workloads while meeting stringent federal compliance requirements. This role demands a passion for automation, observability, and ensuring system integrity in a fast-paced, high-security environment.
Responsibilities
- Develop and optimize software to provision and manage SpaceXAI's infrastructure across on-premise, virtual machine, and classified cloud environments, enabling efficient scaling for US government initiatives.
- Enhance the reliability, performance, and cost-effectiveness of infrastructure to support large-scale AI and application workloads in secure, classified settings.
- Collaborate with SpaceXAI engineers to understand workload requirements and design tailored solutions that meet government-specific needs and compliance standards.
- Implement robust observability, monitoring, and security practices to ensure the integrity, availability, and confidentiality of critical systems, adhering to federal protocols.
- Manage storage infrastructure using Infrastructure-as-Code (IaC) tools such as Pulumi, Terraform, or Ansible, with a focus on secure data handling.
- Drive system reliability through incident management, postmortems, and the definition of clear SLAs and SLOs, while maintaining security and compliance.
- This is an in-person role based in Palo Alto, CA or Washington, DC, with up to 50% travel required.
Requirements
- Active Top Secret (TS) security clearance.
- 5+ years of experience as an Infrastructure Engineer, Site Reliability Engineer, or similar role, with a focus on building and maintaining reliable, scalable systems, preferably in secure or government environments.
- Proficiency in managing storage infrastructure with IaC tools such as Pulumi, Terraform, or Ansible.
- Deep understanding of the Kubernetes stack, including CNI, CRI, CSI, and related components.
- Demonstrated ability to improve system reliability through incident management, postmortems, and defining SLAs/SLOs.
- Excellent communication and documentation skills, with the ability to handle sensitive information concisely and accurately.
Preferred Skills and Experience
- Deep familiarity with installing and using GPU hardware, including setting up drivers, debugging issues, and ensuring reliability.
- Experience with high-traffic web or mobile application workloads, including optimizing Kubernetes for large-scale deployments in classified or federal settings.
- Familiarity with chaos engineering, capacity planning, or similar practices for ensuring system resilience in government projects.
- Proficiency with tools such as Kyverno, ArgoCD, or Go programming for infrastructure automation.
- Strong sense of ownership, curiosity, and enthusiasm for tackling complex technical challenges in secure environments.
- Passion for problem-solving and a proactive drive to deliver impactful results while adhering to security protocols.
- Certifications in security-related fields (e.g., CISSP) or experience in secure federal environments.
Compensation and Benefits
- $180,000 - $440,000 USD base salary.
- Equity.
- Comprehensive medical, vision, and dental coverage.
- Access to a 401(k) retirement plan.
- Short & long-term disability insurance.
- Life insurance.
- Various other discounts and perks.
More jobs at SpaceXAI
ML Infrastructure Engineer
SpaceXAI · Palo Alto, United States
USD 180,000-440,000 per year
Manager, Human Data Operations
SpaceXAI · Dubai, United Arab Emirates, Indonesia, Ireland, India, United States, South Korea, United Kingdom, Japan, Philippines, Singapore, Singapore
USD 108,000-162,000 per year
Software Engineer - Platform Infrastructure (Rust, C++)
SpaceXAI · Palo Alto, United States
USD 180,000-440,000 per year
AI Tutor - Sinhala
SpaceXAI · World
USD 35-45 per hour
Software Engineer - Network Software and Services
SpaceXAI · Dublin, Ireland
EUR 80,000-150,000 per year
Similar jobs
Senior Storage Production Engineer - DGX Cloud
Nvidia · Santa Clara, United States
USD 176,000-333,500 per year
Senior Storage Production Engineer - DGX Cloud
Nvidia · Santa Clara, United States
USD 176,000-333,500 per year
Forward Deployed Engineer - Physical AI Cloud Platform
Nebius · United States
USD 179,500-224,300 per year
Senior Systems Engineer, Storage - DGX Cloud
Nvidia · United States
USD 208,000-414,000 per year
Senior Software Engineer, DGX Cloud Production Engineering
Nvidia · Santa Clara, United States
USD 184,000-356,500 per year
Senior Software Engineer - Cloud Infrastructure
ClickHouse · United States
USD 141,000-208,000 per year
Member of Technical Staff (Software Engineer, Cloud Infrastructure)
Perplexity AI · Palo Alto, United States, San Francisco, United States, New York City, United States
USD 220,000-405,000 per year
Member of Technical Staff (AI Infrastructure Engineer)
Perplexity AI · San Francisco, United States, Palo Alto, United States
USD 220,000-405,000 per year