Senior Software Engineer, Inference

Details

Anthropic’s Inference team builds and maintains the critical systems that serve Claude to millions of users worldwide. The team operates large-scale, compute-agnostic inference deployments and owns the stack from intelligent request routing to fleet-wide orchestration across diverse AI accelerators.

The role focuses on maximizing compute efficiency while enabling high-performance inference infrastructure for next-generation model research. The work involves complex distributed-systems challenges across multiple accelerator families, AI hardware platforms, and cloud environments.

Responsibilities

• Design intelligent routing algorithms to optimize request distribution across thousands of accelerators.
• Autoscale compute fleets to match supply and demand across production, research, and experimental workloads.
• Build production-grade deployment pipelines for releasing new models to millions of users.
• Integrate new AI accelerator platforms to maintain hardware-agnostic inference capabilities.
• Contribute to inference features such as structured sampling and prompt caching.
• Support inference for new model architectures.
• Analyze observability data and tune performance using real-world production workloads.
• Manage multi-region deployments and geographic routing for global customers.

Requirements

• Significant software engineering experience, particularly with distributed systems.
• A bachelor’s degree or equivalent combination of education, training, and experience.
• Education or professional experience in a field relevant to the role.
• Results-oriented approach with flexibility, initiative, and a bias toward impact.
• Interest in machine learning systems and infrastructure.
• Ability to thrive in an environment where technical excellence supports business results and research breakthroughs.
• Strong communication skills and concern for the societal impacts of AI.

Strong candidates may also have experience with:

• High-performance, large-scale distributed systems.
• Implementing and deploying machine learning systems at scale.
• Load balancing, request routing, or traffic management systems.
• LLM inference optimization, batching, and caching strategies.
• Kubernetes and cloud infrastructure, including AWS and GCP.
• Python or Rust.

Benefits

• Competitive compensation and benefits.
• Optional equity donation matching.
• Generous vacation and parental leave.
• Flexible working hours.
• Collaborative office space.

Work Policy

Staff are currently expected to work from one of Anthropic’s offices at least 25% of the time, although some roles may require more office time. Applications are reviewed on a rolling basis with no stated deadline.