Technical Program Manager, Inference

Details

Anthropic’s Inference organization serves Claude to millions of users and enterprise customers across its products and major cloud platforms. The organization builds and operates inference systems spanning accelerator kernels, an in-house inference engine, distributed routing, autoscaling, and capacity management across GPUs, TPUs, and Trainium.

The Technical Program Manager will drive programs across scaling and infrastructure, engineering velocity, the inference engine, and performance. This role owns outcomes end to end across Research, Safeguards, Security, Compute, Infrastructure, API, and external cloud and silicon partners. It involves building lightweight program mechanisms for ambiguous, fast-moving work and using Claude to explore AI-assisted program management and fleet operations.

Responsibilities

• Own programs that scale Claude’s serving fleet across accelerator and cloud platforms from early direction through delivery.
• Drive new capacity online across compute, infrastructure, security, networking, and inference teams.
• Build visibility into supply, demand, and utilization with capacity and finance partners, and drive trade-offs among utilization, cost, latency, and reliability.
• Lead reliability programs and large migrations without disrupting production traffic or research, ensuring incident follow-ups are incorporated into team roadmaps.
• Improve inference engineering productivity by defining relevant metrics, identifying bottlenecks, and improving how software is built, tested, and released across platforms.
• Coordinate roadmaps and priorities across production, research, and safeguards requirements.
• Track performance and efficiency work from opportunity through measured production impact, with correctness treated as a release criterion.
• Own inference readiness for model launches alongside launch TPMs.

Requirements

• 7+ years of experience as a Technical Program Manager or in an equivalent engineering or SRE lead role, owning large, multi-team infrastructure programs end to end in production distributed systems, ML infrastructure, or cloud platforms.
• Technical fluency in large-scale serving infrastructure, including request routing and load balancing, autoscaling, capacity planning, cluster orchestration, deployment and rollout, and observability.
• Working understanding of LLM inference hardware concepts such as prefill and decode, batching, caching, compute, memory bandwidth, and interconnect; alternatively, a demonstrated ability to quickly develop depth in unfamiliar technical domains.
• Experience building program mechanisms from scratch for ambiguous, fast-moving efforts and retiring processes that no longer provide value.
• Ability to drive outcomes across organizations without direct management authority and competing priorities.
• Strong written communication skills, including plans, reviews, and executive updates.
• High ownership and willingness to perform hands-on program work, including writing scripts, running incident bridges, and managing capacity or quotas.
• Commitment to Anthropic’s mission and to serving frontier AI safely and reliably.

Preferred Qualifications

• Direct experience with LLM or ML inference serving at scale, including model servers, batching and scheduling, KV caching, quantization, speculative decoding, or multi-model routing.
• Experience with accelerator infrastructure such as GPUs, TPUs, or Trainium, including cluster bring-up, hardware qualification, or running workloads across multiple accelerator platforms.
• Experience managing capacity across multiple clouds or regions, including demand forecasting, headroom policy, cost modeling, or compute unit economics.
• Experience with developer productivity, release engineering, or CI/CD at scale, especially for hardware-dependent workloads.
• Experience running performance or efficiency programs, with knowledge of rooflines, kernels, ML compilers such as XLA, Triton, or Neuron, numerics, or collective communication.
• Familiarity with Kubernetes-based orchestration and job scheduling.
• Experience working with research teams and translating their needs into infrastructure requirements.
• Practical experience using AI tools, including building automations or agents.

Education and Experience

• Bachelor’s degree or an equivalent combination of education, training, and experience.
• Required field of study: a field relevant to the role as demonstrated through coursework, training, or professional experience.

Benefits

Anthropic offers competitive compensation and benefits, optional equity donation matching, generous vacation and parental leave, flexible working hours, and office space for collaboration. Anthropic sponsors visas and will make every reasonable effort to support visa applications with the assistance of an immigration lawyer.