Software Engineer, Inference

Details

Anthropic’s Inference team builds and maintains the systems that serve large language models (LLMs) to diverse consumers. The team focuses on scaling inference systems, reliability, compute efficiency, and developing new inference capabilities. Work crosses the full inference stack — from request routing to prompt caching — and involves close collaboration with researchers and engineers to deliver robust production services.

Responsibilities

• Design, implement, and maintain high-performance, large-scale distributed systems for inference.
• Optimize inference request routing to maximize compute efficiency and reduce latency.
• Implement autoscaling for compute fleets to match inference demand.
• Develop and deploy inference features (e.g., structured sampling, fine-tuning support).
• Build and maintain batching and caching strategies for LLM optimization.
• Support inference for new model architectures and ensure reliable deployments.
• Analyze observability data and tune performance based on production workloads.
• Collaborate closely with researchers and engineers; participate in pair programming and frequent research discussions.

Requirements

• Significant software engineering experience.
• Bachelor’s degree in a related field or equivalent experience.
• Experience or strong interest in machine learning research and production ML systems.
• Familiarity with high-performance, large-scale distributed systems and scalability engineering.
• Experience with LLM optimization strategies such as batching and prompt caching.
• Experience with Kubernetes and deploying services at scale.
• Proficiency in Python.
• Strong communication and collaboration skills; comfortable with pair programming.

Representative Projects (examples)

• Optimizing inference request routing to maximize compute efficiency.
• Building autoscaling logic for compute fleets.
• Contributing to inference features (structured sampling, fine-tuning).
• Ensuring smooth deployment of inference services and supporting new model architectures.
• Using observability data to tune production performance.

Logistics & Other Information

• Location: Dublin, Ireland (location-based hybrid policy: staff expected to be in an office at least ~25% of the time; some roles may require more office time).
• Education: at least a Bachelor’s degree in a related field or equivalent experience.
• Visa sponsorship: Anthropic does sponsor visas, though not every role/candidate may be eligible; they retain immigration counsel and will make reasonable efforts where they can.
• Applications reviewed on a rolling basis (no deadline).

Benefits

• Competitive compensation and benefits.
• Optional equity donation matching.
• Generous vacation and parental leave.
• Flexible working hours and collaborative office space.
• Guidance on candidate AI usage during the application process.