Research Engineer, Model Performance & Quality

Details

Anthropic’s mission is to create reliable, interpretable, and steerable AI systems. The Model Performance team is responsible for systematically understanding and monitoring model quality in real-time. This role blends research and engineering: you will train production models, develop robust monitoring systems, and create novel evaluation methodologies.

Representative projects

• Build comprehensive training observability systems: design and implement monitoring infrastructure to track how model behaviors evolve throughout training.
• Develop next-generation evaluation frameworks: move beyond traditional benchmarks to create evaluations that capture real-world utility.
• Create automated quality assessment pipelines: build custom classifiers to continuously monitor RL transcripts for complex issues.
• Bridge research and production: partner with research teams to translate cutting-edge evaluation techniques into production-ready systems and work with engineering teams to ensure monitoring infrastructure scales with complex training workflows.

Responsibilities

• Train and evaluate production-scale models and contribute to the full model training pipeline.
• Design and implement monitoring and observability systems for training and production.
• Develop novel evaluation methodologies and automated quality-assessment pipelines.
• Debug complex, distributed training systems and reason about failure modes.
• Collaborate across research and engineering teams to deploy research ideas into production.

Requirements

• Proficiency in Python (note: all interviews for this role are conducted in Python).
• Experience building production ML systems.
• Experience with training, evaluating, or monitoring large language models.
• Comfort debugging complex, distributed systems and interpreting training metrics and model behavior.
• Ability to balance research exploration with engineering rigor and collaborate across diverse teams.
• Education: at least a Bachelor’s degree in a related field or equivalent experience.

Strong (but not required) qualifications

• Experience with reinforcement learning and language model training pipelines.
• Experience designing and implementing evaluation frameworks or benchmarks.
• Background in production monitoring, observability, and incident response.
• Experience with statistical analysis and experimental design.
• Knowledge of AI safety and alignment research.

Logistics and compensation

• Annual base salary: $315,000 - $340,000 USD.
• Location-based hybrid policy: staff are expected to be in one of Anthropic’s offices at least 25% of the time (hybrid schedule).
• Visa sponsorship: Anthropic does sponsor visas where feasible and retains an immigration lawyer to assist.

Benefits & culture

• Competitive compensation and benefits, optional equity donation matching, generous vacation and parental leave, flexible working hours, and collaborative office spaces.
• Emphasis on high-impact, large-scale AI research and frequent research discussions.

Notes

• Interviews conducted in Python.
• Anthropic encourages candidates to apply even if not all qualifications are met and aims to support diverse perspectives in AI safety and research.