Research Engineer, Model Performance & Quality

Details

Anthropic’s mission is to create reliable, interpretable, and steerable AI systems beneficial for users and society. The team consists of researchers, engineers, policy experts, and business leaders working together to build beneficial AI systems.

Responsibilities

• Design and implement comprehensive training observability systems to monitor model behavior evolution during training.
• Develop next-generation evaluation frameworks that capture real-world utility beyond traditional benchmarks.
• Create automated quality assessment pipelines, including custom classifiers to monitor RL transcripts for complex issues.
• Collaborate with research teams to translate cutting-edge evaluation techniques into production-ready systems and ensure scalable monitoring infrastructure.

Requirements

• Proficient in Python with experience building production ML systems.
• Experience with training, evaluating, or monitoring large language models.
• Strong curiosity about debugging complex, distributed systems and analyzing failure modes.
• Collaborative problem-solver who works across diverse teams covering all stages of the model training pipeline.
• Ability to balance research exploration with engineering rigor.
• Strong analytical skills for interpreting training metrics and model behavior.
• Desire to impact the quality and safety of deployed AI systems.

Strong candidates may have:

• Experience with reinforcement learning and language model training pipelines.
• Experience designing and implementing evaluation frameworks or benchmarks.
• Background in production monitoring, observability, and incident response.
• Experience with statistical analysis and experimental design.
• Knowledge of AI safety and alignment research.

Strong candidates need not have:

• Formal certifications or academic credentials.
• Academic research experience or publication history.
• Prior experience specifically in AI safety or evaluation.

This role offers a unique opportunity to shape model quality assessment approaches while working on critical systems for advancing AI capabilities.

Logistics

• Education: Bachelor's degree in a related field or equivalent experience required.
• Hybrid policy: Staff expected in office at least 25% of the time, some roles may require more.
• Visa sponsorship available with efforts to assist candidates.

Benefits

Competitive compensation and benefits, optional equity donation matching, generous vacation and parental leave, flexible working hours, and modern office spaces.

About Anthropic

Anthropic believes in large-scale collaborative AI research focusing on impact and trustworthy AI, emphasizing communication and empirical science approaches similar to physics and biology.