Senior Software Engineer, AI Evals

Details

Sentry helps developers fix errors and performance issues before users notice. Trusted by more than 200,000 organizations, Sentry is building its AI-native future.

As a Senior Software Engineer on Sentry's AI/ML team, you will build evaluation infrastructure that measures the accuracy, reliability, and real-world performance of AI systems. You will design datasets, benchmarks, and test harnesses that turn ambiguous AI behavior into measurable signals, helping the team ship AI with confidence.

Responsibilities

• Design and build robust evaluation frameworks to measure accuracy, reliability, regressions, and edge cases in AI systems.
• Create and curate high-quality datasets, golden test cases, and benchmarks grounded in real production data.
• Build automated test harnesses and metrics pipelines to continuously evaluate models, prompts, and agentic workflows.
• Partner with applied AI engineers and product leaders to define what “good” looks like and translate it into measurable criteria.
• Own the evaluation lifecycle for major AI initiatives, from early experimentation through production monitoring.
• Help influence model design through improved evaluation and measurement.

Requirements

• At least 5 years of professional experience and a bachelor's degree in computer science, machine learning, or a related field.
• Experience building testing, evaluation, or data infrastructure for complex systems; AI/ML experience is strongly preferred.
• Ability to write production-quality code in Python and TypeScript.
• Experience working with structured and unstructured datasets, labeling workflows, or data quality pipelines.
• Familiarity with modern ML systems and evaluation techniques, including offline metrics, online evaluation, and regression testing for models or prompts.
• Experience evaluating LLMs, agentic systems, or AI-assisted developer tools is a bonus.
• Strong interest in correctness, rigor, and measurement in AI systems.
• Ability to turn ambiguous product goals and model behavior into concrete tests and metrics.
• Ability to work effectively in cross-functional environments.

Compensation and Benefits

• Base salary: $240,000–$280,000 USD per year.
• Eligible candidates may participate in employee benefit plans and programs, including incentive compensation, equity grants, paid time off, and group health insurance coverage.

Equal Opportunity

Sentry is committed to providing equal employment opportunities regardless of race, color, ancestry, religion, sex, national origin, sexual orientation, age, citizenship, marital status, disability, gender identity, veteran status, or other legally protected characteristic. The company provides reasonable accommodations to employees and candidates with disabilities and strives to build a diverse, inclusive culture.

Sentry's applicant privacy policy provides additional information about how applicant data is handled.