Backend Software Engineer (Evals) – Support Automation Engineering
Used Tools & Technologies
Not specified
Required Skills & Competences
Tag name is followed by "@" symbol and proficiency level value.
About proficiency levels:
- 1-2 — basic awareness. Minimal hands-on experience, and a rudimentary understanding of the technology's purpose;
- 3-6 — daily use. Comfortable and regular usage, capable of handling common tasks and challenges related to the technology;
- 7-9 — you are an expert, you can teach others, you know all the pitfalls and tricks;
- 10 — exceptional knowledge, comprehensive understanding, and adeptness in all aspects of the technology, including advanced problem-solving. Think twice before claiming or demanding such level.
Python @ 5
Distributed Systems @ 3
Data Science @ 3
FastAPI @ 5
API @ 3
LLM @ 3
- 1-2 — basic awareness. Minimal hands-on experience, and a rudimentary understanding of the technology's purpose;
- 3-6 — daily use. Comfortable and regular usage, capable of handling common tasks and challenges related to the technology;
- 7-9 — you are an expert, you can teach others, you know all the pitfalls and tricks;
- 10 — exceptional knowledge, comprehensive understanding, and adeptness in all aspects of the technology, including advanced problem-solving. Think twice before claiming or demanding such level.
Details
About the Team
The Support Automation team at OpenAI scales the organization by applying cutting-edge AI models to real-world challenges, automating and enhancing work across the organization. From customer operations to engineering, the team develops an ecosystem of automation products that empower colleagues and drive impact. The team leverages OpenAI technology (public and pre-released) to improve OpenAI, blending rapid prototyping with a focus on long-term quality, reliability, and reusable solutions.
About the role
You will work as a Backend Software Engineer with experience in ML/LLM-heavy domains to design and build evals infrastructure that measures the quality of OpenAI's support automation. This is a highly cross-functional role focused on building robust backend systems and services that underpin how knowledge is created, accessed, and applied across OpenAI. You will work closely with Data Science and Research partners to design and build evals at scale and integrate OpenAI models into high-leverage workflows.
Responsibilities
- Design eval pipelines that are reliable, reproducible, and extendable.
- Build infrastructure for continuous eval monitoring frameworks (regression/drift monitoring, building robust golden datasets) and feedback loops to strengthen support automation.
- Design, build, and maintain backend services and APIs to support intelligent automation and knowledge systems.
- Integrate and structure data across internal platforms, transforming it into formats optimized for downstream systems and AI workflows.
- Collaborate closely with data, research, and engineering teams to integrate OpenAI models into workflows.
- Own the full development lifecycle of new backend systems and internal platform capabilities.
- Build with scale and maintainability in mind while iterating rapidly on new ideas.
Requirements
- 4+ years of backend engineering experience at product-driven companies (excluding internships).
- Proficiency in backend technologies; tech stack mentioned includes Python, FastAPI, and Postgres.
- Experience designing and scaling distributed systems, APIs, or data processing pipelines.
- Experience building AI agents or applications, including designing evals and improving performance through prompting or scaffolding.
- Familiarity with evaluation methods for LLMs and patterns such as multi-agent workflows, tool use, or long context.
- Experience creating production evals and/or measuring performance of ML/LLM models at scale.
- Pragmatic mindset: comfortable shipping iteratively while building toward a long-term vision.
Benefits & additional info
- Base salary range listed: $255,000 – $405,000 (total compensation also includes equity, potential performance-related bonus(es), and other benefits).
- Medical, dental, and vision insurance with employer contributions to Health Savings Accounts.
- Pre-tax accounts for Health FSA, Dependent Care FSA, and commuter expenses.
- 401(k) retirement plan with employer match.
- Paid parental leave (up to 24 weeks birth parents, 20 weeks non-birthing), paid medical and caregiver leave (up to 8 weeks).
- Paid time off: flexible PTO for exempt employees and up to 15 days annually for non-exempt employees.
- 13+ paid company holidays and multiple coordinated office closures, plus paid sick or safe time as required by law.
- Mental health and wellness support; employer-paid basic life and disability coverage.
- Annual learning and development stipend; daily meals in offices and meal delivery credits as eligible.
- Relocation support for eligible employees and additional taxable fringe benefits (charitable donation matching, wellness stipends).
Other notes
- OpenAI is an equal opportunity employer. Background checks will be administered in accordance with applicable law. Applicants with disabilities can request reasonable accommodations.