AI Engineer - Data and Evals

Details

Glean is building a Work AI platform that combines enterprise search, an AI Assistant, and scalable AI agents on a secure, open platform. The Data and Evals team owns data and evaluation science for Glean's AI products, including evalsets, agentic LLM judges, benchmarks, evaluation infrastructure, and enterprise simulation environments.

The role focuses on turning production usage and research findings into evaluations that guide improvements across agent harnesses, tool use, skill iteration, model routing, and post-training of task-specific models. You will partner with R&D, product, and go-to-market teams to turn evaluation findings into better products and customer outcomes.

Responsibilities

• Build evalsets and benchmarks from expert-curated examples, production data, synthetic data, and past failures, covering edge cases and key user slices.
• Build and scale evaluation platforms and agent observability, including experiment pipelines, trace capture and replay, and dashboards with reproducible and affordable runs.
• Build agentic LLM verifiers for correctness, groundedness, citation quality, and task success, calibrating them against human labels using clear rubrics.
• Run evaluations at every stage, from CI and pre-release suites to sampled production traffic, and report results with confidence intervals.
• Build enterprise simulation environments with simulated users and realistic company data to test multi-step agent tasks.
• Diagnose failures by slice and trace, and turn evaluation results into concrete fixes while weighing quality against latency and cost.
• Bring new research into production and collaborate with R&D, product, and go-to-market teams.

Requirements

• 5+ years of software engineering experience.
• Strong Go and/or Python skills and backend fundamentals, including APIs, data modeling, testing, and distributed pipelines.
• Experience building datasets and benchmarks for LLM use cases, RAG, or agents, including evalset curation and synthetic data generation.
• Familiarity with evaluation methodologies and tradeoffs, including deterministic metrics, LLM-as-a-judge, pointwise and pairwise grading, human labeling, and judge calibration.
• Experience building evaluation or experiment infrastructure that runs reliably at scale, including CI/CD integrations and observability tooling such as OpenTelemetry.
• Strong analytical skills, including the ability to investigate traces and failure cases, write SQL, and use data and metrics to explain quality changes.
• TypeScript is a plus.
• Experience with simulation environments and/or model post-training techniques is a nice-to-have.

Location

• Hybrid role requiring four days per week in either the Mountain View or San Francisco office.

Compensation and Benefits

• Standard base salary range: $200,000–$280,000 annually.
• Certain roles may be eligible for variable compensation, equity, and benefits.
• Medical, vision, and dental coverage.
• Generous time-off policy and 401(k) contribution opportunity.
• Home office improvement stipend.
• Annual education and wellness stipends.
• Regular company events and healthy lunches daily.

Additional Information

Glean is committed to building and sustaining a diverse, inclusive workplace. The interview process includes a brief AI-focused exercise or discussion covering how candidates think about, design, and use AI. Prior Glean experience is not required.