Tech Lead Manager, AI Evaluations

Details

Glean is building a Work AI platform that combines enterprise search, an AI Assistant, and scalable AI agents on a secure, open platform. The platform includes over 100 enterprise SaaS connectors, flexible LLM choice, robust APIs, and enterprise and personal knowledge graphs.

About the Role

The bottleneck in building reliable and useful AI systems is knowing what good looks like, applying production experience to build rigorous evaluations, and using those evaluations to guide continuous product improvement. As Tech Lead Manager of AI Evaluations, you will lead a team of applied AI scientists and engineers and build the evaluation strategy and platform that helps Glean ship reliable AI experiences.

You will translate emerging AI/ML research into reliable evaluation systems, build scalable and reusable evaluation infrastructure and trusted quality signals, and work across R&D, product, and marketing teams to turn evaluation results into better products and customer outcomes.

Responsibilities

• Own the AI evaluation engineering roadmap and technical direction, from defining success criteria and building the evaluation platform to integrating results into release and product decisions.
• Build scalable, reusable evaluation infrastructure for datasets, experiments, graders, trace capture and replay, result analysis, and dashboards.
• Make evaluations reproducible, versioned, cost-aware, and easy for product and engineering teams to use.
• Create and maintain representative evaluation sets using expert-curated examples, governed production signals, synthetic cases, and historical failures.
• Cover common workflows, edge cases, long-tail requests, adversarial inputs, and important user and domain segments while protecting held-out acceptance sets from tuning leakage.
• Define balanced quality measurements for AI products, including correctness, groundedness, retrieval and context relevance, completeness, instruction following, citation quality, user preference, and task success.
• Evaluate agent tool selection and arguments, multi-step trajectories, policy compliance, recovery, and final task state—not only the final response.
• Develop a fit-for-purpose combination of deterministic checks, model-based graders including LLM-based autograders, and human evaluation.
• Establish rubrics, validate and calibrate automated judges against human judgments, and measure evaluator reliability and limitations.
• Integrate evaluation into development and operations through CI regression checks, pre-release suites, model and prompt comparisons, and sampled production evaluation.
• Build feedback loops from user feedback and production traces into new test cases, using monitoring, canaries, and experiments to detect regressions and drift.
• Connect evaluation outcomes to latency, reliability, and cost to help teams diagnose failures, prioritize improvements, and make evidence-based decisions about models, routing, prompts, and system changes.
• Build and lead a high-performing team of applied AI scientists and engineers, coach team members, and partner with product, infrastructure, security, and application teams.

Requirements

• 10+ years of experience in software engineering, ML infrastructure, data systems, or a related technical field, including experience building and operating production systems.
• 5+ years of engineering or applied AI team management experience, with a record of technical leadership, developing engineers and scientists, and delivering cross-functional projects.
• Hands-on experience designing, building, or operating evaluation systems for LLM or generative AI products, including model-based evaluators or autograders.
• Practical understanding of assessing model, retrieval/RAG, assistant, or agent behavior.
• Familiarity with curated and production-derived datasets, benchmark creation, deterministic metrics, rubric-based and pairwise judgments, human review, judge validation and calibration, human-model alignment, and statistical analysis.
• Understanding of agent evaluation, including tool calling, multi-turn behavior, trajectory quality, task completion, and error recovery. Experience with safety testing or red teaming is a plus.
• Strong coding ability in Python or another production language, with solid engineering fundamentals in testing, APIs, data modeling, and reliable distributed systems.
• Experience building scalable data and developer platforms, experiment pipelines, or CI/CD integrations.
• Familiarity with cloud infrastructure, Kubernetes, data stores, and observability is beneficial.
• Strong judgment regarding measurement quality, dataset representativeness, judge calibration, reproducibility, uncertainty, and the risk of optimizing to a narrow metric.
• Knowledge of public benchmarks across domains and capabilities is a plus.
• Product-minded and collaborative, with the ability to translate ambiguous quality goals into clear measures, communicate findings to technical and non-technical audiences, and influence roadmaps across teams.
• MS or PhD in Computer Science, AI, or a related field.

Location and Work Policy

This is a hybrid role requiring four days per week in Glean's Mountain View office.

Compensation and Benefits

The standard base salary range is $250,000–$350,000 annually. Compensation depends on factors including location, level, job-related knowledge, skills, and experience. Certain roles may be eligible for variable compensation, equity, and benefits.

Benefits include medical, vision, and dental coverage, generous time off, a 401(k) plan, a home office improvement stipend, annual education and wellness stipends, regular company events, and healthy lunches daily.

Glean is committed to building and sustaining a diverse, inclusive workplace.