Staff Research Engineer, Post-Training & Evaluation Science

Details

Role Overview

Reddit’s AI Engineering team builds Reddit-native foundational Large Language Models (LLMs). This Staff Research Engineer role owns the science of Reddit’s model development "feedback loop," including how models are evaluated for safety, intelligence, and alignment with Reddit’s culture and language.

Responsibilities

• Define the "Reddit Benchmark" evaluation standard: Own the methodology for rigorously measuring model quality across Safety, Reasoning, representation/retrieval, and Reddit-specific knowledge. Decide what "Reddit-native" means in measurable terms and set the bar for training.
• Own evaluation reliability and statistical rigor: Establish the science behind trustworthy evals, including judge/sample variance, multi-sample scoring, inter-rater/inter-sample agreement, sampling and temperature effects, and calibration of automated judges. Ensure benchmark deltas reflect real changes (not noise). Drive eval practice as a release gate using offline evals against frozen datasets and pre-merge in CI/CD so regressions are caught before endpoints ship.
• Design model-as-a-judge methodology: Own judge selection, prompt design, calibration, and reliability for automated evaluation using frontier external models to enable rapid, trustworthy iteration.
• Set post-training recipes and strategy: Design SFT recipes (data mixtures, curriculum, ablation strategy) that convert base models into helpful, well-aligned endpoints, partnering with engineering to scale.
• Evaluate base and CPT checkpoints: Design checkpoint-selection methodology across CPT experiments and LR studies to select the right base before committing post-training compute.
• Drive synthetic data generation strategy: Define and curate high-quality instruction and evaluation sets to improve generalization where human data is scarce.
• Partner with Safety Engineering: Translate safety policy into classification metrics, probe sets, and CI/CD unit tests, including precision/recall at thresholds, label-noise handling, and false-positive taxonomy for abuse detection (HHV).
• Diagnose post-training instability: Investigate loss curves and eval logs to identify alignment tax and capability degradation, and recommend fixes.
• Lead research direction: Set technical direction for evaluation and post-training across the team, mentor engineers and scientists, and represent the work internally (and externally where appropriate).

Requirements

• 6+ years of professional ML experience (or PhD + 4+) with direct focus on LLM post-training and evaluation.
• PhD or MS in CS, ML, NLP, IR, or a related quantitative field (or equivalent industry research experience).
• Deep expertise in evaluation reliability, including judge/sample variance, multi-sample scoring, calibration, statistical significance, and failure modes of automated evaluation.
• Strong experience building custom, domain-specific evaluation harnesses (e.g., lm-eval-harness, Inspect AI, LightEval). Know strengths and limits of benchmarks like MMLU and GSM8K, and treat eval sets as versioned, frozen, regression-tracked code.
• Experience evaluating both generation and representation/classification: model-as-a-judge for generative quality and precision/recall, PR-AUC, retrieval/MTEB-style metrics, gold-label denoising, and label-noise handling.
• Deep understanding of Continuous Pre-training (CPT) and Instruction Tuning (SFT), and how data quality shapes model behavior.
• Fluency in Python; strong data-pipeline and eval-harness engineering using Hugging Face Transformers, vLLM, lm-eval-harness. Working knowledge of PyTorch and distributed training (FSDP2, DeepSpeed ZeRO-3) sufficient to direct and debug post-training runs.

Benefits

• Comprehensive Healthcare Benefits and Income Replacement Programs
• 401k with Employer Match
• Global Benefit programs (workspace to professional development to caregiving support)
• Family Planning Support
• Gender-Affirming Care
• Mental Health & Coaching Benefits
• Flexible Vacation & Paid Volunteer Time Off
• Generous Paid Parental Leave