Tech Stack
Tag name is followed by "@" symbol and proficiency level value.
About proficiency levels:
- 1-2 — basic awareness. Minimal hands-on experience, and a rudimentary understanding of the technology's purpose;
- 3-6 — daily use. Comfortable and regular usage, capable of handling common tasks and challenges related to the technology;
- 7-9 — you are an expert, you can teach others, you know all the pitfalls and tricks;
- 10 — exceptional knowledge, comprehensive understanding, and adeptness in all aspects of the technology, including advanced problem-solving. Think twice before claiming or demanding such level.
A/B Testing @ 4
AI @ 7
API @ 4
Communication @ 6
Customer Support @ 4
Experimentation @ 4
LLM @ 4
Machine Learning
Python @ 4
SQL @ 7
Statistics @ 7
- 1-2 — basic awareness. Minimal hands-on experience, and a rudimentary understanding of the technology's purpose;
- 3-6 — daily use. Comfortable and regular usage, capable of handling common tasks and challenges related to the technology;
- 7-9 — you are an expert, you can teach others, you know all the pitfalls and tricks;
- 10 — exceptional knowledge, comprehensive understanding, and adeptness in all aspects of the technology, including advanced problem-solving. Think twice before claiming or demanding such level.
Details
Airbnb's AI & ML Enablement team in Supply Tech helps business, field, and product teams use AI to automate high-volume processes, launch trustworthy AI features, build labeled datasets, and maintain workflows as models and policies change.
Responsibilities
- Drive AI workflows and features from design to launch, defining quality standards and measurement plans with engineering, product, and business partners.
- Prioritize projects by business impact and make trade-offs with managers and business and product leads.
- Analyze existing processes and determine how AI can automate them while identifying cases requiring human judgment.
- Design and build AI workflows, including prompts, pipelines, tooling, and quality and drift monitoring.
- Build evaluation frameworks, test sets, accuracy and agreement metrics, and calibrated LLM judges.
- Collaborate with engineers on architecture and tooling decisions and evaluate estimates and scope.
- Translate unclear guidelines into consistent reviewer criteria in collaboration with Product, Legal, and Policy.
- Create labeling guidelines and produce human-verified labels with subject matter experts.
- Run labeling programs at scale with vendor teams and assess quality using agreement metrics and spot checks.
- Define launch criteria for AI features and provide evidence and go/no-go recommendations.
- Build reusable evaluation and workflow components, document learnings, and help other teams reach production faster.
- Keep methods current as models, tools, and policies evolve.
Requirements
- 8+ years of experience in product or program management, analytics, consulting, or applied AI.
- Proven experience taking technical programs from concept to production and owning outcomes such as backlog, cycle time, or launch speed.
- Strong SQL skills and knowledge of quality statistics, including accuracy, precision, recall, sampling, and reviewer agreement.
- Experience shipping an AI workflow, agent, or automation that others rely on, with measurable before-and-after results.
- Technical fluency sufficient to evaluate engineering estimates and understand prompts, pipelines, and evaluations.
- Ability to make messy processes measurable and assess success through business outcomes.
- Strong judgment in evaluating qualitative feedback, reviewer comments, and disagreements.
- Clear communication skills and the ability to work with sales, operations, product managers, and engineers.
- Ability to independently investigate problems and build initial solutions.
Nice to Have
- Hands-on Python and API experience, including scripting data pulls, calling model APIs, or building small automations.
- Experience with LLM evaluation, test-set creation, LLM judge calibration, or model fine-tuning.
- Experience with experimentation and measurement, including A/B testing, metric design, and dashboards.
- Experience in trust and safety, content moderation, customer support automation, annotation operations, or other human-in-the-loop work at scale.
- Background in a two-sided marketplace, travel, or another operations-heavy business.
Location
This position is US remote eligible. Occasional work at an Airbnb office or attendance at offsites may be required as agreed with the manager. Candidates must live in a state where Airbnb, Inc. has a registered entity.
Compensation
The base pay range is $157,000–$171,000 USD. The role may also be eligible for bonus, equity, benefits, and Employee Travel Credits.
Airbnb is committed to inclusion and belonging, equal employment opportunity, and providing reasonable accommodations throughout the recruitment process.
More jobs at Airbnb
Senior Android Engineer, App Experience & Frameworks
Airbnb · United States
USD 196,000-227,000 per year
Software Engineering Intern (BS/MS)
Airbnb · San Francisco, United States
USD 94,800-97,200 per year
Senior Data Engineer, Enterprise Core Datasets, Tech Foundations
Airbnb · United States
USD 196,000-230,000 per year
Engineering Manager, External Search & AI Optimization
Airbnb · United States
USD 212,000-265,000 per year
Software Engineer, DevAI
Airbnb · United States
USD 162,000-190,000 per year
Similar jobs
Engineering Manager, Machine Learning
Airbnb · United States, San Francisco, United States
USD 220,000-265,000 per year
Senior Product & PLG Analyst (Full-Stack Data Scientist)
Nebius · New York City, United States
USD 179,000-224,000 per year
Tech Lead Manager, Growth Data Science
Anthropic · San Francisco, United States, New York City, United States, Seattle, United States
USD 460,000 per year
Lead Data Scientist, Platform Product
Anthropic · San Francisco, United States, New York City, United States, Seattle, United States
USD 285,000-380,000 per year
Senior/Staff Applied Scientist
Glean · Mountain View, United States, San Francisco, United States
USD 180,000-330,000 per year
AI Engineer, GTM Claudification
Anthropic · San Francisco, United States, Seattle, United States
USD 320,000-405,000 per year
Machine Learning Data Scientist - Consumer Forecasting
OpenAI · San Francisco, United States
USD 347,000-445,000 per year
Senior Data Scientist, Ads
Reddit · United States
USD 190,800-267,100 per year