Research Engineer, Data Ingestion

Details

Anthropic’s mission is to create reliable, interpretable, and steerable AI systems. This role sits on the Data Ingestion team and combines hands-on engineering with data research to acquire and improve large-scale pretraining data via a web crawler infrastructure. You will build and scale crawler systems, run experiments to evaluate data quality, and collaborate with Pretraining and Tokens teams to create feedback loops between crawled data and evaluation results.

Responsibilities

• Develop and maintain a large-scale web crawler and related infrastructure
• Design and run experiments to evaluate data quality, extraction methods, and crawling strategies
• Analyze crawled data to identify patterns, gaps, and improvement opportunities
• Build pipelines for data ingestion, analysis, and quality improvement
• Build specialized crawlers for high-value data sources
• Collaborate with Pretraining and Tokens teams to create feedback loops between crawled data and data evaluation results
• Collaborate with team members on improving data acquisition processes
• Participate in code reviews and debugging sessions

Requirements

• Experience with data research, including designing experiments and analyzing results
• Experience working on web crawlers or large-scale data acquisition systems
• Comfortable operating in a hybrid research-engineering role that balances system building with experimentation
• At least a Bachelor's degree in a related field or equivalent experience

Compensation

• Annual salary: $320,000 - $405,000 USD
• Total compensation package may include equity, benefits, and incentive compensation

Logistics & Other Details

• Location: San Francisco, CA
• Location-based hybrid policy: staff expected to be in an office at least 25% of the time
• Visa sponsorship: Anthropic will make reasonable efforts to sponsor visas for hires when possible
• Anthropic encourages applicants from diverse backgrounds and those who may not meet every listed qualification to apply

Nice-to-know / How we work

• The team values large-scale empirical AI research and frequent collaborative research discussions
• Work includes close collaboration across research and engineering teams and emphasis on communication