Senior AI Engineer, Unstructured AI

Details

Join Collibra’s Unstructured AI team to work on context engineering and AI systems that retrieve, structure, and leverage context for accurate results at scale. The role involves owning end-to-end technical delivery, from feature prototypes through stable production deployments across enterprise environments. You will build full-stack systems that ingest, process, and enrich large volumes of unstructured content, including PDFs, contracts, reports, and other documents.

This is a hybrid role based in the New York office, with at least two days per week required in the office.

Responsibilities

• Ship complex systems under ambiguity, defining scope and acceptance criteria when needed.
• Write and review production-grade backend code using Python and FastAPI.
• Build and deploy document-processing systems for large-scale unstructured data environments.
• Integrate data from enterprise sources such as SharePoint, Salesforce, and internal APIs to provide context for AI features.
• Partner with engineering, product, and sales teams from prototype through rollout.
• Occasionally contribute to modern frontend development.
• Own technical delivery for key product areas and develop enterprise-grade AI product features.
• Architect high-performance pipelines and context-engineering capabilities for accurate and reliable results.

Requirements

• Strong proficiency in Python for data processing, API development, and integrations.
• Hands-on experience with LLM-based and AI-driven enrichment models, including classification, entity extraction, deduplication, and PII detection.
• Production experience with Spark or comparable big data frameworks, including tuning and debugging jobs at scale.
• Experience shipping tested and reviewed production services rather than only notebooks.
• Experience independently defining an MVP for a vaguely scoped problem, documenting assumptions, and delivering the solution.
• Solid understanding of data pipelines, microservice architecture, and API design.
• Experience ingesting and processing data from third-party enterprise sources, including SharePoint/OneDrive, Salesforce, and SaaS-based knowledge bases.
• Familiarity with metadata systems, data cataloging, or document AI workflows.
• Knowledge of model evaluation best practices.
• Experience with search relevance.
• Practical experience with agentic engineering, including agent loops, guardrails, structured outputs, context management, testing, review gates, and repository conventions.
• Bachelor’s degree or equivalent related work experience.
• Strong communication and stakeholder-management skills across technical and business teams.
• This position is not eligible for visa sponsorship.

Measures of Success

• Within the first month, develop a deep understanding of the product vision and unstructured data stack and ship an initial set of end-to-end features.
• Within the third month, take full ownership of technical delivery for key product areas and build robust document-processing capabilities.
• Within the sixth month, drive ambitious, enterprise-grade AI product features that solve data-at-scale challenges.

Benefits

The total rewards package includes bonus potential, equity for eligible roles, a Flex Fund monthly stipend, pension/401k plans, health coverage, and time off. Collibra also provides flexible benefits designed to support employees and their families.