Senior AI Engineer

Details

Grafana Labs is hiring a Senior Engineer (AI & Automation) to own the AI agent infrastructure and automation platform that powers Marketing Operations. This is a remote opportunity for candidates located in Canada (residents of Quebec are not eligible). You will design and ship multi-agent architectures, LLM integrations, and backend services that connect AI models to internal and third-party data platforms and deliver production systems used daily by business teams.

Responsibilities

• Own end-to-end development of multi-agent AI systems, from architecture and implementation through testing, deployment, and ongoing operation
• Build modular, composable agentic systems using orchestration frameworks (LangChain, CrewAI, Anthropic MCP, or similar) that operate 24/7 across teams
• Develop reusable agentic skills that agents invoke across interfaces (Slack, dashboards, internal apps, CLIs)
• Implement observability and feedback loops including logging, performance metrics, prompt iteration, model evaluation, and cost management
• Establish governance and compliance standards for AI workflows including access controls, audit trails, PII handling, and human-in-the-loop escalation paths
• Build MCP servers, APIs, CLIs, and microservices connecting AI models to business systems (BigQuery, Slack, CRMs, email, calendars, analytics tools)
• Architect data flows for retrieval-augmented generation (RAG), connecting LLMs to internal knowledge bases, customer data, and real-time business context
• Build serverless or containerized services (GCP Cloud Functions, Cloud Run) that scale with usage and integrate with Grafana's cloud infrastructure
• Partner with RevOps, Demand Generation, Regional Marketing, and SDR teams to scope high-impact automation problems and build measurable solutions
• Design and deploy workflows using orchestration tools (n8n, Workato, or custom platforms) with CI/CD, testing, and production reliability standards
• Produce documentation, playbooks, and enablement materials to allow partner teams to operate independently

Requirements

• 8+ years of software engineering experience with depth in backend development, systems integration, or data/analytics engineering
• 2+ years hands-on experience applying LLMs/AI to production workflows (beyond prototypes)
• Strong proficiency in Python and JavaScript/Node.js with Git-based workflows, code review practices, and testing discipline
• Hands-on experience with LLM frameworks and patterns including prompt engineering, RAG, function calling/tool use, structured output parsing, and evaluation
• Experience building and operating multi-agent systems at scale including agent decomposition, orchestration patterns, state management, and production monitoring
• Deep familiarity with Google Cloud Platform, BigQuery, and serverless/containerized services (Cloud Functions, Cloud Run)
• Understanding of LLM failure modes and production mitigations including confidence thresholds, fallback logic, human escalation, and cost/latency management
• Proven ability to identify high-leverage problems and deliver end-to-end with minimal direction
• Fluent with AI-assisted development tools (GitHub Copilot, Cursor, Claude Code) and pragmatic AI-assisted development practices
• Strong communication skills to explain complex systems to both engineers and business stakeholders

Bonus Points

• Experience with vector databases or retrieval pipelines (Pinecone, Weaviate, ChromaDB, Qdrant, pgvector)
• Familiarity with marketing or sales platforms (Salesforce, Customer.io, HubSpot, Marketo, Outreach)
• Experience with frontend frameworks (React, Slack Block Kit) for building user-facing AI tool interfaces
• Observability tooling for AI systems (LangSmith, Weights & Biases, custom evaluation frameworks)
• Experience with workflow orchestration platforms (n8n, Temporal, Prefect, Airflow)
• Familiarity with Model Context Protocol (MCP) or similar standards
• Prior work automating marketing, sales, or customer success workflows in a B2B SaaS environment
• Active participation in open-source communities

Compensation

• In Canada, the base compensation range for this role is CAD 164,490 - CAD 197,389. Actual compensation may vary based on level, experience, and skillset as assessed throughout the interview process.
• All roles include Restricted Stock Units (RSUs).

Why You’ll Thrive at Grafana Labs

• 100% remote, global culture with a high-trust, low-ego environment
• Scaling organization with meaningful work and transparent communication
• Strong emphasis on developer productivity and AI-assisted development
• In-person onboarding and generous annual leave policy (30 days per annum, includes Grafana Shutdown Days)

Equal Opportunity & Privacy

Grafana Labs is an equal opportunities employer. For information about how personal data is used after applying, see the company's privacy policy.