Staff Backend Engineer - Second Horizon

Details

Grafana Labs is the company behind the open observability cloud. Grafana Cloud is a fully managed observability platform built for scale with a focus on open source, open standards, and open ecosystems. The company is 100% remote with a global team and serves millions of users and thousands of customers.

This role is part of a skunkworks initiative to bring observability and reliable, governed enterprise context to AI agents and other consumers across the business. The team is building an AI-native data intelligence / context system that provides agents with reliable access to context: data, metadata, definitions, lineage, quality signals, and institutional knowledge. This is an early-stage, high-autonomy role responsible for designing and shipping the first production backend services for that context layer.

Responsibilities

• Design, implement, test, and operate core backend services for context ingestion, indexing, retrieval orchestration, API access, source configuration, and system administration.
• Define and build a scalable SaaS foundation for a multi-tenant service, including tenant isolation, usage tracking, quotas, audit logs, background jobs, and reliable service boundaries.
• Build APIs and service interfaces to power agent-facing retrieval workflows, delivering context, provenance, confidence signals, and warnings to AI agents, MCP tools, CLIs, and internal applications.
• Partner across product and infrastructure to balance fast experimentation with long-term reliability as the project moves from prototype to production.
• Instrument and operate services with metrics, logs, traces, alerts, and dashboards; use observability tools to understand system behavior and improve reliability.
• Contribute to technical direction: architecture, service boundaries, storage choices, API contracts, deployment patterns, and engineering practices for a new product area.
• Communicate effectively across a dynamic, collaborative environment and take ownership of the solutions you develop.

Requirements

• Strong engineering experience building production-grade, user-facing software systems and delivering software used by real users.
• Experience with LLMs, prompt engineering, and building applications powered by generative AI.
• Exposure to cloud-native environments (AWS, GCP, or Azure).
• Experience using observability tools to understand and troubleshoot system behavior (metrics, logs, traces, alerts, dashboards).
• Comfortable iterating quickly, releasing prototypes, collecting feedback, and maturing systems to be production-grade.
• Demonstrated initiative, ownership, and ability to work through ambiguity.

Nice to have

• Experience building or working with agent frameworks or multi-agent workflows.
• Experience as a data analyst or working with data platforms (Looker, Tableau, PowerBI, Snowflake, Databricks).
• Experience building tools for data engineering.

Compensation & Benefits

• In the US, base compensation range: $174,986 - $209,983. Actual compensation may vary based on level, experience, and skillset. Benefits include equity, bonus (if applicable), and other benefits listed by Grafana Labs.
• 100% remote company; in-person onboarding is provided.
• Global annual leave policy of 30 days per annum (subject to local legislation) with Grafana Shutdown Days.

Additional details

• This is a remote opportunity targeting candidates located in the United States or Canada (residents of Quebec are not eligible).
• Grafana Labs is an equal opportunity employer and may utilize AI tools in its recruitment process.