Staff Backend Engineer - Grafana Second Horizon

Details

Grafana Labs is building an AI-native data intelligence system that gives agents reliable, governed access to enterprise context. The system retrieves data, metadata, definitions, lineage, quality signals, and institutional knowledge so agents can understand and act on real-time enterprise data.

This role will help build the first production services for the context layer management system. You will design and ship backend architecture for ingestion, context storage, retrieval APIs, and agent-facing integrations. The role requires comfort with ambiguity, pragmatic architectural decision-making, and evolving systems from internal experimentation to production-grade SaaS.

Responsibilities

• Design, implement, test, and operate backend services for context ingestion, context indexing, retrieval orchestration, API access, source configuration, and system administration.
• Define and build the architecture for a scalable, multi-tenant SaaS service, including tenant isolation, usage tracking, quotas, audit logs, background jobs, and reliable service boundaries.
• Build APIs and service interfaces that enable AI agents, MCP tools, CLIs, and internal applications to retrieve relevant context, provenance, confidence signals, and warnings.
• Partner across product and infrastructure to balance rapid experimentation with long-term reliability as the project moves from prototype to production.
• Instrument services with metrics, logs, traces, alerts, and dashboards, and use observability tools to understand system behavior and improve reliability.
• Help shape architecture, service boundaries, storage choices, API contracts, deployment patterns, and engineering practices for a new product area.
• Communicate effectively and contribute across teams in a dynamic, collaborative environment.
• Take full ownership of AI solutions, ensuring they are scalable, maintainable, innovative, and aligned with real user workflows.

Requirements

• Strong experience building production-grade, user-facing software systems.
• Ability to tackle complex engineering problems independently and make pragmatic architectural decisions with minimal supervision.
• Familiarity with AI technologies and frameworks, with a focus on delivering practical, high-quality solutions.
• Experience with rapid prototyping, experimentation, feedback collection, and iterative development.
• Demonstrated initiative, ownership, and ability to define scope in ambiguous environments.
• Effective communication skills and a collaborative, solutions-oriented mindset.
• Experience with LLMs, prompt engineering, and building GenAI-powered applications.
• Proven track record of delivering software to production that is actively used by customers or users.
• Exposure to cloud-native environments such as AWS, GCP, or Azure.
• Experience using observability tools to understand and troubleshoot system behavior.

Bonus Qualifications

• Experience with agent frameworks or multi-agent workflows.
• Experience as a data analyst or working with data platforms such as Looker, Tableau, Power BI, Snowflake, or Databricks.
• Experience building tools for data engineering.

How We Work

Grafana Labs is a remote-first team that meets regularly over video and does most of its work asynchronously and in writing. The company values creativity, diverse perspectives, clear communication, autonomy, ownership, and collaboration.

Benefits

• Equity and bonus, if applicable.
• 100% remote global culture.
• Career growth pathways.
• In-person onboarding.
• Global annual leave policy of 30 days per annum, including 3 Grafana Shutdown Days, subject to local legislation.
• Transparent communication, empowered teams, and an open-source culture.