Senior Backend Engineer - Databases Tempo | US | Remote

Details

Grafana Labs is hiring a Senior Backend Engineer to work on Tempo, its open-source distributed tracing backend behind Grafana Cloud Traces and Grafana Enterprise Traces. The role focuses on evolving Tempo into a platform for observability products and AI-driven workflows, while improving reliability, scalability, query performance, and operational excellence.

Responsibilities

• Deliver significant, multi-month technical projects from problem framing through rollout, including trace aggregation APIs, Limitless Tempo, autoscaling cells and customer limits, and query engine improvements.
• Own major components of Tempo across ingestion, storage, querying, and metrics generation.
• Write and present design documents, participate in design reviews, make trade-offs involving performance, cost, and complexity, and document technical decisions.
• Build structured, deterministic, and discoverable APIs for Grafana products, LLM-driven assistants, and external integrators.
• Own operational outcomes against SLOs such as P99 write latency, incident recurrence, and total cost of ownership per ingested gigabyte.
• Improve automation, parameterized rollouts, actionable alerting, and Zero Ops capabilities.
• Collaborate with Product and sibling teams, including App Observability, Asserts, Drilldown, and Grafana Assistant.
• Mentor junior and mid-level engineers through code reviews, design feedback, and pairing.
• Participate in on-call duties, incident response, and post-incident learning.
• Contribute to the Tempo open-source project, review external contributions, and engage with the community.

Example Projects

• Extending TraceQL metrics and designing LLM-friendly response types.
• Building autoscaling systems with customer limits, hysteresis, predictive scaling, and safe scale-down.
• Supporting bursty, high-cardinality, agent-generated workloads.
• Improving query performance through new data formats, smarter query pipelines, and targeted optimizations.
• Developing parameterized rollouts and multi-cell operations tooling.
• Building customer-facing configuration and observability for limits and self-service.

Requirements

• Track record of delivering complex, multi-month projects spanning design, delivery, and operations.
• Significant hands-on experience building and operating distributed data systems in production, such as ingestion pipelines, storage engines, or query execution systems.
• Strong software craftsmanship, including the ability to write clean, robust, performant, and maintainable software.
• Strong experience with Go, or relevant systems-language experience in Rust, C, or C++ with the ability to develop in Go.
• Experience owning production services, participating in on-call rotations, reducing operational toil, and working with SLOs.
• Customer-focused and pragmatic approach, with the ability to break complex problems into short feedback loops.
• Clear written and verbal communication skills for design documents, reviews, and asynchronous collaboration in a fully remote environment.

Bonus Qualifications

• Experience with tracing, OpenTelemetry, or large-scale observability systems.
• Experience designing query languages, SQL- or TraceQL-like engines, or programmatic APIs for services and agents.
• Experience with columnar storage formats such as Parquet or purpose-built on-disk formats for analytical workloads.
• Experience operating multi-tenant, multi-cell SaaS infrastructure at scale on Kubernetes.
• Experience building structured APIs, metadata or discovery endpoints, deterministic outputs, or evaluation harnesses for AI and LLM consumers.
• Open-source contribution or maintainership experience.
• Experience using Grafana, Prometheus, Loki, or Tempo on call or in a homelab.
• Experience working in a fully remote, globally distributed team.

Work Environment

Grafana Labs is a remote-first company. The team works primarily asynchronously and communicates through writing and regular video meetings. The role includes on-call participation and contribution to an open-source project. Modern AI coding assistants may be used within security guidelines, with company-funded usage support.

Compensation and Benefits

The US on-target earnings/base compensation range is USD $154,445–$185,334. Actual compensation may vary based on level, experience, and skillset. The role includes Restricted Stock Units (RSUs). Benefits include a global annual leave policy of 30 days per year, with 3 days reserved for Grafana Shutdown Days. Grafana also provides in-person onboarding.