Staff Software Engineer - Databases, Tempo

Details

Grafana Labs is the company behind the open observability cloud. Grafana Cloud is a fully managed observability platform built for scale. We are a 100% remote company with 1,600+ team members across 40+ countries.

This is a remote opportunity, and we would be interested in applicants located in Spain, Sweden, United Kingdom, Ireland or Germany.

Role overview

We build Tempo, the open-source distributed tracing backend behind Grafana Cloud Traces and Grafana Enterprise Traces (GET). Tempo enables searching traces, generating metrics from spans, and connecting tracing data with logs, metrics, and profiles across the Grafana stack. In 2026 Tempo is shifting from foundational work to product and operational excellence and evolving from a SaaS database into a platform powering Grafana’s next generation of observability products.

Over the next year you will help make Grafana Cloud Traces reliable for customers, achieve operational excellence at scale (growing cell counts and autoscaling), evolve Tempo into a platform enabler (higher-density APIs, trace aggregation, TraceQL metrics math, machine/LLM-friendly interfaces), push query and ingestion performance, and prepare Tempo for agent-driven and AI-powered workflows.

Responsibilities

• Lead multi-quarter technical initiatives from problem framing through rollout (examples: trace aggregation APIs, Limitless Tempo, autoscaling cells and customer limits, query engine improvements).
• Own architecture of core Tempo components: ingestion, storage, query, and metrics generation. Drive design reviews, make trade-offs on performance, cost, and complexity, and document design rationale.
• Design APIs for humans and agents, shaping Tempo’s interfaces to be structured, deterministic, and discoverable for downstream products and agents.
• Drive operational excellence with concrete SLOs (e.g., P99 write latency, incident recurrence, TCO per ingested GB), automation, parameterized rollouts, and actionable alerts.
• Partner with Product and sibling teams (App Observability, Asserts, Drilldown, Grafana Assistant) to understand consumption and unblock downstream use cases.
• Mentor engineers through code review, design feedback, pairing, and clear technical writing.
• Participate in on-call for the services you help build and contribute to incident response and post-incident learning.
• Contribute to open source: engage the community, review external contributions, and help steer Tempo in the open.

Example problems you could work on:

• Trace aggregation and higher-density APIs, extend TraceQL metrics, design LLM-friendly response types.
• End-to-end autoscaling: customer limits, Tempo cells with hysteresis, predictive scaling for spikes, safe scale-down.
• Agent-scale ingestion and query: guardrails for bursty, high-cardinality, agent-generated workloads.
• Query performance: new data formats, smarter query pipelines, targeted optimizations for Drilldown and Traces workflows and 30-day query ranges.
• Rollouts and multi-cell operations: parameterized rollouts, push-button deploys, tooling to grow safely into triple-digit cell counts.

Requirements

• Technical leadership: track record of leading complex, multi-quarter initiatives spanning design, delivery, and operations.
• Deep systems experience: substantial hands-on experience building and operating distributed data systems in production (ingestion pipelines, storage engines, query execution, or similar).
• Strong software craftsmanship: write clean, robust, performant software and make pragmatic optimization decisions.
• Strong Go, or a clear path to Go. Deep experience in other systems languages (Rust, C, C++) translates well.
• Operational mindset: owned production services, carried a pager, reduced toil, and treated SLOs as a product feature.
• Customer focus and pragmatism: break problems into short feedback loops (MVP, learn, iterate).
• Leadership through writing and collaboration: lead via design docs, reviews, and shipped code in an asynchronous, remote environment.

Bonus points

• Experience with tracing, OpenTelemetry, or large-scale observability systems.
• Experience designing query languages, SQL/TraceQL-like engines, or APIs consumed programmatically by services/agents.
• Experience with columnar storage formats (e.g., Parquet) or purpose-built on-disk formats for analytical workloads.
• Experience operating multi-tenant, multi-cell SaaS infrastructure at scale on Kubernetes.
• Experience building for AI/LLM consumers: structured APIs, metadata/discovery endpoints, deterministic outputs, evaluation harnesses.
• Open-source contribution or maintainership and comfort engaging a public community.
• Experience as an on-call user of Grafana, Prometheus, Loki, or Tempo.
• Experience in a fully remote, globally distributed team.

How we work

We are remote-first, meet regularly over video, and do most work asynchronously and in writing. Tempo is relied upon to monitor critical applications and infrastructure; the team expects contributions that make Tempo more reliable and useful.

Grafana Labs invests in developer productivity and allows modern AI coding assistants (company-funded usage budget) within security guidelines. You’ll also have access to frontier models (examples listed in the posting).

Compensation & rewards

In the United Kingdom, the base compensation range for this role is GBP 103,958 - GBP 124,750. Actual compensation may vary by level, experience, and skillset. Benefits include equity, bonus (if applicable), and other benefits linked from the company careers page.

Other notes

• 100% remote company with in-person onboarding.
• Global annual leave policy of 30 days per annum (subject to local legislation).