Staff Software Engineer - Databases, Tempo

Details

Grafana Labs is the company behind the open observability cloud and Grafana Cloud, a fully managed observability platform. We are a 100% remote company with a global team and an open-source culture. This role is a remote opportunity; we are particularly interested in applicants located in Spain, Sweden, United Kingdom, Ireland, or Germany.

Tempo is the open-source distributed tracing backend that powers Grafana Cloud Traces and Grafana Enterprise Traces. Tempo enables searching traces, generating metrics from spans, and connecting tracing data with logs, metrics, and profiles across the Grafana stack. In 2026 Tempo is shifting from foundational work to product and operational excellence and evolving into a platform to power Grafana’s next-generation observability products.

Responsibilities

• Lead multi-quarter technical initiatives from problem framing through rollout (examples: trace aggregation APIs, Limitless Tempo, autoscaling cells and customer limits, query engine improvements).
• Own the architecture of core Tempo components: ingestion, storage, query, and metrics generation; drive design reviews and document trade-offs.
• Design APIs for human and agent consumption; shape structured, discoverable interfaces for downstream products and LLM-driven assistants.
• Drive operational excellence: own outcomes against concrete SLOs (P99 write latency, incident recurrence, TCO per ingested GB), reduce toil via automation, and enable parameterized rollouts.
• Partner with Product and related teams (App Observability, Asserts, Drilldown, Grafana Assistant) to understand consumption patterns and unblock product work.
• Mentor engineers through code review, design feedback, pairing, and clear technical writing.
• Participate in on-call for the services you help build and contribute strongly to incident response and post-incident learning.
• Contribute to open source: engage the Tempo community, review external contributions, and help steer the project publicly.

Example problems you may work on:

• Trace aggregation and higher-density APIs, extending TraceQL metrics and designing LLM-friendly response types.
• End-to-end autoscaling for Tempo cells including hysteresis, predictive scaling, and safe scale-down.
• Guardrails and optimizations for agent-scale ingestion and high-cardinality workloads.
• Query performance improvements: new data formats, smarter query pipelines, and optimizations for large query ranges.
• Rollouts and multi-cell operations: parameterized rollouts, tooling for safe growth to triple-digit cell counts.
• Customer-facing limits and self-service configuration to reduce escalations.

Requirements

(“What Makes You a Great Fit”)

• Technical leadership: proven track record leading complex initiatives across design, delivery, and operations.
• Deep systems experience: substantial hands-on experience building and operating distributed data systems in production (ingestion pipelines, storage engines, query execution, or similar).
• Strong software craftsmanship: ability to write clean, robust, performant code and make pragmatic trade-offs.
• Strong Go, or a clear path to Go: Tempo is written in Go; deep experience in other systems languages (Rust, C, C++) translates well.
• Operational mindset: owned production services, carried a pager, reduced toil, and treated SLOs as product features.
• Customer focus and pragmatism: break problems into short feedback loops and iterate based on learning.
• Leadership through writing and collaboration: clear communication in a fully remote, asynchronous environment.

Bonus Points

• Experience with tracing, OpenTelemetry, or large-scale observability systems.
• Experience designing query languages or SQL/TraceQL-like engines and APIs for programmatic consumers.
• Experience with columnar storage formats (e.g., Parquet) or purpose-built on-disk formats for analytical workloads.
• Experience operating multi-tenant, multi-cell SaaS infrastructure at scale on Kubernetes.
• Experience building for AI/LLM consumers: structured APIs, metadata/discovery endpoints, deterministic outputs, evaluation harnesses.
• Open-source contribution or maintainership and comfort engaging community in the open.
• Experience as an on-call user of Grafana, Prometheus, Loki, or Tempo.
• Experience in fully remote, globally distributed teams.

How We Work

We are remote-first, meet regularly over video, and do most work asynchronously in writing. We value transparency, autonomy, and clear communication. Tempo is relied upon by prominent global organizations, and Staff engineers are expected to contribute ideas that make Tempo reliable, useful, and loved.

We invest heavily in developer productivity and allow modern AI coding assistants within security guidelines, backed by a company-funded usage budget. Engineers also have access to frontier models (e.g., GPT-5.6, Claude Fable, Gemini Pro 3.1) for prototyping, test generation, refactors, documentation, and incident follow-ups—paired with strong code review and quality standards.

Compensation & Rewards

• Spain base compensation range: EUR 94,025 - EUR 112,830 (actual compensation may vary based on level, experience, and skillset).
• All roles include Restricted Stock Units (RSUs).
• Compensation ranges are country-specific; recruiters will discuss market-specific pay ranges for applicants outside the listed location.

Additional Notes

• 100% remote company with a global culture.
• In-person onboarding and a global annual leave policy of 30 days per annum (3 days reserved for Grafana Shutdown Days), subject to local legislation.
• Grafana Labs is an equal opportunity employer.

You may not meet every requirement; if this role excites you, we encourage you to apply.