Staff Software Engineer - Databases, Tempo

Details

Grafana Labs is hiring a Staff Software Engineer to work on Tempo, its open-source distributed tracing backend powering Grafana Cloud Traces and Grafana Enterprise Traces. Tempo supports trace search, span-to-metric generation, and connections between tracing data, logs, metrics, and profiles across the Grafana stack.

The role focuses on evolving Tempo from a SaaS database into a platform for observability products, including App Observability, Asserts, Traces Drilldown, and AI-driven assistants. Key areas include operational excellence, autoscaling, trace aggregation, TraceQL metrics, API design, query performance, and support for large-scale, high-cardinality and AI-generated workloads.

Responsibilities

• Lead multi-quarter technical initiatives from problem framing through rollout, such as trace aggregation APIs, Limitless Tempo, autoscaling cells and customer limits, and query engine improvements.
• Own the architecture of core Tempo components, including ingestion, storage, querying, and metrics generation.
• Lead design reviews and make trade-offs involving performance, cost, and complexity.
• Design structured, deterministic, and discoverable APIs for human users, services, agents, LLM-driven assistants, and external integrators.
• Drive operational excellence against SLOs such as P99 write latency, incident recurrence, and total cost of ownership per ingested gigabyte.
• Improve automation, parameterized rollouts, alerting, and Zero Ops practices.
• Partner with Product and sibling teams, including App Observability, Asserts, Drilldown, and Grafana Assistant.
• Mentor engineers through code reviews, design feedback, pairing, and technical writing.
• Participate in on-call responsibilities and incident response for services owned or supported by the team.
• Contribute to the Tempo open-source project, review external contributions, and engage with the community.
• Use AI coding assistants for prototyping, test generation, refactoring, documentation, and incident follow-ups while maintaining strong code review and quality standards.

Example Projects

• Extending TraceQL metrics, designing LLM-friendly response types, and making Tempo a first-class data source for Grafana's AI assistant.
• Building end-to-end autoscaling for customer limits and Tempo cells, including hysteresis, predictive scaling, and safe scale-down.
• Adding guardrails for bursty, high-cardinality, agent-generated ingestion and query workloads.
• Improving query performance through new data formats, smarter query pipelines, targeted optimizations, and support for 30-day query ranges.
• Developing parameterized rollouts, push-button deployments, and tooling for multi-cell operations.
• Improving customer-facing limits, configuration, and observability to reduce escalations.

Requirements

• A track record of leading complex, multi-quarter initiatives spanning design, delivery, and operations.
• Substantial hands-on experience building and operating distributed data systems in production, such as ingestion pipelines, storage engines, or query execution systems.
• Strong software craftsmanship, including the ability to write clean, robust, performant, and maintainable software.
• Strong Go experience, or the ability to transition to Go. Experience with Rust, C, or C++ is also relevant.
• Experience owning production services, participating in on-call rotations, reducing operational toil, and using SLOs as a product feature.
• A customer-focused and pragmatic approach based on short feedback loops, MVP delivery, learning, and iteration.
• Clear communication and leadership through design documents, reviews, collaboration, and shipped code in a remote, asynchronous environment.

Bonus Qualifications

• Experience with tracing, OpenTelemetry, or large-scale observability systems.
• Experience designing query languages, SQL- or TraceQL-like engines, or APIs consumed programmatically by services or agents.
• Experience with columnar storage formats such as Parquet or purpose-built on-disk formats for analytical workloads.
• Experience operating multi-tenant, multi-cell SaaS infrastructure at scale on Kubernetes.
• Experience building structured APIs, metadata or discovery endpoints, deterministic outputs, or evaluation harnesses for AI and LLM consumers.
• Open-source contribution or maintainership experience.
• Experience using Grafana, Prometheus, Loki, or Tempo on-call, including in a homelab.
• Experience working in a fully remote, globally distributed team.

Work Environment

This is a remote opportunity. Grafana Labs is a remote-first company that works primarily asynchronously and in writing, with regular video meetings. The company supports modern AI coding assistants and provides a company-funded usage budget. Frontier models mentioned include GPT-5.6, Claude Fable, and Gemini Pro 3.1.

Compensation And Benefits

The Germany base compensation range is EUR 109,709 to EUR 131,651 per year. Actual compensation may vary based on level, experience, and skills. Benefits include equity, a bonus where applicable, and other company benefits. Grafana Labs also offers a global annual leave policy of 30 days per year, including three Grafana Shutdown Days, and in-person onboarding. Local legislation applies.

Grafana Labs is an equal opportunities employer.