Senior Backend Engineer - Databases Tempo | Canada | Remote

📍 Canada

CAD 164,500-197,400 per year

SENIOR

✅ Remote

Tech Stack
Tag name is followed by "@" symbol and proficiency level value. About proficiency levels:

1-2 — basic awareness. Minimal hands-on experience, and a rudimentary understanding of the technology's purpose;

3-6 — daily use. Comfortable and regular usage, capable of handling common tasks and challenges related to the technology;

7-9 — you are an expert, you can teach others, you know all the pitfalls and tricks;

10 — exceptional knowledge, comprehensive understanding, and adeptness in all aspects of the technology, including advanced problem-solving. Think twice before claiming or demanding such level.

AI @ 4 API @ 4 Communication @ 7 Go @ 7 Grafana @ 4 Kubernetes @ 4 LLM @ 4 Observability @ 4 OpenTelemetry @ 4 Parquet @ 4 Prometheus @ 4 Rust @ 7 SQL @ 4

Details

Grafana Labs is hiring a Senior Backend Engineer to work on Tempo, its open-source distributed tracing backend behind Grafana Cloud Traces and Grafana Enterprise Traces. The role focuses on evolving Tempo into a scalable platform for observability products, AI-driven assistants, and high-volume distributed data workloads.

The position is fully remote and open to candidates in Canada and the United States. The team works asynchronously, collaborates across product and engineering teams, and contributes to open source.

Responsibilities

Deliver significant, multi-month technical projects from problem framing through rollout, including trace aggregation APIs, Limitless Tempo, autoscaling cells and customer limits, and query engine improvements.
Own major components of Tempo across ingestion, storage, query execution, and metrics generation.
Write and present design documents, participate in design reviews, and make trade-offs involving performance, cost, and complexity.
Build structured, deterministic, discoverable APIs for Grafana products, LLM-driven assistants, agents, and external integrators.
Improve operational excellence using SLOs, automation, parameterized rollouts, actionable alerts, and toil reduction.
Work with Product and sibling teams including App Observability, Asserts, Drilldown, and Grafana Assistant.
Mentor junior and mid-level engineers through code reviews, design feedback, and pairing.
Participate in on-call, incident response, and post-incident learning.
Contribute to the Tempo open-source project, review external contributions, and engage with the community.

Example Projects

Extending TraceQL metrics and creating LLM-friendly response types.
Building end-to-end autoscaling for customer limits and Tempo cells.
Developing guardrails for bursty, high-cardinality, agent-generated workloads.
Improving query performance through new data formats, query pipelines, and targeted optimizations.
Supporting parameterized rollouts and multi-cell operations at triple-digit cell counts.
Improving customer-facing configuration and observability for limits and self-service.

Requirements

Track record of delivering complex, multi-month projects spanning design, implementation, delivery, and operations.
Significant hands-on experience building and operating distributed data systems in production, such as ingestion pipelines, storage engines, or query execution systems.
Strong software craftsmanship, including the ability to write clean, robust, performant, and maintainable software.
Strong Go experience or experience with systems languages such as Rust, C, or C++ and a path to proficiency in Go.
Experience owning production services, participating in on-call rotations, reducing toil, and working with service-level objectives.
Customer-focused and pragmatic approach, with the ability to use short feedback loops to analyze, design, deliver, and iterate.
Strong written and verbal communication skills for a fully remote, asynchronous environment.

Bonus Qualifications

Experience with tracing, OpenTelemetry, or large-scale observability systems.
Experience designing query languages, SQL- or TraceQL-like engines, or APIs for programmatic service and agent consumption.
Experience with columnar storage formats such as Parquet or purpose-built analytical storage formats.
Experience operating multi-tenant, multi-cell SaaS infrastructure on Kubernetes.
Experience building structured APIs, metadata and discovery endpoints, deterministic outputs, or evaluation harnesses for AI and LLM consumers.
Open-source contribution or maintainership experience.
Experience using Grafana, Prometheus, Loki, or Tempo on-call.
Experience working in a fully remote, globally distributed team.

Benefits

CAD $164,490–$197,389 OTE/base compensation range in Canada.
Restricted Stock Units (RSUs).
100% remote work and a global, collaborative culture.
Company-funded usage budget for AI coding assistants.
In-person onboarding.
Global annual leave policy of 30 days per year, including three Grafana Shutdown Days.
Career growth opportunities and transparent communication.

Grafana Labs is an equal opportunities employer.