Used Tools & Technologies
Not specified
Required Skills & Competences
Tag name is followed by "@" symbol and proficiency level value.
About proficiency levels:
- 1-2 — basic awareness. Minimal hands-on experience, and a rudimentary understanding of the technology's purpose;
- 3-6 — daily use. Comfortable and regular usage, capable of handling common tasks and challenges related to the technology;
- 7-9 — you are an expert, you can teach others, you know all the pitfalls and tricks;
- 10 — exceptional knowledge, comprehensive understanding, and adeptness in all aspects of the technology, including advanced problem-solving. Think twice before claiming or demanding such level.
Security @ 4
Go @ 7
Grafana @ 4
Kubernetes @ 4
Prometheus @ 4
SQL @ 4
Leadership @ 4
Communication @ 4
Parquet @ 4
Rust @ 7
API @ 4
Technical Leadership @ 4
LLM @ 4
OpenTelemetry @ 4
Observability @ 4
AI @ 4
- 1-2 — basic awareness. Minimal hands-on experience, and a rudimentary understanding of the technology's purpose;
- 3-6 — daily use. Comfortable and regular usage, capable of handling common tasks and challenges related to the technology;
- 7-9 — you are an expert, you can teach others, you know all the pitfalls and tricks;
- 10 — exceptional knowledge, comprehensive understanding, and adeptness in all aspects of the technology, including advanced problem-solving. Think twice before claiming or demanding such level.
Details
Grafana Labs builds the open observability cloud and is the company behind Grafana Cloud, a fully managed observability platform. Tempo is the open-source distributed tracing backend used by Grafana Cloud Traces and Grafana Enterprise Traces. This role focuses on evolving Tempo from a SaaS database into a platform that powers Grafana’s next generation of observability products.
Responsibilities
- Lead multi-quarter technical initiatives from problem framing through rollout (examples: trace aggregation APIs, Limitless Tempo, autoscaling cells, query engine improvements).
- Own the architecture of core Tempo components: ingestion, storage, query, and metrics generation. Drive design reviews, document trade-offs and rationale.
- Design APIs for humans and agents to enable product and LLM-driven consumption (structured, deterministic, discoverable interfaces).
- Drive operational excellence and own outcomes against concrete SLOs (P99 write latency, incident recurrence, TCO per ingested GB). Reduce toil through automation, parameterized rollouts, and actionable alerts.
- Partner with Product and sibling teams (App Observability, Asserts, Drilldown, Grafana Assistant) to understand consumers of Tempo and ship what unblocks them.
- Mentor engineers through code review, design feedback, pairing, and clear written communication.
- Participate in on-call for the services you help build; be a force multiplier in incident response and post-incident learning.
- Contribute to open source: engage the community, review external contributions, and help steer the project in the open.
Requirements
- Demonstrated technical leadership: track record leading complex, multi-quarter initiatives spanning design, delivery, and operations.
- Deep systems experience building and operating distributed data systems in production: ingestion pipelines, storage engines, query execution, or similar.
- Strong software craftsmanship: write clean, robust, maintainable, performant software and make pragmatic decisions about optimization vs. shipping.
- Strong Go experience, or a path to it. (Tempo is written in Go; deep experience in other systems languages such as Rust, C, or C++ translates well.)
- Operational mindset: owned production services, carried a pager, reduced toil, and treated SLOs as a product feature.
- Customer focus and pragmatism: break complex problems into short feedback loops (MVP, learn, iterate).
- Leadership through writing and collaboration in a fully remote, asynchronous environment.
Bonus Points
- Experience with tracing, OpenTelemetry, or large-scale observability systems.
- Experience designing query languages, SQL/TraceQL-like engines, or programmatically consumable APIs.
- Experience with columnar storage formats (e.g., Parquet) or purpose-built on-disk formats for analytical workloads.
- Experience operating multi-tenant, multi-cell SaaS infrastructure at scale on Kubernetes.
- Experience building for AI/LLM consumers: structured APIs, metadata/discovery endpoints, deterministic outputs, evaluation harnesses.
- Open-source contribution or maintainership; comfort engaging a community in the open.
- Experience as an on-call user of Grafana, Prometheus, Loki, or Tempo.
- Experience in a fully remote, globally distributed team.
How we work
We are remote-first, working mostly asynchronously and in writing. Tempo’s team expects contributions that improve reliability, usability, and developer experience. Developer productivity is emphasized, including optional AI coding assistants (company-funded usage budget) and access to frontier models under security guidelines.
Example problems you could work on
- Trace aggregation and higher-density APIs: extend TraceQL metrics and design LLM-friendly response types.
- Autoscaling end to end: customer limits and Tempo cells, with hysteresis and predictive scaling.
- Agent-scale ingestion and query: guardrails for bursty, high-cardinality, agent-generated workloads.
- Query performance: new data formats, smarter query pipelines, and targeted optimizations for common Drilldown and Traces workflows.
- Rollouts and multi-cell operations: parameterized rollouts and tooling to safely grow cell counts without proportional alert noise.
Compensation & Rewards
In the Republic of Ireland, the base compensation range for this role is EUR 117,600 - EUR 141,120. Actual compensation may vary based on level, experience, and skillset as assessed in the interview process. Benefits include equity, bonus (if applicable), and other benefits listed on the company careers page.
Why You’ll Thrive at Grafana Labs
- 100% remote, global culture; transparent communication and innovation-driven environment.
- Scaling organization with meaningful work and career growth pathways.
- Open source roots and empowered teams with a high-trust, low-ego culture.
- In-person onboarding and a global annual leave policy (30 days per annum, subject to local legislation).