Used Tools & Technologies
Not specified
Required Skills & Competences
Tag name is followed by "@" symbol and proficiency level value.
About proficiency levels:
- 1-2 — basic awareness. Minimal hands-on experience, and a rudimentary understanding of the technology's purpose;
- 3-6 — daily use. Comfortable and regular usage, capable of handling common tasks and challenges related to the technology;
- 7-9 — you are an expert, you can teach others, you know all the pitfalls and tricks;
- 10 — exceptional knowledge, comprehensive understanding, and adeptness in all aspects of the technology, including advanced problem-solving. Think twice before claiming or demanding such level.
Security @ 4
Go @ 7
Grafana @ 4
Kubernetes @ 4
Prometheus @ 4
SQL @ 4
Leadership @ 4
Communication @ 4
Parquet @ 4
Rust @ 7
API @ 4
Technical Leadership @ 4
LLM @ 4
OpenTelemetry @ 4
Observability @ 4
AI @ 4
- 1-2 — basic awareness. Minimal hands-on experience, and a rudimentary understanding of the technology's purpose;
- 3-6 — daily use. Comfortable and regular usage, capable of handling common tasks and challenges related to the technology;
- 7-9 — you are an expert, you can teach others, you know all the pitfalls and tricks;
- 10 — exceptional knowledge, comprehensive understanding, and adeptness in all aspects of the technology, including advanced problem-solving. Think twice before claiming or demanding such level.
Details
Grafana Labs builds Grafana Cloud, a managed observability platform. Tempo is the open-source distributed tracing backend powering Grafana Cloud Traces and Grafana Enterprise Traces. This role is a remote Staff Software Engineer position focused on Tempo, evolving it from a SaaS database into a platform that powers Grafana’s next-generation observability products, improving performance, scalability, operational excellence, and enabling AI/LLM-driven workflows.
Responsibilities
- Lead multi-quarter technical initiatives from problem framing through rollout (e.g., trace aggregation APIs, autoscaling cells, query engine improvements).
- Own architecture of core Tempo components: ingestion, storage, query, and metrics generation; drive design reviews and document design rationale.
- Design APIs for humans and agents; shape structured, deterministic, discoverable interfaces for downstream products and LLM-driven assistants.
- Drive operational excellence and own outcomes against concrete SLOs (P99 write latency, incident recurrence, TCO per ingested GB).
- Partner with Product and sibling teams (App Observability, Asserts, Drilldown, Grafana Assistant) to understand consumption and unblock them.
- Mentor engineers through code review, design feedback, pairing, and high-quality documentation.
- Participate in on-call for owned services and lead incident response and post-incident learning.
- Contribute to the Tempo open-source project: engage the community and review external contributions.
- Use modern AI coding assistants as desired (company-funded usage budget) within security guidelines.
Example problems
- Trace aggregation and higher-density APIs: extend TraceQL metrics, design LLM-friendly response types.
- Autoscaling end to end: customer limits, Tempocells autoscaling with hysteresis and predictive scaling.
- Agent-scale ingestion and query: guardrails for bursty, high-cardinality, agent-generated workloads.
- Query performance: new data formats, smarter query pipelines, targeted optimizations for Drilldown and Traces workflows, and 30-day query ranges.
- Rollouts and multi-cell operations: parameterized rollouts, push-button deploys, tooling for safe growth into triple-digit cell counts.
- Limits and self-service: customer-facing configuration and observability to reduce escalations.
Requirements / What Makes You a Great Fit
- Technical leadership: track record leading complex, multi-quarter initiatives spanning design, delivery, and operations.
- Deep systems experience: substantial hands-on experience building and operating distributed data systems in production (ingestion pipelines, storage engines, query execution, or similar).
- Strong software craftsmanship: write clean, robust, performant software and make pragmatic trade-offs.
- Strong Go, or a path to it: Tempo is written in Go; deep experience in other systems languages (Rust, C, C++) translates well.
- Operational mindset: owned production services, carried a pager, reduced toil, and treated SLOs as a product feature.
- Customer focus and pragmatism: break problems into short feedback loops and iterate from MVPs.
- Leadership through writing and collaboration: lead through design docs, reviews, and shipped code in a fully remote asynchronous environment.
Bonus Points
- Experience with tracing, OpenTelemetry, or large-scale observability systems.
- Experience designing query languages, SQL/TraceQL-like engines, or APIs intended to be consumed programmatically.
- Experience with columnar storage formats (e.g., Parquet) or purpose-built on-disk formats for analytical workloads.
- Experience operating multi-tenant, multi-cell SaaS infrastructure at scale on Kubernetes.
- Experience building for AI/LLM consumers: structured APIs, metadata/discovery endpoints, deterministic outputs, evaluation harnesses.
- Open-source contribution or maintainership, and comfort engaging a community in the open.
- Experience as an on-call user of Grafana, Prometheus, Loki, or Tempo.
- Experience in a fully remote, globally distributed team.
How We Work
- Remote-first, asynchronous collaboration with regular video meetings. Emphasis on transparency, autonomy, and written communication.
- Expect to contribute ideas that improve reliability and usability of Tempo and the broader Grafana stack.
- In-person onboarding is provided.
Compensation & Rewards
- Sweden base compensation range: SEK 878,578 - SEK 1,054,294. Actual compensation may vary by level, experience, and market.
- Benefits include equity, bonus (if applicable), and other benefits listed on Grafana’s careers page. Country-specific compensation ranges apply.
Why You’ll Thrive at Grafana Labs
- 100% remote global culture, scaling organization, transparent communication, innovation-driven environment, open-source roots, empowered teams, career growth pathways, approachable leadership, and in-person onboarding.
- Global annual leave policy of 30 days per annum (with 3 shutdown days), subject to local legislation.
Equal Opportunity Employer
Grafana Labs is an equal opportunities employer and welcomes applications from everyone. The company may utilize AI tools in its recruitment process to assist in matching CVs to job postings.