Staff Backend Engineer - Grafana Databases, Loki Ingest

Details

Grafana Labs is a remote-first, open-source company building observability tooling used by millions worldwide. The Databases team owns telemetry databases including Mimir (metrics), Loki (logs), Tempo (traces), and Pyroscope (profiles). These projects are open source and operate as Cloud services and on-premise solutions. The services are implemented in Go, run on Kubernetes, and leverage cloud object storage across GCP, Azure, and AWS. This role is a remote opportunity for applicants in Canadian time zones only.

Responsibilities

• Influence the team roadmap and your own career objectives.
• Work with your team to deliver new features, iterate based on results, and drive projects from ideation through to production operations.
• Design, build, operate, and maintain critical systems with ownership of reliability, performance, and availability.
• Participate in on-call rotations and take operational ownership of the services you run.
• Embrace open-source culture and contribute to related OSS projects.
• Mentor and support other team members, participate in design discussions, and collaborate closely with teammates.
• Learn and gain deeper understanding of the cloud product and large distributed system codebases.

Requirements

• Solid experience with at least one programming language (Grafana Labs primarily uses Go; familiarity with Python, C, C++, Rust or similar is transferable).
• Experience delivering projects end-to-end: gathering requirements, brainstorming, shipping to customers, and iterating.
• Some experience with developing software that runs in the Cloud or with systems engineering.
• Some experience running complex distributed systems at scale.
• Experience writing clean, robust, and performant software that is easily maintained by others.
• Strong communication skills and the ability to work effectively in a fully remote team.
• Customer focus, pragmatic problem solving, and a bias toward action.

Bonus (Nice to Have)

• Experience working with Kubernetes.
• Experience running Kafka at scale.
• Practical experience using Grafana and Prometheus in operational roles (including on-call experience).
• Exposure to microservices architecture and distributed systems.
• Familiarity with on-call/operator/SRE tasks and infrastructure as code.

Compensation & Benefits

• Canada base compensation range: CAD 174,176 - CAD 209,011 (actual compensation may vary by level, experience, and skillset).
• Roles include Restricted Stock Units (RSUs), potential bonus, and other benefits.
• 100% remote-first company with global culture, defined career growth pathways, and a 30-day annual leave policy with 3 reserved Grafana shutdown days.

About the Team & Tech Stack

• The Databases team maintains Mimir, Loki, Tempo, and Pyroscope.
• Services are primarily written in Go and run on multiple Kubernetes clusters.
• Infrastructure uses cloud object storage and operates across major cloud providers (GCP, Azure, AWS).
• The broader ecosystem includes Prometheus, Grafana Agent, Mimir proxies, and OpenTelemetry.

Equal Opportunity & Privacy

Grafana Labs is an equal opportunity employer and will recruit, train, compensate, and promote regardless of race, religion, color, national origin, gender, disability, age, veteran status, and other characteristics. The company may utilize AI tools to assist recruitment and provides a privacy policy regarding applicant data.