Staff Software Engineer - Databases SRE

Details

Grafana Labs is seeking a Staff Software Engineer - SRE to support high-value Grafana Cloud customers by increasing the reliability of cloud databases based on Mimir, Loki, Tempo, and Pyroscope. These databases are provided as a SaaS product across AWS, GCP, and Azure in all regions.

The SRE team is embedded within the Mimir, Loki, and Tempo squads and focuses on ensuring that Grafana Cloud database products deliver exceptional reliability for high-SLA customers. The role operates at the intersection of customer needs, production systems, and product engineering.

Responsibilities

• Partner closely with product engineering squads using an embedded model.
• Own production reliability for high-SLA and complex customer environments.
• Design and implement automation to scale reliability practices and eliminate toil.
• Ensure customers meet SLO targets.
• Define and evolve per-tenant SLOs and reliability models.
• Proactively reduce SLO burn and prevent repeat incidents.
• Serve as a primary escalation point and participate in on-call rotations.
• Lead customer-impacting incident response and post-incident reviews.
• Contribute to design documents and code reviews.
• Influence feature design to ensure production scalability and operability.
• Improve alert quality and reduce noisy escalations.
• Investigate ways to reduce SLO budget burn through monitoring, automation, self-healing, and autoscaling improvements.
• Improve customer observability within their environments.
• Design and implement reliable, scalable, and fault-tolerant solutions.
• Collaborate with engineering leaders to influence product strategy, roadmaps, and technical designs.
• Teach Site Reliability Engineering practices and communicate reliability best practices early in the development lifecycle.
• Participate in incident response from investigation through resolution, post-incident review, and customer communication when required.
• Use AI coding assistants for prototyping, test generation, refactoring, documentation, and incident follow-ups within security guidelines and with code review standards.

Requirements

• 8+ years of engineering experience, including 4+ years in SRE, CRE, or production engineering. Formal customer reliability engineering experience is strongly preferred.
• Strong Kubernetes experience in AWS, GCP, or Azure.
• Familiarity with infrastructure-as-code tooling such as Helm, Terraform, and Jsonnet.
• Strong technical leadership experience, including leading teams through projects, mentoring engineers, and acting as a force multiplier.
• Experience operating multi-tenant systems in production.
• Strong experience designing and implementing SLOs.
• Experience with one or more programming languages, such as Go, Python, or Java.
• Experience with Linux operating system internals and knowledge of networking, cloud storage, and scaling.
• Excellent problem-solving and troubleshooting skills.
• Experience participating calmly and actively in blame-free incident response, following up on actions, and writing high-quality post-incident reviews.
• Ability to reason about performance, scaling, and failure modes.
• Ability to work autonomously and with self-direction within an engineering team.
• Ability to partner deeply with product engineering teams.
• Intellectual curiosity, transparency, a bias toward action, and a collaborative attitude.

Benefits

• 100% remote work and a global culture.
• Equity, bonus where applicable, and other company benefits.
• Company-funded usage budget for AI coding assistants.
• In-person onboarding.
• 30 days of annual leave per year, including three Grafana Shutdown Days, subject to local legislation.
• Career growth pathways and an empowered, transparent working culture.

The role is open to candidates in the United Kingdom, Sweden, Spain, or Germany. In Spain, the base compensation range is €94,025–€112,830. Actual compensation may vary based on level, experience, and skill set.