Tech Stack
Tag name is followed by "@" symbol and proficiency level value.
About proficiency levels:
- 1-2 — basic awareness. Minimal hands-on experience, and a rudimentary understanding of the technology's purpose;
- 3-6 — daily use. Comfortable and regular usage, capable of handling common tasks and challenges related to the technology;
- 7-9 — you are an expert, you can teach others, you know all the pitfalls and tricks;
- 10 — exceptional knowledge, comprehensive understanding, and adeptness in all aspects of the technology, including advanced problem-solving. Think twice before claiming or demanding such level.
AI @ 7
AWS @ 3
Azure @ 3
Codex
Design Patterns
GCP @ 3
Go @ 4
Grafana @ 9
Helm @ 3
Java @ 4
Kubernetes @ 3
Linux @ 4
Networking @ 4
Observability
Python @ 4
SRE @ 9
Security
Terraform @ 3
- 1-2 — basic awareness. Minimal hands-on experience, and a rudimentary understanding of the technology's purpose;
- 3-6 — daily use. Comfortable and regular usage, capable of handling common tasks and challenges related to the technology;
- 7-9 — you are an expert, you can teach others, you know all the pitfalls and tricks;
- 10 — exceptional knowledge, comprehensive understanding, and adeptness in all aspects of the technology, including advanced problem-solving. Think twice before claiming or demanding such level.
Details
The Opportunity
We are looking for a Senior Software Engineer - SRE to help support our highest value Grafana Cloud customers by increasing the reliability of our Cloud databases that are based on Mimir, Loki, Tempo, and Pyroscope. We provide these databases as a SaaS product from AWS, GCP, and Azure across all regions.
What You’ll Be Doing
The SRE team is embedded within the Mimir and Loki squads and focuses on ensuring that Grafana Cloud’s database products deliver exceptional reliability for our highest-SLA customers. In this role, you will:
- Partner closely with product engineering squads (embedded model)
- Own production reliability for high-SLA and complex customer environments
- Design and implement automation to scale our reliability practices
- Ensure our customers meet our SLO targets
- Define and evolve per-tenant SLOs and reliability models
- Proactively reduce SLO burn to prevent repeat incidents
- Serve as a primary escalation point and on-call for relevant incidents
- Lead customer-impacting incident response and post-incident reviews
- Contribute to design docs and code reviews
- Influence feature design to ensure production scalability and operability
- Build automation to eliminate toil where needed
- Improve alert quality and reduce noisy escalations
There is an on-call component to this role. Grafana Labs hires globally (remote-first) to ensure on-call remains healthy and aligned to approximately 12 daylight hours per day, working closely with counterparts in other regions to provide balanced coverage and shared ownership.
Grafana Labs invests heavily in developer productivity, including modern AI coding assistants as part of your daily workflow (within security guidelines) and a company-funded usage budget. They encourage pragmatic AI-assisted development (faster prototyping, test generation, refactors, documentation, and incident follow-ups) paired with strong code review and quality standards. You’ll also have access to frontier models (e.g., GPT-Codex 5/3, Claude Opus 4.6, Gemini 3 Pro).
What Makes You a Great Fit
- Regular 1:1s to with your manager and colleagues
- Reviewing and creating SLOs, proactively investigating ways to reduce budget burn for those SLOs (including improvements to monitoring, automation, increasing self-healing, auto-scaling, etc.)
- Improve observability of customers within their environments
- Designing and implementing solutions to ensure reliability and scalability of your environments can meet rapidly increasing demands
- Develop fault-tolerant design patterns ensuring reliability is considered at all stages of the service lifecycle
- Collaborate with Engineering Leaders to define and influence product strategy, roadmaps, and technical designs
- Participate in PR review and collaborate with other engineers on their Design Docs
- Teach others about Site Reliability Engineering and communicate best practices to be applied early in development of new features and functionality
- Participate in Incident Response when applicable (investigation through resolution, PIR, and communication with customers via Bridge calls where necessary)
Requirements
- 6+ years engineering experience, 3+ in SRE/CRE/production engineering (strong preference for formal customer reliability engineering experience)
- Strong Kubernetes experience in AWS, GCP, or Azure, and familiarity with infrastructure-as-code tooling (Helm, Terraform, Jsonnet, etc.)
- Experience operating multi-tenant systems in production
- Strong experience designing and implementing SLOs
- Experience with one or more programming languages (e.g. Go, Python, Java, etc.)
- Experience with Linux operating systems internals, and some knowledge of networking, cloud storage, and scaling
- Excellent problem-solving and troubleshooting skills
- Experience participating in blame-free Incident Response, following up on actions, and writing high quality PIRs (Post Incident Reviews / post-mortem documents)
- Ability to reason about performance, scaling, and failure modes
- Comfortable working with autonomy and self-direction
- Ability to partner deeply with product engineering teams
- Intellectual curiosity, transparency, high bias towards action, and kindness
Compensation & Rewards
In the United States, the compensation range for this role is USD 154,445 - USD 185,334. Actual compensation may vary based on level, experience, and skillset as assessed throughout the interview process. All roles include Restricted Stock Units (RSUs).