Staff Backend Software Engineer - Loki

Details

Grafana Labs is hiring a Staff Backend Software Engineer to join the Databases team and work on Loki, its open-source log aggregation system. The role supports Grafana Cloud, on-premise deployments, and distributed telemetry databases used across major cloud providers.

The Databases team works on Mimir for metrics, Loki for logs, Tempo for traces, and Pyroscope for profiles. These are multi-tenant distributed systems implemented primarily in Go and running on Kubernetes across Google Cloud Platform, Microsoft Azure, and Amazon Web Services. The broader engineering organization also works on Prometheus, Grafana Alloy, Mimir proxies, and OpenTelemetry.

This is a remote role available to candidates located in Australia or Singapore.

Responsibilities

• Design, build, operate, and maintain critical systems, owning their reliability, performance, and availability.
• Influence the product roadmap and define career objectives.
• Deliver new features, evaluate results, and iterate on solutions.
• Drive projects from initial ideation through production operations.
• Contribute to open-source projects beyond the immediate team when appropriate.
• Participate in on-call rotations and take ownership of the services operated by the team.
• Mentor and support team members, participate in design discussions, and collaborate effectively.
• Develop a deeper understanding of Grafana Cloud, its customers, and large distributed systems.
• Work independently and communicate effectively in a remote engineering organization.

Requirements

• Solid experience with at least one programming language. Grafana Labs primarily uses Go; experience with Python, C, C++, Rust, or similar languages is also relevant.
• Experience delivering projects from requirements gathering and ideation through product delivery.
• Experience developing cloud-based software or working in systems engineering.
• Experience writing clean, robust, performant, and maintainable software.
• A self-starting, action-oriented approach.
• Customer focus and an interest in building intuitive products that meet user needs.
• Pragmatic problem-solving skills, including the ability to break complex challenges into modular solutions and iterate using feedback.
• Strong collaboration and communication skills.

Bonus Qualifications

• Experience writing low-level code for databases, storage layers, or query engines.
• Experience with Kubernetes.
• Experience using Grafana or Prometheus in operational roles, on-call work, or hobby and homelab projects.
• Exposure to microservices architecture and distributed systems.
• Familiarity with on-call work, operations, SRE tasks, or infrastructure as code.

Benefits

• Base compensation in Singapore of SGD 167,000 to SGD 209,000 per year.
• Equity, bonus where applicable, and other benefits.
• 100% remote, global work culture.
• Company-funded usage budget for modern AI coding assistants.
• Access to frontier models from OpenAI, Anthropic, and Google.
• In-person onboarding.
• Global annual leave policy of 30 days per year, including three Grafana Shutdown Days where applicable.
• Career growth opportunities and a transparent, collaborative culture.