Software Engineer - Synthetic Monitoring

Details

Grafana Labs is seeking a Software Engineer to join the Synthetic Monitoring squad. The team builds and operates Grafana Cloud Synthetic Monitoring, a multi-tenant distributed system used to proactively verify critical user journeys, APIs, and services.

Synthetic Monitoring is implemented in Go and runs on Kubernetes across AWS, GCP, and Azure. It executes billions of synthetic tests each month from probe locations worldwide and customer-operated private probes. The role is full-stack, covering backend services and agents, frontend workflows, execution pipelines, telemetry, and cross-product integrations.

Responsibilities

• Own features end-to-end across the stack, from API and data model design in Go through execution pipelines to user-facing TypeScript/React workflows.
• Design and evolve systems involving scheduling, probe lifecycle, telemetry ingestion, and API architecture.
• Build cross-product and AI-assisted workflows connecting Synthetic Monitoring to Grafana Cloud.
• Develop governance and organization capabilities for enterprise-scale Synthetic Monitoring.
• Break down complex and ambiguous problems into incremental deliverables and iterate based on feedback.
• Ensure quality through testing, system observability, and documentation.
• Participate in the team’s on-call rotation and take ownership of production services.
• Work collaboratively in a remote-first environment and communicate design decisions clearly in written and spoken English.
• Use AI tools for exploration, code review, design reasoning, prototyping, test generation, refactoring, documentation, and incident follow-ups while maintaining strong code review and quality standards.

Technologies

• Go for backend services, APIs, and the Synthetic Monitoring agent
• TypeScript and React for the Grafana Synthetic Monitoring application
• Kubernetes across AWS, GCP, and Azure
• k6 as the execution engine for scripted and browser checks
• Grafana ecosystem technologies including Mimir, Loki, and Prometheus

Requirements

• Strong Go experience or a path to it. Production backend experience in Rust, Java, or C++ may translate well; backend experience limited to JavaScript/Node.js runtimes is not sufficient.
• Experience owning and operating production services, including understanding on-call responsibilities and reliability practices.
• Systems-thinking skills, including reasoning about component interactions, data flow, failure modes, cost, and operational behavior.
• A pragmatic approach focused on short feedback loops, incremental delivery, prototyping, and independent work.
• Understanding of a user-centered software development process.
• Ability to work collaboratively, contribute to discussions, disagree constructively, and commit to team decisions.
• Comfortable working in a remote-first company.
• Thoughtful use of AI tools as part of an engineering workflow.

Bonus Experience

• Designing or operating multi-tenant SaaS systems at scale
• Observability tools such as Grafana, Datadog, New Relic, or Sentry
• Kubernetes in production
• k6, Playwright, or browser automation at scale
• Contributing to or maintaining open-source projects

Benefits

• Equity and bonus, if applicable
• 100% remote global culture
• In-person onboarding
• 30 days of annual leave per year, including three Grafana Shutdown Days where applicable
• Career growth pathways and a transparent, collaborative work environment