Staff AI Engineer - Grafana AI/ML

📍 United States

USD 175,000-220,000 per year

SENIOR

✅ Remote

Tech Stack
Tag name is followed by "@" symbol and proficiency level value. About proficiency levels:

1-2 — basic awareness. Minimal hands-on experience, and a rudimentary understanding of the technology's purpose;

3-6 — daily use. Comfortable and regular usage, capable of handling common tasks and challenges related to the technology;

7-9 — you are an expert, you can teach others, you know all the pitfalls and tricks;

10 — exceptional knowledge, comprehensive understanding, and adeptness in all aspects of the technology, including advanced problem-solving. Think twice before claiming or demanding such level.

AI @ 4 AWS @ 4 Azure @ 4 DevOps @ 4 Docker @ 4 Experimentation GCP @ 4 GenAI Generative AI @ 4 Grafana Kubernetes @ 4 LLM Observability @ 4 Prompt Engineering @ 4 Terraform @ 4

Details

Grafana Labs is seeking an AI Software Engineer to develop, test, and ship AI-powered features that improve infrastructure and observability quality through automation. The role focuses on expanding AI agents across the observability stack to assist users with incident response. This is a highly autonomous position involving rapid experimentation, cross-functional collaboration, and ownership of scalable, maintainable solutions.

Responsibilities

Develop high-performance AI features that help users detect, triage, and resolve incidents using observability data and tools.
Prototype, test, validate, ship, and evolve LLM- and agent-powered workflows for incident lifecycle management and automated analysis.
Collaborate with data analysts, product managers, and designers to shape AI-driven product features.
Integrate agentic components with internal tools, alerting systems, runbooks, and developer workflows.
Use AI and automation tools to improve product functionality and development workflows.
Own AI solutions from development through delivery, ensuring they are scalable, maintainable, and aligned with real user workflows.
Communicate effectively and contribute across teams in a dynamic, collaborative environment.

Requirements

Strong experience building production backend and/or full-stack software systems.
Experience with LLMs, prompt engineering, and building generative AI applications.
A proven track record of delivering software that reached production and is actively used by customers or users.
Experience or exposure to cloud-native environments such as AWS, GCP, or Azure.
Experience using observability tools to understand and troubleshoot system behavior.
Ability to work independently, handle ambiguity, define scope, experiment quickly, and iterate based on user feedback.
Effective collaboration skills with engineers, product managers, designers, and other stakeholders.

Bonus Points

Experience with agent frameworks or multi-agent workflows.
Experience with infrastructure or DevOps tooling such as Kubernetes, Docker, Terraform, or similar deployment technologies.
Familiarity with model fine-tuning techniques.
Experience building observability tooling.

Benefits

Base compensation range in the United States of USD 174,986–220,000.
Equity through Restricted Stock Units (RSUs).
Bonus, if applicable, and other benefits.
100% remote work and a global culture.
Company-funded usage budget for modern AI coding assistants.
Access to frontier models from OpenAI, Anthropic, and Google.
In-person onboarding.
Global annual leave policy of 30 days per annum, including three Grafana Shutdown Days, subject to local legislation.