Senior AI Engineer - Grafana AI/ML

📍 United States

USD 127,700-203,900 per year

SENIOR

✅ Remote

Tech Stack
Tag name is followed by "@" symbol and proficiency level value. About proficiency levels:

1-2 — basic awareness. Minimal hands-on experience, and a rudimentary understanding of the technology's purpose;

3-6 — daily use. Comfortable and regular usage, capable of handling common tasks and challenges related to the technology;

7-9 — you are an expert, you can teach others, you know all the pitfalls and tricks;

10 — exceptional knowledge, comprehensive understanding, and adeptness in all aspects of the technology, including advanced problem-solving. Think twice before claiming or demanding such level.

AI @ 4 AWS @ 4 Azure @ 4 Communication @ 7 DevOps @ 4 Docker @ 4 Experimentation GCP @ 4 GenAI Generative AI @ 4 Grafana Kubernetes @ 4 LLM Observability @ 4 Prompt Engineering @ 4 Security Terraform @ 4

Details

Grafana Labs is seeking a Senior AI Engineer to develop AI-driven features for observability, incident response, and infrastructure automation. The role involves building, testing, and scaling LLM- and agent-powered workflows that help users detect, triage, and resolve incidents using observability data and tools.

The team operates with a high degree of autonomy and ownership, emphasizing rapid experimentation, user feedback, collaboration, and shipping impactful, maintainable software. Engineers may use modern AI coding assistants and frontier models from OpenAI, Anthropic, and Google within security guidelines.

Responsibilities

Develop and deliver high-performance AI features for detecting, triaging, and resolving incidents.
Prototype, test, validate, ship, and iterate on LLM- and agent-powered workflows for incident lifecycle management and automated analysis.
Collaborate with data analysts, product managers, and designers to shape AI-driven product features.
Integrate agentic components with internal tools, alerting systems, runbooks, and developer workflows.
Use AI and automation tools to improve product functionality and development workflows.
Communicate effectively and contribute across cross-functional teams.
Own AI solutions through development, ensuring they are scalable, maintainable, innovative, and aligned with real user workflows.

Requirements

Strong experience building production software systems, including backend and/or full-stack systems.
Experience with LLMs, prompt engineering, and building applications powered by generative AI.
A proven track record of delivering software that reached production and is actively used by customers or users.
Familiarity with AI technologies and frameworks, with a practical focus on delivering real-world solutions.
Experience working in cloud-native environments such as AWS, GCP, or Azure.
Experience using observability tools to understand and troubleshoot system behavior.
Ability to work independently, handle complex engineering problems, define scope in ambiguous situations, and drive projects forward.
Strong communication and collaboration skills.

Bonus Qualifications

Experience with agent frameworks or multi-agent workflows.
Experience with infrastructure or DevOps tooling such as Kubernetes, Docker, Terraform, or similar deployment technologies.
Familiarity with model fine-tuning techniques.
Experience building observability tooling.

Benefits

Equity, bonus where applicable, and other company benefits.
Restricted Stock Units for all roles.
100% remote work environment.
Company-funded usage budget for AI coding assistants.
In-person onboarding.
Global annual leave policy of 30 days per year, including three Grafana Shutdown Days.
Career growth opportunities and a transparent, collaborative culture.