Staff AI Engineer - Grafana AI/ML

Details

Grafana Labs is seeking an AI Software Engineer with a strong software engineering background, a quick iteration mindset, and a passion for experimentation. You will develop, test, and ship AI-powered features that improve infrastructure and observability through automation and expand AI agent capabilities across the observability stack to assist users with incident response. This is a remote opportunity for applicants in Canada time zones only.

Responsibilities

• Build and deliver high-performance AI features that help users detect, triage, and resolve incidents using observability data and tools.
• Rapidly prototype, test, and validate solutions with real users, including LLM- and agent-powered workflows for incident lifecycle management and automated analysis tasks.
• Collaborate with data analysts, product managers, and designers to shape AI-driven product features.
• Integrate agentic components with internal tools, alerting systems, runbooks, and developer workflows.
• Use AI and automation tools to enhance product functionality and development workflows.
• Communicate effectively and contribute across teams in a dynamic, collaborative environment.
• Take ownership of AI solutions, ensuring they are innovative, scalable, maintainable, and aligned with real user workflows.

Grafana Labs supports the use of modern AI coding assistants within security guidelines, including access to frontier models from OpenAI, Anthropic, and Google. AI-assisted development may be used for prototyping, test generation, refactoring, documentation, and incident follow-ups with code review and quality standards.

Requirements

• Strong experience building production software systems, including backend and/or full-stack systems.
• Familiarity with AI technologies and frameworks, with a practical focus on delivering real-world solutions.
• Experience with LLMs, prompt engineering, and building applications powered by generative AI.
• Proven track record of delivering software that reached production and is actively used by customers or users.
• Experience or exposure to cloud-native environments such as AWS, GCP, or Azure.
• Experience using observability tools to understand and troubleshoot system behavior.
• Ability to work independently, tackle complex engineering problems, define scope in ambiguous situations, and drive projects forward.
• Effective collaboration and communication skills when working with engineers, product managers, designers, and other stakeholders.

Bonus Points

• Experience with agent frameworks or multi-agent workflows.
• Experience with infrastructure or DevOps tooling such as Kubernetes, Docker, Terraform, or similar deployment tools.
• Familiarity with model fine-tuning techniques.
• Experience building observability tooling.

Benefits

• 100% remote work in a global culture.
• Restricted Stock Units (RSUs) for all roles.
• Company-funded usage budget for AI coding assistants.
• Career growth pathways.
• In-person onboarding.
• Global annual leave policy of 30 days per annum, including three Grafana Shutdown Days, subject to local legislation.
• Transparent communication, empowered teams, and an open-source-driven culture.

Grafana Labs is an equal opportunities employer. The company may utilize AI tools in its recruitment process to assist in matching information provided in CVs to job postings, while the recruitment team continues to manually review applications.