Staff Software Engineer - OpenTelemetry Python Instrumentation

📍 Germany
📍 Spain
📍 United Kingdom
📍 Ireland

EUR 121,000-151,000 per year

SENIOR

✅ Remote

Tech Stack
Tag name is followed by "@" symbol and proficiency level value. About proficiency levels:

1-2 — basic awareness. Minimal hands-on experience, and a rudimentary understanding of the technology's purpose;

3-6 — daily use. Comfortable and regular usage, capable of handling common tasks and challenges related to the technology;

7-9 — you are an expert, you can teach others, you know all the pitfalls and tricks;

10 — exceptional knowledge, comprehensive understanding, and adeptness in all aspects of the technology, including advanced problem-solving. Think twice before claiming or demanding such level.

AI @ 7 API Communication @ 7 Debugging @ 4 GenAI Generative AI @ 7 Go Grafana @ 7 Leadership @ 4 Observability @ 7 OpenTelemetry @ 7 Profiling @ 4 Prometheus @ 3 Python @ 7 Rust @ 4 Software Development @ 7 Technical Leadership @ 4

Details

Grafana Labs is seeking a Staff Software Engineer to develop and improve the OpenTelemetry Python SDK and instrumentation libraries, with a strong focus on generative AI applications and agent workflows. This is a hands-on, remote role requiring deep Python expertise, experience with production software and asynchronous code, and an interest in open-source observability.

Responsibilities

Collaborate with open-source communities to develop and maintain the OpenTelemetry Python SDK and instrumentation libraries.
Work with maintainers and vendors on specifications, APIs, and implementation decisions.
Build Python instrumentation for AI agents, connecting agent runs, model calls, and tool execution across asynchronous code.
Contribute to OpenTelemetry conventions for generative AI, considering sensitive data and metric cardinality.
Improve Python SDK performance using profiling and benchmarks, while evaluating runtime overhead, compatibility, and maintainability.
Act as an OpenTelemetry expert and promote best practices around OpenTelemetry, semantic conventions, technologies, and frameworks.
Support the technical direction and vision of the team and contribute to OpenTelemetry strategy discussions.
Own projects from ambiguous problem definition through design, implementation, testing, documentation, and release.
Partner with Go-to-Market, Product, and Support teams, join customer conversations, investigate instrumentation issues, and develop fixes and practical guidance.
Coach and mentor team members and help coordinate and deliver projects spanning multiple systems.

Requirements

8+ years of software development experience.
Strong proficiency in Python.
Experience debugging production software, language internals, libraries, and asynchronous code.
Interest in AI applications and agent systems, and curiosity about instrumenting and troubleshooting them.
Experience working in open-source communities and willingness to become an active member of the OpenTelemetry community.
Familiarity with observability tooling such as Grafana, Prometheus, and OpenTelemetry.
Experience with cloud-based software or systems engineering.
Understanding of production environments and operational risks.
High agency and strong communication skills for effective remote collaboration.

Bonus Qualifications

Experience with open-source observability projects, particularly OpenTelemetry or Prometheus.
Experience with Python instrumentation techniques, profiling, runtime diagnostics, or asynchronous context propagation.
Experience in hands-on technical leadership, setting technical direction, and leading project teams.
Experience instrumenting AI frameworks or agent workflows, or working with OpenTelemetry semantic conventions for generative AI.
Experience contributing to or maintaining open-source projects, with successful pull requests and community collaboration.
Rust experience for exploring Python SDK performance improvements, including Rust-backed implementations.

Benefits

Restricted Stock Units (RSUs).
100% remote, global culture.
Career growth pathways and a dedicated individual contributor career path.
In-person onboarding.
Global annual leave policy of 30 days per annum, including three Grafana Shutdown Days.
Transparent communication, empowered teams, and an open-source-focused culture.

Compensation

The compensation range in Ireland is EUR 121,000–151,000 per year. Actual compensation may vary based on level, experience, and skillset. Compensation ranges are country-specific, and recruiters will discuss the applicable market range and benefits for applicants applying from other locations.