Staff Software Engineer - OpenTelemetry Python Instrumentation

Details

Grafana Labs is looking for a Staff Software Engineer to develop and improve the OpenTelemetry Python SDK and instrumentation libraries, with a strong focus on generative AI applications and agent workflows. This is a remote, hands-on role requiring deep Python expertise, software engineering experience, and an interest in AI and observability.

Responsibilities

• Develop and maintain the OpenTelemetry Python SDK and instrumentation libraries in collaboration with open-source maintainers and other vendors.
• Build Python instrumentation for AI agents, including agent runs, model calls, and tool execution across asynchronous code.
• Contribute to OpenTelemetry conventions for generative AI, considering sensitive data and metric cardinality.
• Improve Python SDK performance using profiling and benchmarks, evaluating runtime overhead, compatibility, and maintainability.
• Promote best practices around OpenTelemetry, semantic conventions, and observability technologies.
• Support the technical direction and vision of the team.
• Own projects from ambiguous problem definition through design, implementation, testing, documentation, and release.
• Collaborate with Go-To-Market, Product, and Support teams, join customer conversations, investigate instrumentation issues, and turn findings into fixes and practical guidance.
• Contribute to OpenTelemetry and other open-source communities.
• Coach and mentor team members and help coordinate large, cross-system initiatives.

Requirements

• 8+ years of software development experience.
• Strong proficiency in Python.
• Experience debugging production software, language internals, libraries, and asynchronous code.
• Interest in AI applications and agent systems, and curiosity about instrumenting and troubleshooting them.
• Experience working in open-source communities and willingness to become an active member of the OpenTelemetry community.
• Familiarity with observability tooling such as Grafana, Prometheus, and OpenTelemetry.
• Experience with cloud-based software or systems engineering and understanding of production environments and operational risks.
• Strong ownership, communication, planning, technical direction, and mentoring skills.

Bonus Qualifications

• Experience with open-source observability projects, particularly OpenTelemetry or Prometheus.
• Experience with Python instrumentation techniques, profiling, runtime diagnostics, or asynchronous context propagation.
• Experience in hands-on technical leadership, setting technical direction, or leading project teams.
• Experience instrumenting AI frameworks or agent workflows, or working with OpenTelemetry semantic conventions for generative AI.
• Experience contributing to or maintaining open-source projects.
• Rust experience for exploring Python SDK performance improvements, including Rust-backed implementations.

Compensation and Benefits

• UK compensation range: GBP 107,000–GBP 134,000 per year.
• Restricted Stock Units (RSUs).
• 100% remote work and a global culture.
• Career growth pathways and an individual-contributor career path.
• In-person onboarding.
• Global annual leave policy of 30 days per year, including three Grafana Shutdown Days.

The role is remote, and applications are welcomed from Germany, the United Kingdom, Ireland, and Spain.