Staff Software Engineer - OpenTelemetry Python Instrumentation

Details

Grafana Labs is hiring a Staff Software Engineer to develop and improve the OpenTelemetry Python SDK and instrumentation libraries, with a strong focus on generative AI applications and agent workflows. This is a remote opportunity open to applicants in Germany, the United Kingdom, Ireland, and Spain.

Grafana Labs operates Grafana Cloud, an observability platform integrating metrics, logs, and traces with Grafana. The OpenTelemetry SDKs team works with OpenTelemetry Special Interest Groups and the wider open-source community to help users instrument their software successfully.

The role follows an upstream-first approach, contributing innovations to the OpenTelemetry project while bringing them to Grafana Labs products and offerings. Staff Engineers are expected to deliver work spanning multiple systems, provide technical direction, coach and mentor team members, and resolve technology and product-process issues.

Responsibilities

• Develop and maintain the OpenTelemetry Python SDK and instrumentation libraries.
• Collaborate with open-source communities, maintainers, and vendors on specifications, APIs, and implementation decisions.
• Build Python instrumentation for AI agents, connecting agent runs, model calls, and tool execution across asynchronous code.
• Contribute to evolving OpenTelemetry conventions for generative AI, with attention to sensitive data and metric cardinality.
• Improve Python SDK performance using profiling and benchmarks, while evaluating runtime overhead, compatibility, and maintainability.
• Promote best practices for OpenTelemetry, semantic conventions, and related technologies and frameworks.
• Support the technical direction and vision of the team and contribute to OpenTelemetry strategy discussions.
• Own projects from ambiguous problem definition through design, implementation, testing, documentation, and release.
• Partner with Go-to-Market, Product, and Support teams, participate in customer conversations, investigate instrumentation issues, and turn findings into fixes and practical guidance.
• Communicate effectively in a largely remote engineering organization.

Requirements

• At least 8 years of software development experience.
• Strong proficiency in Python.
• Experience debugging production software, language internals, libraries, and asynchronous code.
• Interest in AI applications and agent systems, including how to instrument and troubleshoot them.
• Experience working in open-source communities or willingness to become an active member of the OpenTelemetry community.
• Familiarity with observability tooling such as Grafana, Prometheus, and OpenTelemetry.
• Experience with cloud-based software or systems engineering and understanding of production environments and operational risks.
• Ability to take ownership of complex problems, set technical direction, and enable other engineers.

Bonus Qualifications

• Experience with open-source observability projects, ideally OpenTelemetry or Prometheus.
• Experience with Python instrumentation techniques, profiling, runtime diagnostics, or asynchronous context propagation.
• Experience in hands-on technical leadership, setting technical direction, and leading project teams.
• Experience instrumenting AI frameworks or agent workflows, or working with OpenTelemetry semantic conventions for generative AI.
• Contributions to or maintenance of open-source projects, including successful pull requests and community collaboration.
• Rust experience for exploring Python SDK performance improvements, including Rust-backed implementations.

Benefits

• Compensation in Germany of EUR 113,000–141,000 per year.
• Restricted Stock Units (RSUs).
• 100% remote, global work culture.
• Career growth pathways for individual contributors.
• Thirty days of annual leave per annum, including three Grafana Shutdown Days, subject to local legislation.
• In-person onboarding.
• Transparent communication, empowered teams, and an open-source-focused culture.