Senior Software Engineer - ClickPipes (CDC/Streaming)

Details

The ClickPipes - Database Integrations team builds the platform that enables real-time data replication from databases into ClickHouse at petabyte scale. The team works on database internals, distributed systems, schema evolution, data type compatibility, low-latency replication, and durable execution frameworks. Database integrations are built on and contributed to through the open-source PeerDB project.

Responsibilities

Build Data-Intensive Systems

• Design and develop high-throughput integrations with PostgreSQL, MySQL, MongoDB, data lakes such as Iceberg and Delta Lake, and data warehouses such as BigQuery and Snowflake.
• Handle real-world production edge cases, including unconventional database setups, data type internals, database upgrades and failovers, and large transactions.
• Design integration solutions that enable users to fully utilize ClickHouse's performance and throughput.

Own End-to-End Reliability

• Debug complex production issues using runtime diagnostics such as pprof and Parca, along with metrics, logging, and tracing.
• Build and improve infrastructure and tools to increase system reliability, reduce incident response time, and simplify or automate operations.
• Write clear public and internal documentation.
• Participate in an on-call rotation.

Drive Product Innovation

• Work directly with customers to understand integration requirements and identify gaps in the existing product.
• Collaborate cross-functionally with internal teams to ensure operational efficiency.
• Lead technical discussions and influence product roadmaps.

Requirements

• 5+ years of industry experience building data-intensive software solutions.
• Proficiency in Go, or systems programming experience with a willingness to ramp up quickly in Go.
• Cloud-native experience deploying and operating services on at least one major cloud platform: AWS, Google Cloud Platform, or Microsoft Azure.
• Practical experience with Kubernetes.
• Strong problem-solving and production debugging skills.
• Clear written communication, including design documents and code reviews, and strong verbal communication in technical discussions, customer calls, and incident response.

Bonus Qualifications

• Experience with database replication technologies, including change data capture and logical replication.
• Experience with durable execution frameworks such as Temporal.
• Experience with data formats and protocols such as Avro, Parquet, and Protobuf.
• Experience with modern data processing frameworks such as Kafka, Spark, and Flink.
• Experience maintaining or contributing to open-source software.

Benefits

• Flexible work environment at a globally distributed, remote-friendly company operating in more than 25 countries.
• Employer contributions toward healthcare.
• Stock options for every new team member.
• Flexible time off in the United States and generous leave entitlement in other countries.
• USD 500 home office setup benefit for remote employees.
• Opportunities to participate in company-wide offsites and global gatherings.

The role is remote in any country where ClickHouse has a hiring presence. ClickHouse provides equal employment opportunities and prohibits discrimination and harassment.