We're rebuilding our real-time analytics pipeline from the ground up and looking for a senior data engineer to architect and own the implementation. Technical challenge: Computing metrics from complex event sequences across long-lived entities (spanning days/weeks). Currently running MongoDB → Kafka → ClickHouse but hitting some limitations with the current setup.
You'd be designing a stateful stream processing architecture to handle: - Event succession patterns and sequence analysis
- Out-of-order and late-arriving data
- Retroactive updates and backfilling
- Metric versioning and controlled rollout
- Reliability and accuracy guarantees
Looking for someone who's built production stream processing pipelines and dealt with similar challenges at scale. You'll be working closely with our CTO and Director of Engineering with full ownership of technical decisions and setting up the foundation as we build out our data team.