Lead Streaming Engineer

We are looking for a Lead Streaming Engineer to build the ingestion path for the analytics platform of a national-scale foundational identity programme in West Africa. You will own how data gets out of the identity system and into the lake — change data capture, the streaming buffer, stream processing, and the batch and log paths alongside them. This is the component that determines whether the entire platform can be trusted, and it is the one that touches the production identity environment.

The constraints are the interesting part. The source system cannot absorb query load, so capture is log-based. Delivery is at-least-once, so the pipeline has to deduplicate correctly or every recovery event inflates the national enrolment figures. Events arrive out of order and the source schema changes underneath you.

Personal identifiers must be pseudonymised inside the pipeline, before anything is written. And there is a historical migration of over one hundred million records to land alongside the live stream. If you have spent time explaining to people why their pipeline is silently losing or duplicating data, this role will be familiar.

Must-have criteria

● At least seven years in data engineering, of which three or more building production streaming or change data capture pipelines.

● Production experience of log-based change data capture — Debezium or equivalent — including schema change handling and connector operation.

● Apache Kafka in production: topic and partition design, retention, consumer groups, and the practical consequences of at-least-once delivery.

● Stream processing at scale, in Apache Spark Structured Streaming, Flink or equivalent.

● Has diagnosed and resolved a real duplication or ordering defect in production, and can describe both the diagnosis and the design change.

● Workflow orchestration — Airflow or equivalent.

Desirable

● Writing to an open table format, including compaction and file sizing.

● A large historical migration, in the order of one hundred million records or more.

● MOSIP schema familiarity.

● Ingestion from systems holding personal or sensitive data under a data protection regime.

● Java and Python.

● Kubernetes.

Expected hours: 40.0 – 45.0 per week

Application Question(s):

• Have you delivered software into a multi-party programme, where the organisation deploying it was not the one building it?
• Have you delivered production work in an on-premise or hybrid environment (not cloud-only)?

Experience:

• On-premises: 7 years (Required)

Work Location: In person