Kafka & Integration Architect

Purpose

To own the event streaming and integration architecture across the platform, establish the integration contract with the wider consortium, and deliver the change data capture pipeline that feeds the metadata, billing, management information and audit functions throughout the life of the platform.

Key responsibilities

• Define the Kafka platform architecture: cluster topology, Strimzi on Kubernetes in KRaft mode, broker sizing and storage, retention and quota policies, and the separation between development, staging and the shared production cluster.
• Establish and agree the consortium integration contract — covering topic naming and ownership conventions, namespace allocation, access control model, certificate issuance and rotation, and schema registry ownership.
• Deliver the change data capture pipeline using Debezium to move data from PostgreSQL sources to the metadata store, covering snapshot strategy, schema evolution, offset and history topic management, and failure recovery.
• Establish event schema governance: registry, versioning and compatibility policy, event catalogue, and the review gate to prevent incompatible schemas from reaching a shared cluster.
• Design cross-site streaming for disaster recovery between the primary and secondary sites, including consumer offset behaviour during failover.
• Define integration standards for Spring Boot subsystems — producer and consumer patterns, idempotency, dead-letter handling, and replay and backfill.
• Package all deliverables as parameterised Helm charts with externalised configuration for deployment by the integration team.
• Produce operational runbooks for cluster and connector operations, written for an operator who was not present during the build.
• Develop the nominated Integration Engineer to independent capability.

Essential experience

• Kafka operated at cluster level in production at scale — covering broker configuration, partition and replication strategy, access control, quota and retention design, and capacity planning. Application-level Kafka experiencealone is insufficient for this role.
• Strimzi on Kubernetes in production, including KRaft mode and mutual TLS for inter-broker and client authentication.
• Kafka Connect and Debezium in production, with direct experience of the snapshot strategy, schema evolution on a live source, and connector failure recovery.
• Schema registry governance across teams under separate management.
• Has negotiated and enforced an integration contract across organisational boundaries.

Desirable experience

• Cross-site mirroring for disaster recovery, including consumer offset behaviour on failover.
• PostgreSQL and MySQL familiarity sufficient to reason about change data capture at source and sink.
• National identity, payments or telecommunications core platforms.

Application Question(s):

• Have you delivered software into a multi-party programme, where the organisation deploying it was not the one building it?
• Have you delivered production work in an on-premise or hybrid environment (not cloud-only)?

Experience:

• On-premises: 7 years (Required)

Work Location: In person