Technical Consultant - Python & Kafka

Position Summary

Hands-on Senior Engineer responsible for enhancing an existing event-driven digital repository platform, with focus on Kafka reliability, Python services, structured logging, event lineage, reconciliation, and completion tracking. This is an implementation-focused role working within an established architecture.

Key Responsibilities

Develop and maintain Python services using Kafka, PostgreSQL, MongoDB, object storage, and REST APIs.Implement structured JSON logging, correlation, error context, and actionable diagnostics.Improve Kafka reliability including offsets, consumer groups, rebalancing, retries, DLQs, and message-loss investigation.Implement event tracking, lineage, reconciliation, and workflow completion verification.Build observability using OpenTelemetry and contribute to OpenLineage/Marquez implementation.Apply event-driven patterns including idempotency, delivery semantics, and failure recovery.Write automated tests and support deployments in Kubernetes, Kustomize, and ArgoCD.Troubleshoot distributed processing issues across services, messaging, databases, and external systems.

Must Have

Python Advanced production development; concurrency/threading; testing and maintainable code.Apache Kafka Producers/consumers, offsets, consumer groups, rebalancing, retries, DLQ, delivery semantics.PostgreSQL Strong SQL and JSON/JSONB capabilities; operational/event data.Structured Logging Observability JSON logging standards, correlation, severity, diagnostics, distributed-system observability.

Good to Have

OpenTelemetryOpenLineage / MarquezKubernetes, Kustomize ArgoCDGrafana / Grafana LokiEvent-driven architecture idempotency, at-least/exactly-once concepts, transaction boundaries, failure recovery.

Nice to Have

MongoDBAWS S3 / S3-compatible storageREST APIs and distributed data-processing platformsExperience with digital repository, media, or data-processing workflows.