Kafka Migration Engineer

Role Overview

We are looking for a highly skilled Kafka Migration Engineer to join the ETO Factory team and support a high-visibility on-premises Apache Kafka to AWS MSK migration project .

The candidate will be responsible for Kafka cluster discovery, workload assessment, capacity planning, AWS MSK configuration, migration execution, testing, validation, performance tuning, automation, and legacy Kafka cluster decommissioning .

The ideal candidate should have strong hands-on experience with Apache Kafka, AWS MSK, distributed messaging systems, cloud migration, Linux, programming, automation, and CI/CD , along with the ability to collaborate with application, infrastructure, and platform teams.

Key Responsibilities

• Assess on-premises Kafka clusters and perform discovery across:
• Topics
• Partitions
• Consumer groups
• Throughput patterns
• Retention policies
• Inter-service dependencies
• Evaluate workloads based on:
• Message volume
• Business criticality
• Ordering requirements
• Latency requirements
• Consumer group complexity
• Determine appropriate AWS MSK cluster placement, sizing, and configuration .
• Plan and provision AWS MSK infrastructure, including:
• Broker sizing
• Storage
• Networking
• Security
• Execute UAT and production parallel testing using MirrorMaker 2 or similar replication tools.
• Validate message delivery outcomes and troubleshoot migration discrepancies.
• Perform Kafka/MSK performance tuning and optimization to ensure scalability and production readiness.
• Collaborate with application, infrastructure, and platform teams to finalize:
• Migration plans
• Producer/consumer cutover strategies
• Validation criteria
• Rollback plans
• Execute production cutover activities, including:
• Consumer group migration
• Offset synchronization
• Post-release validation
• Support successful decommissioning of legacy on-premises Kafka clusters .
• Design and implement automation frameworks for repeatable, large-scale Kafka-to-MSK migrations.
• Develop scripts and utilities for:
• Topic creation
• ACL migration
• Consumer group management
• Migration validation
• Explore AI-assisted engineering and automation approaches to improve migration efficiency and quality.
• Document migration procedures, risks, operational learnings, and best practices.

Experience

• 3–5 years of hands-on engineering experience in distributed messaging systems.
• Strong experience working in collaborative, team-based engineering environments.

Programming

Strong proficiency in at least one of:

• Python
• Java
• Shell scripting
• Similar programming/scripting languages

Systems & DevOps

• Strong Unix/Linux fundamentals.
• Experience troubleshooting application, deployment, environment, and runtime issues.
• Understanding of:
• SDLC
• CI/CD
• Change Management
• Kubernetes-based deployments