Kafka Data Engineer

We are looking for an experienced Kafka Data Engineer with strong hands-on expertise in Apache Kafka, Kafka Topic/Cluster Migration, Data Streaming, and Distributed Systems. The candidate will be responsible for planning and executing Kafka migrations, cluster-to-cluster replication, data validation, performance tuning, and ensuring minimal downtime during migration activities.

Key Responsibilities

• Design and execute Kafka topic and cluster migrations across environments, clusters, or cloud platforms.
• Perform Kafka cluster upgrades, broker migrations, and data replication using MirrorMaker 2.0, Confluent Replicator, Cluster Linking, or similar technologies.
• Migrate and validate topics, schemas, consumer groups, offsets, ACLs, and configurations.
• Monitor Kafka throughput, latency, consumer lag, partition health, and overall cluster performance.
• Troubleshoot Kafka connectivity, replication, partition balancing, and consumer lag issues.
• Develop automation scripts using Python, Shell scripting, Java, or Scala.
• Work with application teams to validate producers, consumers, and streaming pipelines after migration.
• Implement Kafka security using SSL/TLS, SASL, Kerberos, RBAC, and ACLs.
• Collaborate with DevOps, Cloud, and Data Engineering teams for infrastructure and deployment activities.
• Prepare migration plans, rollback strategies, technical documentation, and post-migration reports.

Mandatory Skills

• Strong hands-on experience with Apache Kafka Administration and Development.
• Proven experience in Kafka Topic Migration / Cluster Migration / Cross-Cluster Replication.
• Hands-on experience with MirrorMaker 2.0, Confluent Replicator, or Cluster Linking.
• Strong knowledge of Kafka Connect, Kafka Streams, Schema Registry, and Confluent Platform.
• Good understanding of Kafka partitions, brokers, replication, consumer groups, offsets, and consumer lag.
• Experience working in Linux/Unix environments.
• Proficiency in Python, Java, or Scala.
• Strong understanding of distributed systems, event-driven architecture, and messaging systems.
• Experience with at least one cloud platform such as AWS MSK, Azure Event Hubs/Kafka, or GCP Pub/Sub.

Good to Have

• Experience with Databricks, Spark Streaming, Apache Flink, or other real-time processing technologies.
• Knowledge of Docker and Kubernetes.
• Exposure to Terraform / Infrastructure as Code and CI/CD pipelines.
• Experience with monitoring tools such as Prometheus, Grafana, Splunk, or ELK.
• Kafka or Confluent certification is an added advantage.

Key Skills

Kafka | Kafka Migration | MirrorMaker 2.0 | Kafka Administration | Confluent Platform | Kafka Connect | Cluster Linking | Kafka Replication | Python/Java | Distributed Systems

Disclaimer: This job posting has been aggregated from external source. Role details, content, and availability are subject to change. Applicants are advised to confirm the latest information directly on the company website before applying.