Apache Flink Transformation Engineer

Key Responsibilities

• Design, develop, and maintain high-performance real-time data processing applications using Apache Flink.
• Develop in-flight transformation logic leveraging Apache Flink, Kafka Streams, KSQLDB, and Kafka Single Message Transforms (SMTs).
• Build and maintain scalable mapping frameworks for transforming source system data into harmonized enterprise data models.
• Implement complex event processing, enrichment, filtering, aggregation, and windowing logic for streaming workloads.
• Optimize streaming pipeline performance, ensuring low latency, high throughput, and efficient resource utilization.
• Design solutions to handle late-arriving and out-of-order events using event-time processing, watermarks, checkpointing, and state management.
• Integrate streaming applications with Kafka, Schema Registry, APIs, and downstream analytical platforms.
• Implement robust error handling, retry mechanisms, dead-letter queues (DLQ), and monitoring for streaming applications.
• Collaborate with Solution Architects, Data Architects, API teams, and business stakeholders to define transformation rules and streaming integration patterns.
• Develop automated testing frameworks and CI/CD pipelines for streaming applications.
• Monitor production streaming jobs and troubleshoot performance, scalability, and reliability issues.

Required Skills & Experience

• 610+ years of experience in Data Engineering with strong expertise in real-time streaming solutions.
• Hands-on experience with Apache Flink for enterprise-scale stream processing.
• Strong knowledge of Apache Kafka, Kafka Streams, KSQLDB, Kafka Connect, and Single Message Transforms (SMTs).
• Experience designing event-driven data architectures and real-time data integration pipelines.
• Strong understanding of event-time processing, watermarks, windowing, checkpointing, stateful stream processing, and fault tolerance.
• Experience developing data transformation and harmonization frameworks for enterprise data platforms.
• Proficiency in Java and/or Scala; Python knowledge is desirable.
• Experience with Avro, Protobuf, JSON, Schema Registry, and serialization techniques.
• Strong SQL skills and experience working with large-scale structured and semi-structured datasets.
• Experience with cloud platforms such as Azure, AWS, or GCP is preferred.
• Familiarity with CI/CD, Git, Docker, Kubernetes, and monitoring tools is an advantage.

Preferred Qualifications

• Experience with Confluent Platform and enterprise Kafka deployments.
• Exposure to modern data lakehouse platforms such as Databricks, Snowflake, or Delta Lake.
• Knowledge of CDC patterns, API integration, and enterprise integration architectures.
• Experience working in Agile/Scrum delivery environments.
• Excellent analytical, problem-solving, and stakeholder communication skills.