Data Streaming Engineer - Apache Flink & Spark

We are looking for an experienced Data Streaming Engineer with strong hands-on expertise in Apache Flink and Apache Spark to design, develop, and optimize large-scale real-time and batch data processing solutions.

The candidate should have strong experience in Flink DataStream API, Flink Table API/SQL, Spark Core, Spark SQL, and Spark Structured Streaming, with a solid understanding of distributed stream processing and high-volume data pipelines.

Mandatory Skills

• 8-10 years of experience in Data Engineering / Big Data / Streaming.
• Strong hands-on experience with Apache Flink.
• Expertise in Flink DataStream API.
• Strong knowledge of Flink Table API Flink SQL API.
• Experience with stateful stream processing, event-time processing, windowing, watermarks, and late-event handling.
• Strong understanding of Flink checkpointing, fault tolerance, and distributed deployment.
• Strong experience with Apache Spark.
• Hands-on expertise in Spark Core, Spark SQL, and Spark Structured Streaming.
• Experience developing high-volume ETL/data transformation pipelines.
• Strong understanding of Spark execution architecture, resource management, performance tuning, and optimization.
• Strong understanding of distributed systems and real-time data processing.
• Proficiency in at least one programming language: Java / Scala / Python.

Good to Have

• Apache Kafka / Azure Event Hubs or other messaging platforms.
• Azure / AWS / GCP cloud experience.
• Docker / Kubernetes.
• CI/CD and DevOps practices.
• Experience with Data Lakes / Lakehouse architectures.
• Microservices and event-driven architecture.
• Monitoring, logging, troubleshooting, and data quality.

Key Responsibilities

• Design and develop scalable real-time streaming pipelines using Flink and Spark.
• Build event-driven and near-real-time data processing solutions.
• Develop batch and streaming ETL workflows.
• Optimize pipelines for low latency, high throughput, reliability, and scalability.
• Implement Flink state management, checkpointing, watermarks, and fault tolerance.
• Tune Spark jobs and optimize resource utilization.
• Troubleshoot production data processing and streaming issues.
• Implement monitoring, logging, data quality, governance, and security standards.
• Collaborate with Data Engineers, Architects, and business stakeholders.

Disclaimer: This job posting has been aggregated from external source. Role details, content, and availability are subject to change. Applicants are advised to confirm the latest information directly on the company website before applying.