Data Engineer

Role & responsibilities

• Design, develop, and maintain scalable data processing pipelines using Apache Spark.
• Develop batch and real-time data processing solutions using Spark Core and Structured Streaming.
• Work extensively with Spark SQL, DataFrames, and Dataset APIs.
• Develop Spark applications using the Java Spark API.
• Build reliable and high-performance ETL/ELT data pipelines.
• Perform Spark performance tuning and optimization for large-scale workloads.
• Troubleshoot Spark jobs related to performance, memory, serialization, partitioning, and resource utilization.
• Implement efficient solutions using concurrency and multithreading.
• Use Java Streams API for efficient data processing and transformation.
• Design and implement appropriate data models for analytical and operational use cases.
• Work with large datasets and distributed processing environments.
• Ensure data quality, reliability, scalability, and performance of data pipelines.
• Collaborate with data architects, software engineers, analysts, and other stakeholders.
• Monitor and troubleshoot production data pipelines and resolve performance or data-quality issues.
• Follow engineering best practices around code quality, testing, deployment, monitoring, and documentation.

Mandatory Skills

Apache Spark

Strong hands-on experience with:

• Apache Spark Core
• Spark SQL
• Spark Structured Streaming
• Spark DataFrames
• Spark Dataset API
• Java Spark API
• Spark Performance Tuning
• Concurrency & Multithreading
• Java Streams API