Big Data Engineer Java, Apache Spark, AWS EMR, Lambda, EKS & Airflow

Job Summary

Synechron is seeking a Big Data Engineer with 5+ years of experience in designing, developing and deploying scalable and reliable data-processing solutions on AWS using Java and Apache Spark. The role will focus on implementing big data processing pipelines using Apache Spark on AWS EMR, developing and deploying serverless applications using AWS Lambda, utilizing Amazon EKS for container orchestration and microservices management, and designing workflow orchestration using Apache Airflow. The position will contribute to business objectives by delivering reliable, scalable and maintainable data-processing solutions that support efficient data operations and application services.

Software Requirements

Required

• Java: Strong hands-on experience in Java development.
• Apache Spark: Proficiency in Apache Spark for distributed data processing.
• AWS EMR: Experience implementing big data processing pipelines using Apache Spark on AWS EMR.
• AWS Lambda: Experience developing and deploying serverless applications using AWS Lambda.
• AWS Service Integration: Experience integrating AWS Lambda with other AWS services.
• Amazon EKS: Experience utilizing Amazon EKS for container orchestration and microservices management.
• Apache Airflow: Experience designing and implementing workflow orchestration for complex data pipelines.
• AWS: Experience designing, developing and deploying scalable and reliable data-processing solutions on AWS using Java and Spark.
• Project-Supported Versions: Ability to work with the versions of Java, Apache Spark, AWS services and related tools supported by the project.

Overall Responsibilities

• Design, develop and deploy scalable and reliable data-processing solutions on AWS using Java and Apache Spark.
• Implement and maintain big data processing pipelines using Apache Spark on AWS EMR.
• Develop and deploy serverless applications using AWS Lambda and integrate them with other AWS services.
• Utilize Amazon EKS for container orchestration and microservices management.
• Design, implement and maintain Apache Airflow workflows for complex data pipelines.
• Develop maintainable Java applications and data-processing components aligned with technical requirements.
• Contribute to solution design, technical discussions and implementation decisions within agreed project standards.
• Test, troubleshoot and resolve application, pipeline, integration and deployment issues.
• Support reliable operation of data-processing solutions across relevant development, testing and production environments.
• Monitor application and pipeline behavior and contribute to improvements in scalability, reliability and maintainability.
• Collaborate with engineering, architecture, QA and delivery teams to clarify requirements and coordinate implementation activities.
• Contribute to code reviews, technical documentation, release activities and knowledge sharing.
• Apply appropriate data protection and cloud security practices during development and deployment.
• Use AWS resources efficiently and consider sustainable practices that reduce unnecessary compute, storage and network consumption.
• Deliver assigned work in accordance with agreed priorities, timelines and quality expectations.