Big Data Engineer

• Utilize the Big Data Hadoop Ecosystem to efficiently process and analyze large datasets
• Develop and optimize data workflows leveraging Hadoop Spark and Spark SQL for largescale data analytics
• Develop Python applications and scripts to automate data processing and integration within the Hadoop ecosystem
• Collaborate with cross functional teams to understand data requirements and deliver optimized solutions aligned with business goals
• Ensure data quality integrity and governance across distributed systems
• Integrate various data ingestion and transformation tools such as HDFS Hive Sqoop Oozie and Kafka to support data processing needs
• Stay updated with emerging technologies and best practices in Hadoop Spark Python and big data frameworks
• Optimize system performance and troubleshoot issues related to big data infrastructure to ensure high availability and reliability

Roles and Responsibilities

• Design develop and deploy scalable big data solutions using Hadoop and Spark ecosystems
• Implement and optimize Spark SQL and PySpark jobs to enhance performance and resource utilization
• Monitor and maintain Hadoop clusters for stability capacity planning and performance tuning
• Collaborate with data engineers analysts and stakeholders to deliver actionable insights and high-quality data solutions
• Conduct code reviews provide technical guidance and mentor junior team members
• Participate actively in Agile ceremonies including sprint planning code reviews and release cycles
• Document technical specifications processes and best practices related to big data projects
• Ensure security and compliance standards are met within big data environments

Preferred candidate profile