Pyspark Engineer

Roles and Responsibilities

• Design, develop, test, deploy, and maintain large-scale data processing pipelines using PySpark on Cloudera.
• Collaborate with cross-functional teams to identify business requirements and design solutions that meet those needs.
• Develop complex SQL queries to extract insights from structured and unstructured data sources.
• Implement data quality checks and validation processes to ensure high-quality data ingestion.
• Troubleshoot issues related to data extraction, transformation, loading (ETL), and reporting.

Desired Candidate Profile

• 5-10 years of experience in IT industry with expertise in Big Data technologies such as Hadoop ecosystem (HDFS, YARN, MapReduce).
• Strong understanding of Informatica BDM Development tools for technical data mapping and extraction.
• Proficiency in Apache Oozie workflow management system for scheduling jobs across multiple nodes.
• Experience working with IFRS9 application development is an added advantage.