Azure Data Engineer

Roles and Responsibilities :

• Design, develop, test, deploy and maintain large-scale data pipelines using Azure Data Factory (ADF) to integrate various data sources into a centralized repository.
• Collaborate with cross-functional teams to gather requirements for data processing needs and design solutions that meet business objectives.
• Develop complex SQL queries to extract insights from large datasets stored in Azure Databricks or other relational databases.
• Troubleshoot issues related to ADF pipeline failures, data quality problems, and performance optimization.

Job Requirements :

• 6-8 years of experience in designing and developing data pipelines using Azure Data Factory (ADF).
• Strong understanding of PySpark programming language for working with big data technologies like Hadoop/Hive+.
• Experience with building scalable architectures on Microsoft Azure platform including storage solutions such as Blob Storage/Data Lake Store.