Azure Data Engineer

Role & responsibilities

Job Summary:
We are looking for an experienced Azure Data Engineer to design, develop, and maintain scalable data pipelines and data solutions using Microsoft Azure services. The candidate should have strong experience in data integration, ETL/ELT, cloud data platforms, SQL, and Azure data engineering technologies.

Key Responsibilities

• Design and develop scalable data pipelines and ETL/ELT workflows on Azure.
• Develop data integration solutions using Azure Data Factory (ADF).
• Work with Azure Databricks for data processing, transformation, and analytics.
• Develop complex SQL queries, stored procedures, views, and data transformations.
• Build and maintain data solutions using Azure Data Lake Storage (ADLS Gen2).
• Work with Azure Synapse Analytics for data warehousing and analytical workloads.
• Perform data cleansing, transformation, validation, and quality checks.
• Implement incremental data loads, CDC, partitioning, and performance optimization.
• Integrate data from various sources such as databases, APIs, files, and cloud applications.
• Monitor and troubleshoot data pipelines and resolve production issues.
• Implement security, access control, and governance using Azure services.
• Collaborate with Data Scientists, BI Developers, Architects, and business stakeholders.
• Implement CI/CD for data engineering workflows using Azure DevOps/Git.
• Optimize data pipelines and cloud resources for performance and cost.

Required Skills

• Strong experience in Azure Data Engineering.

• Hands-on experience with:

• Azure Data Factory (ADF)

• Azure Databricks

• Azure Data Lake Storage Gen2

• Azure Synapse Analytics

• Strong SQL skills.

• Good knowledge of Python/PySpark.

• Strong understanding of ETL/ELT concepts and data warehousing.

• Experience with data modeling, partitioning, and performance tuning.

• Knowledge of REST APIs and different data formats such as JSON, CSV, Parquet, and Avro.

• Experience with Git and Azure DevOps.

• Understanding of cloud security, IAM, and data governance.

Preferred candidate profile