Data Engineer

Job Summary

Key Responsibilities

• Design, develop, and maintain scalable ETL/ELT pipelines using Azure Data Factory (ADF).

• Build and optimize complex data transformation logic using Azure Databricks (PySpark and SQL).

• Lead and execute on premises to Azure cloud migrations for data platforms, ETL workflows, and applications.

• Develop, deploy, and support applications using Azure App Service and Azure Functions.

• Design data models and perform query optimization and performance tuning using Azure SQL.

• Implement and manage data storage solutions using Azure Data Lake Storage Gen2 (ADLS Gen2) and Delta Lake.

• Apply robust data orchestration, scheduling, and error-handling patterns to ensure reliable data processing.

• Leverage core Azure services such as VNets, Storage Accounts, Key Vault, and IAM to build secure and scalable cloud solutions.

• Monitor, troubleshoot, and optimize data pipelines and Azure resources using appropriate monitoring tools.

Required Skills & Qualifications

• Strong hands-on experience with Azure Data Factory (ADF).

• Proven expertise in Azure Databricks, including PySpark and SQL-based data transformations.

• Hands-on experience in developing and deploying solutions using Azure App Service and Azure Functions.

• Solid experience in on premises to Azure migration projects, particularly for data platforms.

• Advanced SQL skills, including data modeling, query optimization, and performance tuning in Azure SQL.

• Working knowledge of Delta Lake, ADLS Gen2, and modern data lake architectures.

• Good understanding of core Azure services and cloud security best practices.

• Experience with monitoring and troubleshooting using Azure Monitor and Application Insights.