Azure Data Engineer

Job Summary

We are looking for an experienced Azure Data Engineer to design, develop, and maintain scalable data pipelines and cloud-based data solutions using Microsoft Azure. The ideal candidate will have strong expertise in Azure Data Factory, Azure Databricks, PySpark, SQL, and data warehousing concepts.

Key Responsibilities

• Design, develop, and maintain scalable ETL/ELT data pipelines using Azure Data Factory.
• Build and optimize data processing workflows using Azure Databricks and PySpark.
• Develop complex SQL queries, stored procedures, and data transformation logic.
• Integrate data from multiple sources, including databases, APIs, files, and cloud storage.
• Implement data lake and lakehouse solutions using Azure Data Lake Storage Gen2.
• Work with Delta Lake and implement efficient data processing and storage strategies.
• Ensure data quality, accuracy, security, and reliability across data pipelines.
• Optimize data pipelines and SQL queries for performance and cost efficiency.
• Implement monitoring, logging, and error-handling mechanisms.
• Collaborate with data architects, analysts, and business stakeholders to deliver data solutions.
• Support CI/CD pipelines and deployment automation for data engineering workloads.
• Follow data governance, access control, and cloud security best practices.

Required Skills

• Strong experience with Microsoft Azure data services.
• Hands-on experience with Azure Data Factory (ADF).
• Experience with Azure Databricks and PySpark.
• Strong SQL skills and database concepts.
• Experience with Azure Data Lake Storage Gen2 (ADLS).
• Knowledge of ETL/ELT processes and data pipeline development.
• Experience with data modeling, data warehousing, and data transformation.
• Understanding of Delta Lake and lakehouse architecture.
• Experience with data pipeline monitoring, troubleshooting, and optimization.

Nice-to-Have Skills

• Experience with Azure Synapse Analytics or Microsoft Fabric.
• Knowledge of Python and Scala.
• Experience with Azure DevOps, Git, and CI/CD pipelines.
• Familiarity with Terraform or other infrastructure-as-code tools.
• Understanding of data governance, security, and compliance.
• Experience with streaming technologies such as Azure Event Hubs or Kafka.

Experience Requirements

• 10+ years of experience in data engineering.
• Strong hands-on experience developing and supporting Azure-based data solutions.
• Experience working on enterprise-scale data integration and transformation projects.