Azure Data Engineer

Job highlights

• Strong hands-on experience with Azure Data Factory and Azure Databricks, expertise in ETL/ELT pipeline design, proficiency in Python, PySpark, Scala, and Hive
• Design, develop, and maintain data pipelines and warehousing solutions using Azure services; optimize data processing; implement CI/CD pipelines; support cloud migration and modernization

About the Role

We are looking for an experienced and passionate Azure Data Engineer to join our team at Capgemini. You will work with leading global brands to design, implement, and optimize modern data solutions using Microsoft Azure cloud services.

Key Responsibilities

• Design, develop, and maintain end-to-end data ingestion, transformation, and processing pipelines using Azure Data Factory (ADF) and Azure Databricks (ADB).
• Develop scalable ETL/ELT solutions to integrate data from multiple sources into Azure data platforms.
• Build and optimize data processing frameworks using PySpark, Python, Scala, and Hive within Azure Databricks environments.
• Implement and manage enterprise-scale data warehousing solutions using Azure SQL Database, Azure SQL Data Warehouse, and Azure Synapse Analytics.
• Collaborate with business analysts, architects, and stakeholders to gather requirements and translate them into technical solutions.
• Ensure data quality, data governance, security, and compliance standards are maintained across all data assets.
• Perform performance tuning, query optimization, and troubleshooting of large-scale data processing workloads.
• Design, build, and support distributed data workflows and real-time/batch processing solutions.
• Develop and maintain CI/CD pipelines for data engineering applications using Azure DevOps and related tools.
• Support cloud migration initiatives and modernization of legacy data platforms to Azure.
• Create technical documentation, deployment procedures, and operational support guides.
• Participate in code reviews and follow industry best practices for software engineering and data architecture.

Required Skills & Experience

Primary Skills (Mandatory)

Candidates must have strong hands-on experience in at least one of the following:

• Azure Data Factory (ADF)
• Azure Databricks (ADB)

Core Technical Skills

• Strong experience with Azure Databricks and Azure Data Factory

• Expertise in designing ETL/ELT pipelines and data integration solutions

• Hands-on experience with:

• Azure SQL Database

• Azure SQL Data Warehouse

• Azure Synapse Analytics

• Proficiency in:

• Python

• PySpark

• Scala

• Hive

• Experience working with large-scale structured and unstructured datasets

• Strong understanding of data warehousing concepts and dimensional modeling

• Experience in CI/CD implementation and deployment automation in Azure environments

• Knowledge of Azure DevOps, Git, and version control practices

• Understanding of cloud-native architecture, security, and monitoring best practices

Preferred Skills

• Experience with Delta Lake and Spark optimization techniques
• Knowledge of Azure Data Lake Storage (ADLS Gen2)
• Exposure to real-time data processing and streaming solutions
• Azure certifications related to Data Engineering or Cloud Technologies
• Experience working in Agile/Scrum environments