Azure Data Engineer

We are seeking an experienced Data Engineer (Azure) to design, develop, and maintain scalable cloud-based data engineering solutions. The ideal candidate will have strong hands-on expertise in Azure Synapse Analytics, Azure Data Factory, Azure Databricks, Azure Data Lake Storage Gen2, Python, PySpark, Spark SQL, and Power BI.

The role will be responsible for building robust ETL/ELT pipelines, implementing incremental and CDC-based data processing, optimizing distributed data workloads, developing enterprise data models, and supporting end-to-end data platform delivery across Development, QA, and Production environments.

The successful candidate should be comfortable owning technical solutions independently, collaborating with architects and business stakeholders, participating in Agile delivery, and mentoring other engineers.

Key Responsibilities

• Design, develop, and maintain scalable ETL/ELT pipelines using Azure Data Factory, Azure Databricks, Azure Synapse Analytics, Azure Storage, Azure Key Vault, Synapse SQL Pools, Azure Functions, Logic Apps and Azure Data Lake Storage Gen2.
• Develop data ingestion and transformation solutions using Python, PySpark, Spark SQL, and SQL.
• Build and optimize data pipelines for large-scale batch and distributed data processing workloads.
• Configure and manage Azure Data Factory linked services, datasets, integration runtimes, pipelines, triggers, parameters, and activities.
• Develop data solutions using Azure Synapse SQL pools, Synapse Spark pools, Databricks clusters, notebooks, Azure Functions, Logic Apps, and Azure Storage.
• Implement secure credential and secrets management using Azure Key Vault.
• Develop incremental data-loading frameworks using Delta Lake, MERGE operations, upserts, deduplication, and Change Data Capture (CDC).
• Design reusable and metadata-driven pipeline frameworks to support multiple data sources and business domains.
• Develop Python-based database integrations with SQL Server, Oracle, and MySQL, including executing stored procedures and processing returned datasets.
• Design and implement data warehouse and lakehouse solutions using dimensional modeling, Star Schema, Snowflake Schema, and Slowly Changing Dimensions (SCD).
• Implement data-quality validation, exception handling, logging, monitoring, alerting, and operational support mechanisms.
• Optimize Spark jobs, SQL queries, pipeline execution, cluster configurations, and cloud resources for performance and cost efficiency.
• Develop and maintain Power BI semantic models, dashboards, and reports, including Power Query and DAX-based solutions.
• Implement CI/CD processes for promoting data engineering code across Development, QA, and Production environments using GitHub or Azure DevOps.
• Follow source-control, branching, pull-request, code-review, testing, and deployment best practices.
• Support data governance and security using Databricks Unity Catalog, role-based access control (RBAC), and appropriate data-access policies.
• Participate in Agile/Scrum ceremonies including sprint planning, backlog refinement, daily stand-ups, demos, and retrospectives.
• Conduct code reviews, mentor junior and mid-level engineers, and contribute to engineering standards and best practices.
• Collaborate with data architects, business stakeholders, BI developers, and other engineering teams to translate requirements into scalable technical solutions.
• Independently own technical deliverables from design and development through testing, deployment, and production support.

Required Skills & Qualifications

• 4+ years of hands-on experience in Data Engineering, with strong expertise in:

• Azure Synapse Analytics

• Azure Data Factory

• Azure Databricks

• Azure Data Lake Storage Gen2

• Azure Storage

• Strong experience designing and maintaining scalable ETL/ELT pipelines.

• Strong proficiency in Python, PySpark, Spark SQL, and SQL.

• Experience working with Pandas, NumPy, and Python database connectivity libraries.

• Hands-on experience connecting Python applications to SQL Server, Oracle, and MySQL.

• Experience executing stored procedures through Python and processing returned datasets.

• Strong understanding of Azure Data Factory components, including:

• Linked Services

• Integration Runtimes

• Datasets

• Pipelines

• Activities

• Triggers

• Parameters

• Experience working with Azure Databricks clusters, notebooks, Spark workloads, and distributed data processing.

• Experience with Synapse Spark pools and SQL pools.

• Hands-on experience implementing incremental loads, Delta Lake processing, MERGE/upsert operations, deduplication, and CDC.

• Strong experience with GitHub or Azure DevOps and CI/CD-based code promotion across Dev, QA, and Production.

• Experience developing Power BI semantic models, dashboards, and reports.

• Strong knowledge of Power Query and DAX.

• Experience implementing data-quality checks, logging, monitoring, exception handling, and pipeline alerting.

• Experience securely managing credentials and secrets using Azure Key Vault.

• Strong analytical and problem-solving skills.

• Excellent communication and collaboration skills.

• Ability to independently own technical delivery and troubleshoot production issues.

Preferred Qualifications

• Experience designing end-to-end Azure data platforms using enterprise data warehouse and Medallion Lakehouse architectures.

• Strong knowledge of dimensional modeling, including:

• Star Schema

• Snowflake Schema

• Fact and Dimension Tables

• Slowly Changing Dimensions (SCD)

• Experience developing metadata-driven and reusable pipeline frameworks.

• Experience optimizing:

• Spark workloads

• SQL queries

• Data pipelines

• Cluster configurations

• Cloud resource utilization and costs

• Knowledge of Databricks Unity Catalog, data governance, data lineage, and role-based access control.

• Experience with data security and governance best practices in Azure environments.

• Experience conducting code reviews and establishing engineering standards.

• Experience mentoring junior and mid-level data engineers.

• Experience working in Agile/Scrum development environments.

Technical Environment

Cloud & Azure: Azure Data Factory, Azure Databricks, Azure Synapse Analytics, ADLS Gen2, Azure Storage, Azure Functions, Logic Apps, Azure Key Vault

Programming & Data Processing: Python, PySpark, Spark SQL, SQL, Pandas, NumPy

Databases: SQL Server, Oracle, MySQL, Azure SQL, Synapse SQL

Data Engineering: ETL/ELT, Delta Lake, CDC, Incremental Loads, MERGE/Upserts, Deduplication, Data Quality

Data Modeling: Dimensional Modeling, Star Schema, Snowflake Schema, SCD, Data Warehouse, Medallion Lakehouse