Senior Azure Data Engineer

Role Overview

We are seeking a highly skilled and experienced Senior Azure Data Engineer to join our dynamic data engineering team. The ideal candidate brings 5 to 7 years of hands-on professional experience building, managing, and optimizing scalable, enterprise-grade data pipelines within the Microsoft Azure ecosystem.

In this role, you will leverage deep expertise in SQL, Python, and PySpark alongside Azure Databricks to architect high-performing ETL/ELT pipelines, modernize legacy data assets, and deliver reliable data products for analytics and downstream consumption.

Immediate Joiners preferred

PF Service History is Mandatory

Key Responsibilities

Pipeline Architecture & Development

• ETL/ELT Design: Design, build, and deploy robust, scalable data ingestion and transformation pipelines using Azure Data Factory (ADF), Azure Databricks, and core Azure services.
• Core Development: Write, test, and maintain optimized, production-ready code using Python, PySpark, and SQLfor large-scale batch and complex data transformations.
• Platform Modernization: Modernize and refactor legacy data pipelines to improve data throughput, code maintainability, and overall reliability.

Performance & Optimization

• Performance Tuning: Optimize PySpark jobs, SQL queries, and Databricks clusters to drastically reduce execution runtime and cloud computing costs.
• Quality & Governance: Write unit/integration tests and establish data validation frameworks to guarantee end-to-end data integrity and precision.

Modern Azure & Fabric Ecosystem Integration

• Azure Integration: Integrate data solutions across the Azure ecosystem, including Azure Data Lake Storage Gen2 (ADLS Gen2), Azure Synapse Analytics, and Azure Key Vault.
• Microsoft Fabric: Apply working knowledge of Microsoft Fabric to help streamline and unify enterprise analytics, data engineering, and AI workloads.

Collaboration & DevOps

• CI/CD & DevOps: Implement modern software engineering principles, maintaining version control via Git (Azure DevOps / GitHub) and automated deployment pipelines.
• Cross-Functional Partnering: Partner closely with data architects, analysts, data scientists, and business stakeholders to translate complex business needs into technical specifications.

Required Qualifications

Experience

• 57 years of dedicated experience in Data Engineering, with a heavy focus on the Microsoft Azure data stack.

Mandatory Technical Skills

• SQL: Expert-level mastery of advanced SQL (window functions, query plan optimization, complex joins, indexing, and partitioning strategies).
• Python & PySpark: Advanced proficiency in Python and PySpark for big data processing, data manipulation, and parallel processing execution.
• Azure Databricks & ADF: Proven hands-on experience designing and orchestrating data pipelines using Azure Databricks (Spark SQL / PySpark) and Azure Data Factory.

Core Azure Infrastructure

• Strong practical experience working with Azure Data Lake Storage Gen2 (ADLS Gen2) and Azure Synapse Analytics (or Azure SQL DW).
• Hands-on knowledge or working experience with Microsoft Fabric architecture and unified analytics patterns.

Software Engineering & CI/CD

• Proficient with Git workflows, unit testing frameworks (e.g., pytest), and CI/CD automated deployment practices within Azure DevOps or GitHub.
• Demonstrated ability to profile, troubleshoot, and optimize resource-heavy Spark jobs and SQL queries.

Preferred / Bonus Qualifications

• Certifications: Microsoft Certified: Azure Data Engineer Associate (DP-203).
• Streaming Data: Experience handling real-time streaming data with Azure Event Hubs, Azure Stream Analytics, or Apache Kafka.
• Data Security & Governance: Familiarity with Microsoft Purview, fine-grained access control (RBAC), data masking, and security protocols in Azure.