Senior Data Engineer Azure and Microsoft Fabric

Role & responsibilities

Senior Data Engineer Azure & Microsoft Fabric

Position Overview

We are looking for a Senior Data Engineer with 6–8 years of experience in designing and developing scalable data engineering solutions using Azure Data Factory (ADF), Microsoft Fabric, PySpark, SQL, and modern cloud data platforms.

The ideal candidate should have strong hands-on experience in ETL/ELT development, data modeling, SQL Stored Procedures, Azure Data Lake Storage, and data warehouse concepts. Experience with Microsoft Fabric, Fabric Notebooks, PySpark, and AI-assisted development tools such as Claude/Claude Code or GitHub Copilot will be an added advantage.

Experience

6–8 Years

Primary Skills

• Azure Data Factory (ADF)
• Microsoft Fabric / Fabric Data Factory
• Fabric Notebooks
• PySpark
• SQL / T-SQL
• SQL Stored Procedures
• Data Modeling
• Azure Data Lake Storage Gen2 (ADLS Gen2)
• Data Warehouse / Lakehouse
• Azure DevOps / Git
• AI-assisted development tools – Claude, Claude Code, GitHub Copilot

Key Responsibilities

Data Engineering & Platform Development

• Design, develop, and maintain scalable ETL/ELT pipelines using Azure Data Factory and Microsoft Fabric Data Factory.
• Develop data engineering solutions using Fabric Notebooks, PySpark, and SQL.
• Analyze, transform, integrate, and process data from multiple sources.
• Develop and maintain Lakehouse, Data Warehouse, and Medallion Architecture implementations.
• Develop and optimize complex SQL queries, T-SQL, and Stored Procedures.
• Design and implement effective data models, including Fact/Dimension structures and Star Schema.
• Work with Azure Data Lake Storage Gen2 (ADLS Gen2) and other Azure data services.
• Ensure data quality, consistency, governance, security, validation, reconciliation, and exception handling.
• Contribute to scalable and reusable data engineering frameworks and solutions.

Azure & Cloud Data Engineering

• Work extensively with Azure Cloud architecture and data services.
• Collaborate with Solution Architects and technical stakeholders to design scalable and reliable cloud data solutions.
• Troubleshoot data pipelines, integration issues, and production incidents.
• Participate in technical design discussions, code reviews, and production support activities.

AI-Assisted Engineering

• Leverage AI-assisted development tools such as Claude, Claude Code, GitHub Copilot, and Microsoft Fabric AI capabilities to improve development productivity.
• Apply AI-assisted techniques for coding, debugging, documentation, unit testing, optimization, and troubleshooting.
• Identify opportunities to use Generative AI to automate repetitive data engineering and development activities.
• Follow organizational guidelines and best practices when using AI-assisted development tools.

Collaboration & Delivery

• Work closely with Product Owners, Business SMEs, Data Analysts, Data Scientists, BI Developers, QA teams, and Architects to understand requirements and deliver data solutions.
• Participate in Agile ceremonies, technical discussions, design reviews, and code reviews.
• Support data consumers and stakeholders with technical troubleshooting and data-related requirements.
• Contribute to coding standards, engineering best practices, documentation, and reusable components.

DevOps & Automation

• Implement CI/CD pipelines using Azure DevOps and/or GitHub.
• Follow Git-based version control and branching best practices.
• Automate deployment, testing, monitoring, and operational processes.
• Follow modern DataOps and engineering practices for continuous and reliable delivery.

Required Skills

• 6–8 years of hands-on experience in Data Engineering.
• Strong hands-on experience with Azure Data Factory (ADF).
• Strong knowledge of Azure SQL Database / SQL Server, T-SQL, and Stored Procedures.
• Strong understanding of data modeling and data warehouse concepts.
• Experience with Azure Data Lake Storage Gen2 (ADLS Gen2).
• Hands-on experience with ETL/ELT pipeline development and data integration.
• Good knowledge of Python.
• Working knowledge of PySpark and Fabric Notebooks.
• Good understanding of Microsoft Fabric, including Lakehouse, Warehouse, and Data Factory.
• Experience with Azure DevOps and Git.
• Strong analytical, problem-solving, troubleshooting, and communication skills.

Good to Have

• Microsoft Fabric – Data Factory, Lakehouse, Warehouse, and Notebooks.
• Advanced PySpark experience.
• Azure Logic Apps.
• Power BI Semantic Models.
• Experience with Medallion Architecture.
• Experience with DataOps practices.
• Experience using Claude / Claude Code or other AI-assisted development tools.
• Knowledge of Generative AI applications in data engineering.

Key Technology Stack

Azure Data Factory | Microsoft Fabric | Fabric Notebooks | PySpark | SQL | T-SQL | Stored Procedures | ADLS Gen2 | Data Modeling | Lakehouse | Data Warehouse | Azure DevOps | Git | Python | Claude | Claude Code | GitHub Copilot