Azure Data Engineer

Senior Azure Data Engineer

Experience Required: 7- 10+ years overall, with 2+ years hands-on in Microsoft Fabric or equivalent experience in Azure Synapse/Azure Databricks with strong Fabric readiness.

Role Summary

We are seeking a Senior Data Engineer with strong Microsoft Fabric expertise to design, build, and optimize scalable data pipelines and Lakehouse solutions on Azure. This role involves hands-on engineering across the Microsoft Fabric platform, including OneLake, Data Factory, Lakehouse, Warehouse, Notebooks, Spark, and Power BI Direct Lake, while collaborating with architects, analysts, and business stakeholders.

Key Responsibilities

• Design, build, and maintain end-to-end data pipelines in Microsoft Fabric using Data Factory pipelines/dataflows, Notebooks, and Spark jobs.
• Build and manage Lakehouse and Warehouse architectures on OneLake, implementing Bronze/Silver/Gold layers.
• Develop and optimize PySpark/Spark SQL transformations within Fabric Notebooks.
• Implement Delta Lake tables with proper partitioning, compaction, and schema evolution handling.
• Integrate real-time data using Fabric Real-Time Intelligence, Eventstream, and KQL Database.
• Build semantic models and enable Direct Lake mode in Power BI.
• Migrate existing workloads from Synapse Analytics / Azure Databricks / SSIS to Microsoft Fabric where applicable.
• Implement data quality, validation, and error-handling frameworks.
• Apply data governance and security controls using Fabric workspace roles, OneLake security, and Microsoft Purview integration.
• Optimize Fabric capacity utilization and cost.
• Write clean, modular, reusable code and maintain version control using Git integration with Fabric workspaces.
• Collaborate with data architects on solution design and participate in code reviews.
• Mentor junior engineers.
• Troubleshoot pipeline failures, performance bottlenecks, and data discrepancies in production.
• Document data flows, transformation logic, and pipeline dependencies.

Required Skills & Experience

• 7+ years in data engineering with strong hands-on experience in Microsoft Fabric, or equivalent depth in Synapse/Databricks with demonstrated Fabric upskilling.

• Strong working knowledge of:

• Fabric Lakehouse & Warehouse

• OneLake

• Fabric Data Factory pipelines and Dataflows Gen2

• Fabric Notebooks

• PySpark / Spark SQL

• Delta Lake

• Power BI, especially Direct Lake semantic modeling

• Strong SQL skills, including complex queries, stored procedures, and performance tuning.

• Strong Python/PySpark programming for distributed data processing.

• Experience with Fabric Real-Time Intelligence, Eventstream, KQL DB, or equivalent technologies such as Event Hubs/Stream Analytics.

• Understanding of dimensional modeling Star/Snowflake schema and Medallion Architecture.

• Familiarity with Git-based version control and CI/CD for Fabric deployment pipelines.

• Knowledge of Fabric security model workspace roles, item-level permissions, and OneLake data access.

• Prior experience with Azure Data Factory, Azure Synapse Analytics, or Azure Databricks.

Preferred Qualifications

• Microsoft Certified: DP-600 (Fabric Analytics Engineer Associate) or DP-203 (Azure Data Engineer Associate).
• Experience migrating workloads from Synapse/Databricks to Microsoft Fabric.
• Exposure to Microsoft Purview for data governance and lineage.
• Familiarity with Azure DevOps and Fabric Git integration.
• Experience with Semantic Link (SemPy) or Fabric data science/ML capabilities.
• Life Sciences domain experience is preferred.

Soft Skills

• Strong problem-solving skills and ability to independently troubleshoot complex pipeline issues.
• Good communication skills for collaboration with architects, analysts, and business teams.
• Detail-oriented approach to data quality and reliability.
• Ability to adapt quickly to Microsoft's evolving Fabric feature set.
• Strong fundamentals in Spark, SQL, data modeling, and distributed data processing.