Senior Data Engineer, Spark/ Scala, (Microsoft Fabric Migration)

Job Description: Senior Data Engineer (Databricks to Microsoft Fabric Migration)

Job Summary We are seeking an experienced Senior Data Engineer to support a strategic migration of data processing workloads from Databricks to Microsoft Fabric. The migration scope is focused on compute modernization, while enterprise data will continue to reside in Azure Data Lake Storage (ADLS). The ideal candidate will possess strong expertise in Spark, Scala, Python, Airflow, Databricks, and Microsoft Fabric, with proven experience in migration, performance optimization, workflow modernization, and automated testing.

Key Responsibilities

• Analyze existing Databricks notebooks, jobs, and Spark workloads and migrate them to Microsoft Fabric.

• Design and develop scalable data processing solutions using Apache Spark, Scala, and Python.

• Integrate Microsoft Fabric compute workloads with existing ADLS storage architecture.

• Upgrade and modernize Apache Airflow orchestration workflows and scheduling frameworks.

• Develop automated validation and testing frameworks for migration accuracy and quality assurance.

• Create and execute unit tests, regression tests, and reconciliation processes to ensure functional equivalence between Databricks and Fabric workloads.

• Build automation scripts for testing, deployment, and operational monitoring.

• Optimize performance, scalability, reliability, and cost efficiency of migrated workloads.

• Collaborate with architects, data engineers, QA teams, and business stakeholders to ensure successful migration delivery.

• Produce technical documentation, migration playbooks, and knowledge transfer materials. Mandatory Skills

• Apache Spark

• Scala • Python

• Apache Airflow

• Databricks

• Microsoft Fabric

• SQL

• Azure Data Lake Storage (ADLS) Required Experience

• 5+ years of Data Engineering experience in enterprise data platforms. • Strong hands-on expertise in Spark and Scala for large-scale data processing.

• Experience developing and maintaining Databricks notebooks, jobs, and workflows.

• Hands-on experience with Microsoft Fabric Data Engineering and Lakehouse environments.

• Experience upgrading and managing Apache Airflow workflows and DAGs.

• Strong experience in automated testing frameworks, unit testing, and validation automation.

• Experience developing migration utilities, reconciliation scripts, and deployment automation.

• Strong understanding of distributed computing, data transformation, and performance tuning.

• Experience with Git, CI/CD pipelines, and DevOps practices. Preferred Qualifications

• Experience with Azure cloud services and enterprise data platforms. • Knowledge of Fabric Lakehouse, Data Factory, OneLake, and monitoring capabilities.

• Experience with migration projects involving Databricks, Spark, and cloud modernization.

• Microsoft Fabric, Databricks, or Azure certifications preferred. Success Criteria

• Successful migration of Databricks compute workloads to Microsoft Fabric.

• Seamless integration with existing ADLS data storage without data movement.

• Successful Airflow upgrade and workflow modernization.