Senior Fabric Data Engineer (ID#5544)

New Value Solutions is a national IT consulting company. We are searching a Senior Microsoft Fabric Data Engineer who is a hands on builder of Microsoft Fabric environments, the CI/CD pipeline, the metadata driven onboarding framework and the pilot source onboarding.

This is a contract role, estimated to be 300-400 hours over a 3-4 month period. Definite possibilities for more hours as needed. We also have another project for a different client which may require additional assistance.

Responsibilities

• Build Dev, Test and Prod exactly as approved in the foundation design, using Bicep for Azure resources (capacity, gateway subnet, private endpoints, private DNS, Key Vault) and the Microsoft Fabric REST APIs for capacity assignment, domains, workspaces, private links, managed private endpoints, gateway connections and Entra ID group role assignments.

• Run the connectivity proof on non sensitive data to confirm private endpoint access, gateway connectivity and mirroring behaviour under the proposed network model.

• Implement source control and CI/CD: Git integration for workspaces, Azure DevOps Repos and Pipelines, fabric-CI/CD deployment from a build agent inside the client's VNet, parameter files and variable libraries, approval gates and release notes. Demonstrate a change moving Dev to Test to Prod.

• Build the metadata driven onboarding framework: source registry, pipeline and notebook templates, validation and reconciliation templates.

• Onboard the approved pilot source into Bronze, Silver and Gold, schedule and monitor loads, and run reconciliation, freshness, performance, access and recovery tests.

• Write the operations runbook, the onboarding playbook and the environment verification checklist.

• Pair with client engineers throughout and coach them while they onboard part of the pilot themselves.

• Deliver knowledge transfer workshops on CI/CD and release process, onboarding a new source and operations and troubleshooting.

Required experience and skills

• At least two years building on Microsoft Fabric in production: Lakehouse, notebooks (PySpark and Spark SQL), Data Factory pipelines, Copy jobs, Dataflows Gen2, Mirroring, Delta tables and table maintenance.

• Azure DevOps YAML pipelines and Git workflows, including branch policies and pull request review. Experience deploying Fabric items with fabric-CICD or the Fabric REST APIs.

• Infrastructure as code with Bicep (or Terraform) for Azure networking and platform resources.

• Practical experience with VNet data gateways or on premises data gateways and with sources behind private endpoints.

• Strong SQL and Python. Able to design metadata driven, configuration first frameworks rather than one off pipelines.

• Data validation and reconciliation practices: row counts, checksums, freshness checks, failure handling and alerting.

• Able to explain what they build to client engineers and to document it clearly.

Preferred Skills, an asset

• DP-700 (Fabric Data Engineer) or DP-600 certification.

• Oracle and Azure SQL Database source experience, including LogMiner or change data capture prerequisites.

• Power BI semantic model basics, Microsoft Purview, Fabric capacity monitoring and cost management.

• Experience in public sector environments with formal security review.