Microsoft Fabric Data Engineer

Key Responsibilities

1. Data Engineering & Platform Development

• Design and implement enterprise-scale Lakehouse architectures using OneLake and Delta Lake.
• Build and maintain robust batch, incremental, CDC, and near real-time data ingestion pipelines.
• Develop scalable ETL/ELT solutions using Fabric Data Pipelines, Dataflows Gen2, PySpark, and SQL.
• Implement and manage Medallion Architecture (Bronze, Silver, Gold).
• Develop reusable and metadata-driven ingestion and transformation frameworks.
• Integrate data from ERP systems, SAP, REST APIs, SQL Server, Dataverse, and other enterprise applications.
• Design and maintain enterprise data models supporting analytical and operational workloads.

2. Fabric Engineering & Optimization

• Develop and optimize Fabric Lakehouses and Data Warehouses.
• Build advanced Notebook-based transformations utilizing PySpark and Spark SQL.
• Implement Delta Lake capabilities including: Merge/Upsert Change Data Feed (CDF) Time Travel Schema Evolution Optimize Vacuum
• Design scalable storage, partitioning, and file management strategies within OneLake.
• Optimize Spark workloads and Data Warehouse performance.

3. Performance, Reliability & Monitoring

• Tune large-scale Spark and SQL workloads.
• Implement monitoring, alerting, and operational dashboards using Fabric Monitoring Hub and Metrics.
• Conduct performance testing and scalability assessments.
• Troubleshoot pipeline failures and platform performance bottlenecks.
• Define and monitor SLAs for critical data assets and pipelines.
• Ensure high availability and operational excellence across data workloads.

4. DevOps & Platform Automation

• Implement CI/CD using Fabric Git Integration and Deployment Pipelines.
• Build automated deployment processes across Development, Test, and Production environments.
• Develop automated testing frameworks for data quality, schema validation, and regression testing.
• Manage environment configurations, secrets, and deployment parameters.
• Utilize Azure DevOps or GitHub Actions to support release automation and governance.
• Define rollback, release management, and change control processes.

5. Data Quality, Governance & Security

• Implement automated data quality validation frameworks.
• Develop reconciliation, completeness, and consistency checks.
• Manage data lineage, metadata, and documentation standards.
• Implement sensitivity labels, access controls, auditing, and compliance requirements.
• Ensure adherence to organizational standards for governance, security, retention, and PII handling.
• Manage workspace permissions, Managed Identities, RLS, and CLS where required.

6. Technical Leadership & Collaboration

• Translate business requirements into scalable technical solutions.
• Collaborate with Data Architects, Data Analysts, Integration Engineers, and business stakeholders.
• Lead engineering best practices and platform standards.
• Mentor junior engineers and support knowledge-sharing initiatives.
• Produce architecture documentation and technical design specifications.
• Drive continuous improvement of the data platform and engineering practices.

Qualifications and Experience:

• Bachelor's degree in Computer Science, Data Engineering, Information Systems, or a related field.
• Minimum 5 years of experience in Data Engineering.
• Minimum 2 years of hands-on experience with Microsoft Fabric, Azure Data Engineering, Databricks, or modern Lakehouse platforms.
• Proven experience designing and building enterprise-grade data platforms.

Required Technical Skills

• Microsoft Fabric
• OneLake
• Lakehouse
• Delta Lake
• Fabric Data Warehouse
• Data Pipelines
• Dataflows Gen2
• Notebooks
• Monitoring Hub
• Deployment Pipelines
• Git Integration
• Data Engineering
• PySpark
• Spark SQL
• SQL / T-SQL
• ETL / ELT Development
• Data Modeling
• Incremental Processing
• Change Data Capture (CDC)
• Data Quality Frameworks
• Medallion Architecture
• Performance Optimization
• Delta Lake Optimization
• Partitioning Strategies
• File Management
• Query Optimization
• Spark Performance Tuning
• Workload Management
• Integration
• REST APIs
• SQL Server
• SAP
• Azure Storage
• Dataverse
• Enterprise Application Integration
• DevOps
• Git
• Azure DevOps
• GitHub Actions
• YAML Pipelines
• CI/CD
• Automated Testing
• Security & Governance
• Row-Level Security (RLS)
• Column-Level Security (CLS)
• Sensitivity Labels
• Managed Identities
• Data Lineage
• Audit and Compliance Controls

Desired Candidate Profile

What we are looking for:

We are looking for entrepreneurs, techies, passionate, eager to learn, humble, with a positive attitude and a high level of integrity People. Flexible and willing to take challenges, work and live in coffee-producing countries, People who want to build expertise and a career in the coffee business and are ready to go the extra mile.

Qualifications and Experience:

• Bachelor's degree in Computer Science, Data Engineering, Information Systems, or a related field.
• Minimum 5 years of experience in Data Engineering.
• Minimum 2 years of hands-on experience with Microsoft Fabric, Azure Data Engineering, Databricks, or modern Lakehouse platforms.
• Proven experience designing and building enterprise-grade data platforms.