Data Engineer

Data Engineer: Microsoft Fabric, Batch and Streaming Pipelines

Experience: 5+ years (adjust to the client's level) | Hours: Full-time, 40 hrs/week, aligned to client's team time zone

Role Overview
Join a product-focused team to build and run the data pipelines behind customer-facing applications and analytics. You'll own your pipelines from first design through production support, with a focus on Microsoft Fabric.

Key Responsibilities

• Develop robust batch and streaming pipelines that pull data from APIs, databases, event streams, and external vendor systems.
• Create reusable patterns for ingesting, transforming, and storing data, and model it into consistent, trusted datasets.
• Build and troubleshoot workflows in Microsoft Fabric Notebooks and move them into production.
• Put automated testing, version control, and CI/CD around every pipeline you ship.
• Set up data quality checks, alerting, and monitoring so issues surface before users notice them.
• Tune queries, pipelines, and storage to keep runtime and cloud spend under control.
• Maintain clear documentation of data flows, lineage, and metadata.
• Work within agile ceremonies (planning, standups, retros) alongside product managers, developers, and analysts.
• Follow data privacy and security standards, and help mentor junior engineers.

Mandatory Skills

• Programming: Python, Java, or Scala for pipeline development
• Data Frameworks: Hands-on with at least two of Spark, Kafka, Flink, dbt
• Databases: Strong SQL plus working knowledge of NoSQL; solid data modeling fundamentals
• Platform: Microsoft Fabric Notebooks, used in real project work
• Containers & Orchestration: Docker and Kubernetes
• Monitoring: Experience with observability tooling; Datadog strongly preferred
• Performance: Proven SQL and pipeline optimization for speed and cost
• Good to have: Fabric Lakehouse, Warehouse, Data Pipelines, Dataflows Gen2, Semantic Models; Azure or AWS data services; CI/CD tools (GitHub Actions, Jenkins, Azure DevOps); use of generative or agentic AI tools to speed up engineering work