Senior Data Engineer- Azure

Role Summary

We are hiring a Senior Data Engineer to drive the development of scalable, high-performance data pipelines for a migration project to a modern Azure Native architecture. This role will be instrumental in building an AI-ready data platform using Microsoft Fabric (Fabric IQ) and enabling seamless integration with AI/ML workflows (Foundry IQ).

Key Responsibilities

• Design and build robust, scalable data pipelines (batch + near real-time) using Microsoft Fabric (Data Pipelines, Lakehouse)

• Lead migration of existing pipelines (AKS/custom ETL) to Fabric-native architecture

• Implement Medallion Architecture (Bronze Silver Gold) for structured and unstructured healthcare data

• Develop and optimize data models and transformations for analytics and AI use cases

• Build ingestion frameworks for:

• APIs (FHIR, EHR systems)

• Files (JSON, HL7, CSV)

• Ensure data quality, validation (FHIR/claims), and observability across pipelines

• Enable AI/ML workflows by preparing datasets for:

• RAG pipelines

• Feature engineering and model consumption

• Optimize pipeline performance, cost, and scalability

• Collaborate with Architects, AI/ML engineers, and Product teams to translate requirements into data solutions

• Mentor junior engineers and drive engineering best practices

Required Experience

• 4-6 years of experience (3+ years in data engineering)
• Strong hands-on experience with Azure Data Platform
• Proven experience in building and optimizing large-scale data pipelines
• Experience in data platform modernization or migration projects

Technical Skills

• Languages: Python, PySpark, SQL (advanced)

• Azure / Fabric:

• Microsoft Fabric (preferred) OR Azure Data Factory, Synapse, Data Lake

• Data Architecture:

• Lakehouse, Medallion Architecture

• Batch + streaming data processing

• Data Handling:

• JSON, Parquet, CSV, API integrations

• Performance Optimization:

• Partitioning, indexing, query tuning

Nice to Have

• Experience with healthcare data (FHIR, claims, EHR)
• Exposure to RAG pipelines / vector databases / AI data prep
• Familiarity with event-driven architectures
• Understanding of data security & governance in regulated environments

Success Metrics

• High-quality, scalable pipelines delivered on time
• Seamless migration with minimal business disruption
• Improved data performance, reliability, and cost efficiency
• Strong enablement of AI/ML and analytics use cases

What Were Looking For

• Strong hands-on engineer with ownership mindset
• Ability to operate in fast-paced, evolving architecture environment
• Problem-solver with focus on performance, scalability, and data quality
• Mentor and team player

Why This Role Matters

This role is critical to building the foundation of AI-first platform, enabling:

• Scalable data processing via Fabric IQ
• AI-driven intelligence via Foundry IQ
• Future-ready healthcare data platform