Data Engineer

We are seeking an experienced Azure Databricks Engineer with 6+ years of experience in Data Engineering and strong expertise in Azure cloud technologies. The ideal candidate should have hands-on experience building scalable data pipelines using Azure Databricks, PySpark, and Azure Data Factory, along with a solid understanding of US Healthcare data models, claims, provider, member, and clinical datasets.

The candidate will be responsible for designing, developing, and optimizing enterprise-scale data solutions while ensuring compliance with healthcare data standards and governance requirements.

Key Responsibilities

• Design, develop, and maintain scalable ETL/ELT pipelines using Azure Databricks and Azure Data Factory.
• Build and optimize data ingestion, transformation, and processing frameworks using PySpark and Spark SQL.
• Develop and maintain data solutions using Azure Data Lake Storage (ADLS Gen2), Azure Synapse Analytics, and Azure SQL.
• Implement Lakehouse architecture using Delta Lake and Databricks best practices.
• Work with large healthcare datasets including Claims, Membership, Provider, Clinical, EHR/EMR, HL7, and FHIR data.
• Ensure data quality, governance, security, and compliance with healthcare regulations.
• Perform performance tuning and optimization of Databricks clusters, notebooks, and Spark jobs.
• Collaborate with Data Architects, Business Analysts, Data Scientists, and Healthcare SMEs.
• Develop reusable frameworks and CI/CD pipelines using Azure DevOps.
• Support production deployments, monitoring, troubleshooting, and incident resolution.

Required Skills -Technical Skills

• 3+ years of experience in Data Engineering.

• 3+ years of hands-on experience with Azure Databricks.

• Strong expertise in:

• Azure Databricks

• PySpark

• Spark SQL

• Python

• SQL

• Azure Data Factory (ADF)

• Azure Data Lake Storage (ADLS)

• Azure Synapse Analytics

• Delta Lake

• Experience with Data Warehousing concepts, Data Modeling (Star/Snowflake Schema), and ETL/ELT development.

• Knowledge of Medallion Architecture (Bronze, Silver, Gold layers).

• Experience with Git, Azure DevOps, and CI/CD implementation.

• Strong understanding of performance tuning and optimization techniques in Databricks.

US Healthcare Experience (Good to Have)

• Minimum 2+ years of experience working in the US Healthcare domain.

• Strong understanding of:

• Claims Data

• Member Data

• Provider Data

• Clinical Data

• EHR/EMR Systems

• HL7/FHIR Standards

• HIPAA Compliance

• PHI/PII Data Handling

• Experience supporting healthcare analytics and reporting initiatives.

Preferred Skills

• Experience with Azure Event Hub or Kafka.
• Knowledge of MLflow and Databricks Workflows.
• Experience with Power BI.
• Exposure to Microsoft Purview or other Data Governance tools.
• Azure Data Engineer or Databricks Certification preferred.

Educational Qualification

• Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, or a related field.

Key Competencies

• Strong analytical and problem-solving skills.
• Excellent communication and stakeholder management skills.
• Ability to work independently in a fast-paced environment.
• Experience collaborating with global teams and business stakeholders.

Mandatory Skills: Azure Databricks, PySpark, Python, SQL, Azure Data Factory, Azure Synapse, ADLS Gen2, Delta Lake, US Healthcare Domain.

Interview Rounds 3 rounds

Shift Timings 2 PM to 11 PM

Location: Bangalore

Work Mode – 2 days WFO in a week

Regards,

[email protected]