Cloud Data Engineer

Role & responsibilities

• Contribute to the migration of the ETL workloads from on-premises platforms to Azure Cloud, Databricks and Snowflake.
• Develop and implement data platform solutions using Databricks, Azure Data Factory (ADF), Self-hosted Integration Runtime (SHIR), Logic Apps, Azure Data Lake Storage Gen2 (ADLS Gen2), Blob Storage, and Snowflake.
• Review and analyze existing on-premises ETL processes.
• Implement DevOps practices and CI/CD pipelines using GitHub Actions.
• Collaborate with cross-functional teams to ensure seamless integration and data flow.
• Tune and optimize workloads for performance scalability.
• Implement data quality validation, monitoring, and observability processes
• Support the development and maintenance of automated reconciliation and data validation processes.
• Ensure data security and compliance with industry standards.

Required Qualifications:

• Minimum of 3+ years of experience as a Cloud Data Engineer.
• Hands-on experience with Databricks, Azure Cloud data tools (ADF, SHIR, Logic Apps, ADLS Gen2, Blob Storage), Apache Spark, PySpark and Snowflake.
• Experience developing or supporting data migration and cloud modernization initiatives.
• Strong experience in ETL development using on-premises databases and ETL technologies.
• Experience with Python or other scripting languages for data processing.
• Strong SQL development experience including complex queries, stored procedures, views, and ETL transformations.
• Experience with GitHub branching, pull requests, and GitHub Actions
• Experience with code reviews and engineering best practices.
• Experience supporting automated testing and data quality processes.
• Experience working within Agile delivery methodologies.
• Ability to adapt to new technologies and learn quickly.
• Experience using AI-assisted development tools (GitHub Copilot, Databricks Assistant, ChatGPT, Claude, etc.) to improve engineering productivity.

Preferred Qualifications:

• Experience with other cloud platforms and data solutions.
• Experience with data modeling and database design.
• Experience implementing Bronze/Silver/Gold (Medallion) architecture patterns.
• Experience with Delta Lake and Lakehouse architectures and optimization techniques.
• Knowledge of data governance and data quality best-practices.
• Experience with development in Snowflake for data engineering and analytics workloads.
• Experience with healthcare payer data domains including Claims, Membership, Enrollment, Provider, Clinical, and Financial data.
• Experience with Airflow or other orchestration frameworks.
• Experience with performance optimization in Databricks and Snowflake.
• Experience leveraging AI/ML solutions for data engineering workflows and automation.
• Certification in Azure or Databricks.
• Excellent analytical, problem-solving, communication, and collaboration skills.
• Demonstrated ability to work independently, manage ambiguity, and deliver high-quality solutions.
• Demonstrated ability to work within tight deadlines and effectively raise critical issues.Preferred Qualifications:
• Experience with other cloud platforms and data solutions.
• Experience with data modeling and database design.
• Experience implementing Bronze/Silver/Gold (Medallion) architecture patterns.
• Experience with Delta Lake and Lakehouse architectures and optimization techniques.
• Knowledge of data governance and data quality best-practices.
• Experience with development in Snowflake for data engineering and analytics workloads.
• Experience with healthcare payer data domains including Claims, Membership, Enrollment, Provider, Clinical, and Financial data.
• Experience with Airflow or other orchestration frameworks.
• Experience with performance optimization in Databricks and Snowflake.
• Experience leveraging AI/ML solutions for data engineering workflows and automation.
• Certification in Azure or Databricks.