Azure Databricks Data Engineer

Responsibilities:

• Work closely with source data application teams and product owners to design, implement and support analytics solutions that provide insights to make better decisions.
• Implement data migration and data engineering solutions using Azure products and services: (Azure Data Lake Storage, Azure Data Factory, Azure Functions, Event Hub, Azure Stream Analytics, Azure Databricks, etc.) and traditional data warehouse tools.
• Perform multiple aspects involved in the development lifecycle – design, cloud engineering (Infrastructure, network, security, and administration), ingestion, preparation, data modeling, testing, CICD pipelines, performance tuning, deployments, consumption, BI, alerting, prod support.
• Design, develop, and maintain scalable data pipelines and transformation jobs on the Databricks Lakehouse Platform to generate business metrics, KPIs, and curated datasets.
• Develop and optimize batch and streaming data pipelines using Databricks, Apache Spark (PySpark/Scala), and Delta Lake technologies.
• Build and maintain data ingestion frameworks from various enterprise source systems into Azure-based data lakes and Databricks environments.
• Implement and maintain Medallion Architecture (Bronze, Silver, and Gold layers) to support analytics and self-service reporting.
• Develop, optimize, and maintain Delta tables and data models to ensure high-performance data processing and analytics.
• Monitor, troubleshoot, and support Databricks workflows, clusters, and production data pipelines to ensure data reliability and availability.
• Implement data quality checks and validation frameworks to ensure data accuracy, completeness, timeliness, and consistency.
• Optimize Spark jobs, cluster utilization, and query performance to improve scalability and cost efficiency.
• Build and deploy Databricks code, workflows, and infrastructure using GitHub Actions -based CI/CD pipelines.

Requirements:

• 3+ years of experience in Data Engineering and building enterprise-scale data platforms.
• 3+ years of cloud development and data lake experience (prefer Microsoft Azure) including Azure EventHub, Azure Data Factory, Azure Functions, ADX, ASA, Azure Databricks, Azure DevOps, Azure Blob Storage, Azure Data Lake, Azure Power Apps and Power BI
• 3+ years of hands-on experience with Databricks and Apache Spark development using PySpark or Scala.
• Strong experience designing and implementing data pipelines on Azure and Databricks platforms.
• Strong SQL skills and experience with data modeling and dimensional modeling techniques.
• Experience developing and deploying solutions using GitHub Actions and Git-based source control practices.
• Experience with data quality, metadata management, and data governance frameworks.
• Strong troubleshooting, performance tuning, and production support experience in cloud-based data platforms.
• Excellent communication and collaboration skills with the ability to work effectively with global teams.
• Databricks certifications such as Databricks Certified Data Engineer Associate or Professional .

Education:

• Bachelor's degree in Computer Science/Engineering or a related field