Azure Data Engineer

Position Summary

We are seeking a skilled Databricks Developer to design, develop, and optimize data
pipelines and analytics solutions on the Databricks Lakehouse Platform. The ideal
candidate will have 8 years of total experience in data engineering and analytics, with
proven expertise in building robust data pipelines, implementing advanced analytics, and enabling machine learning workflows. You will collaborate with data engineers, analysts, and stakeholders to transform raw data into actionable insights, ensuring performance, scalability, and reliability.

Key Responsibilities

• Design, build, and maintain scalable ETL/ELT pipelines using Databricks, Apache
  Spark, and Delta Lake.
• Develop and optimize PySpark and Spark SQL jobs for performance, cost, and
  reliability on large-scale datasets.
• Implement data ingestion from various sources including REST APIs, streaming
  platforms, and cloud storage into the Lakehouse architecture.
• Build and manage Unity Catalog for data governance, lineage, security, and access
  control.
• Collaborate with data scientists and analysts to enable ML workflows, feature stores, and real-time analytics using Databricks.
• Automate workflows using Databricks Workflows, Jobs API, and CI/CD pipelines with Azure DevOps or GitHub Actions.
• Monitor cluster performance, troubleshoot bottlenecks, and implement best practices for cost optimization.
• Ensure data quality, validation, and testing frameworks are implemented across all
  pipelines.

Required Qualifications
Technical Skills

• 5+ years of hands-on experience with Databricks and Apache Spark
• Strong proficiency in Python and/or Scala
• Expert-level Spark SQL and Delta Lake experience
• Experience with cloud platforms: Azure, AWS, or GCP
• Data warehousing concepts and dimensional modeling

Experience & Education

• Bachelors degree in Computer Science, Engineering, or related field
• 8+years overall in data engineering or related roles
• Experience with Git, CI/CD, and Agile methodologies
• Strong problem-solving and communication skills

Preferred Qualifications

• Databricks Certified Data Engineer Professional
• Unity Catalog
• Delta Live Tables dbt Kafka / Event Hubs
• Azure Data Factory
• Power BI / Tableau