Data Engineer (Azure & Databricks) - Unity Catalog

Job Title: Data Engineer (Azure & Databricks)

Experience Required:5-8 & 8 to 10 Years

About the Role

We are looking for a motivated and detail oriented Data Engineer to join our growing analytics team. In this role, you will work closely with Lead Data Engineers and Architects to develop, maintain, and optimize data pipelines using modern cloud technologies.

You will get hands on experience building our Enterprise Data Warehouse and Lakehouse solutions using Python, Databricks, and Microsoft Azure. This is an excellent opportunity for an emerging data professional to grow their skills in cloud data engineering and distributed computing.

Key Responsibilities

• . Pipeline Development: Build, test, and maintain robust ETL/ELT pipelines to move data from various sources into our Azure Data Lake and Data Warehouse.
• . Coding & Transformation: Write clean, efficient PySpark and Python code in Azure Databricks to clean, transform, and aggregate data.
• . Data Integration: Utilize Azure Data Factory (ADF) to orchestrate data movement and schedule automated workflows.
• . SQL Development: Write and optimize SQL queries for data extraction, analysis, and creating views for downstream reporting tools.
• . Monitoring & Support: Monitor daily batch jobs, troubleshoot failed pipelines, and resolve data quality issues in a timely manner.
• . Collaboration: Work alongside senior engineers to understand architectural designs, participate in code reviews, and learn enterprise best practices for data engineering.
• . Documentation: Maintain clear technical documentation for data pipelines, data dictionaries, and operational runbooks.

Required Skills & Qualifications

• . Experience: 8 to 10 years of hands on experience in Data Engineering, ETL development, or Data Analytics.
• . Unity Catalog experience ranging between 3-4 years
• . Python Programming: Strong proficiency in Python for data manipulation (e.g., using Pandas) and scripting.
• . SQL Mastery: Solid understanding of relational databases and the ability to write complex SQL queries, joins, and aggregations.
• . Big Data / Databricks: Hands on experience with Apache Spark (PySpark or Spark SQL) and familiarity with the Databricks environment.
• . Azure Cloud: Exposure to the Azure data ecosystem, specifically working with Azure Data Lake Storage (ADLS Gen2) and creating pipelines in Azure Data Factory (ADF).
• . Data Warehousing Concepts: Good foundational understanding of Data Warehousing principles (Star schema, facts, and dimensions).
• . Version Control: Experience using Git for source code version control.

Preferred/Bonus Skills

• . Understanding of Delta Lake concepts (Time travel, ACID transactions).
• . Familiarity with CI/CD concepts and deploying code using Azure DevOps or GitHub Actions.
• . Prior experience working in an Agile/Scrum environment.
• . Certifications: Azure Data Fundamentals (DP 900) or Databricks Certified Data Engineer Associate.