Databricks Engineer

Position Overview

We are seeking a Senior Databricks Engineer to lead the design and implementation of a modern Databricks Lakehouse architecture . This role is focused on migrating legacy data solutions into scalable, governed, and performance-optimized pipelines using Databricks, PySpark, and Delta Lake.

The engineer will play a key role in defining and building Bronze, Silver, and Gold data layers , enabling both batch and near real-time processing, and improving the reliability and usability of enterprise data platforms. This position will work closely with data engineers, architects, and stakeholders while providing technical leadership and guidance to more junior team members.

Key Responsibilities

Architecture & Engineering

• Architect and implement end-to-end data solutions within the Databricks Lakehouse environment
• Design and build ingestion, transformation, and serving pipelines using PySpar k, Spark SQ L, and Delta Lake
• Establish and enforce medallion architectur e practices across Bronze, Silver, and Gold layer

Migration & Modernizatio

• nLead the migration of legacy Azure SQ L, SSI S, and AD F pipelines into Databricks-native workflows
• Convert existing T-SQ L and ETL logic into scalable Spark-based processing
• Support modernisation of data platforms with a focus on governance, scalability, and maintainability

Streaming, Governance & Optimization

• Implement Auto Loade r, Delta Lake Change Data Feed (CDF ), and streaming/batch ingestion patterns
• Define and enforce governance using Unity Catalo g, including RBAC and data lineage
• Optimise performance through Photo n, partitionin g, Z-Order indexin g, and cluster tuning

Validation & Technical Leadership

• Guide validation and reconciliation strategies between legacy SQL outputs and Delta-based pipelines
• Perform code reviews and promote engineering best practices
• Mentor mid-level and junior engineers across design, development, and troubleshooting activities

Required Experience & Qualifications

• Bachelor’s degree in Computer Science, Engineering, Information Systems, or equivalent experience
• 6+ years of experien ce in data engineering, wit h 3+ years of hands-on Databricks experience
• Strong expertise i n PySpa rk , Spark S QL, an d Delta Lake
• Proven experience migrating legacy data platforms such a s SQL Serv er , SS IS, an d ADF
• Strong understanding o f ETL/ELT desi gn , data modeli ng, and pipeline architecture
• Experience designing data solutions usin g Bronze / Silver / Go ld layers
• Hands-on experience with th e Azure ecosyst em, includin g AD LS , A DF, and related services
• Strong knowledge of performance tuning techniques for distributed data processing
• Experience leading technical implementation efforts and supporting junior engineers

Preferred Qualifications

• Experience wi th Unity Cata lo g, Lakehouse Federat ion, or CDC frameworks
• Databricks certifications (Associate or Professional level)
• Exposure to large-scale regulated datasets such as healthc ar e, cla ims, or financial data
• Experience with bo th ba tch a nd stream ing data pipelines
• Familiarity with enterprise-grade data governance and lineage practices