Senior Databricks & Governance Engineer

Role Overview We are looking for a Senior Databricks & Governance Engineer with 5+ years of Data Engineering experience to develop scalable data pipelines, implement enterprise data governance, and prepare unstructured data for Generative AI applications in a modern Lakehouse environment.

Key Responsibilities

• Develop and maintain batch and streaming ETL/ELT pipelines using Databricks, PySpark, Python, SQL, and Delta Lake.
• Orchestrate data workflows using Azure Data Factory (ADF).
• Extract and process unstructured data from PDFs, images, logs, and text files.
• Prepare data for RAG, LLM applications, embeddings, and vector search.
• Implement data access controls, security, and lineage using Unity Catalog.
• Integrate Databricks metadata with Collibra using REST APIs.
• Design and maintain Bronze, Silver, and Gold layers using Medallion Architecture.
• Develop dimensional and Lakehouse data models.
• Optimize performance using Z-Ordering, Liquid Clustering, and Delta Lake optimization techniques.
• Ensure data quality, governance, metadata consistency, and pipeline reliability.

Mandatory Skills

• 5+ years of Data Engineering experience
• Strong hands-on experience with Azure Databricks
• PySpark, Python, SQL, and Delta Lake
• Azure Data Factory (ADF)
• Unity Catalog — governance, access controls, and lineage
• Collibra REST API integrations — metadata extraction and synchronization
• GenAI / LLM data preparation, including unstructured data processing
• RAG, embeddings, and vector search
• Lakehouse and Medallion Architecture
• Dimensional data modeling and performance optimization

Skills: pyspark,azure data factory,collibra,genai,delta lake,unity catalog,sql,python,azure databricks