Azure Databricks Developer (SJ-IQ)

Job Description: Senior Developer - Azure Databricks (Unity Catalog & Genie AI)

Job Title: Senior Developer - Azure Databricks, Unity Catalog & Genie AI

Experience: 6-10 Years

Location: Bengaluru / Gurgaon/ Kochi

Employment Type: Full-Time

Role Overview

We are seeking an experienced Senior Developer - Azure Databricks to design, develop, and optimize enterprise-scale data and AI platforms on Azure. The ideal candidate will possess strong expertise in Azure Databricks, Unity Catalog, Delta Lake, Medallion Architecture, and hands-on experience with Databricks Genie AI for conversational analytics and AI-driven data consumption.

The role involves building modern data platforms, implementing governance frameworks, enabling self-service analytics, and supporting AI/GenAI initiatives across business functions.

Key Responsibilities

Data Engineering & Platform Development

• Design and develop scalable data pipelines using Azure Databricks and Apache Spark.
• Implement ETL/ELT solutions for structured, semi-structured, and unstructured data.
• Develop and optimize Delta Lake-based data platforms following Medallion architecture (Bronze, Silver, Gold).
• Create reusable frameworks for ingestion, transformation, data quality, and monitoring.
• Optimize Spark workloads for performance, scalability, and cost efficiency.

Unity Catalog & Data Governance

• Design and implement enterprise data governance using Unity Catalog.
• Configure catalogs, schemas, tables, views, and external locations.
• Implement fine-grained access controls including row-level and column-level security.
• Establish data lineage, metadata management, and governance policies.
• Collaborate with data stewards and security teams to ensure compliance and auditability.

Databricks Genie AI & Self-Service Analytics

• Develop and configure Genie Spaces for conversational analytics.
• Enable natural language querying for business users across governed data domains.
• Create semantic layers, business definitions, and trusted datasets for Genie.
• Partner with business stakeholders to improve data discoverability and self-service adoption.
• Integrate Genie with enterprise AI and analytics use cases.
• Collaborate with Data Scientists and ML Engineers for model operationalization.

Required Qualifications Technical Skills

• 6+ years of experience in Data Engineering.
• 4+ years of hands-on experience with Azure Databricks.
• Strong expertise in:
• Apache Spark (PySpark)
• Delta Lake
• Unity Catalog
• Databricks Workflows
• Databricks SQL
• Experience implementing Medallion Architecture and Lakehouse solutions.
• Strong understanding of data modelling principles and dimensional modelling.
• Experience with Azure ecosystem:
• Azure Data Factory
• Azure Storage (ADLS Gen2)
• Azure Key Vault
• Azure Entra ID (Azure AD)
• Experience with Git, CI/CD, and DevOps practices.

Unity Catalog Expertise

• Catalog and schema management.
• Data lineage and metadata governance.
• Security and access management.
• External locations and storage credentials.
• Regulatory compliance and audit support.

Genie AI Expertise

• Designing Genie Spaces and conversational experiences.
• Creating trusted semantic datasets.
• Natural language query optimization.
• Business glossary and metadata integration.
• User enablement and self-service analytics adoption.