Azure Databricks Platform Admin

Role Overview

Service Line: Data Analytics Unit

Responsibilities

• Azure Databricks Platform Administration: Administer and manage Azure Databricks workspaces, clusters, and jobs
• Configure and maintain Databricks environments (Dev, QA, Prod)
• Monitor cluster performance, job execution, and system health
• Manage notebooks, libraries, and dependencies
• Perform upgrades, patches, and platform maintenance
• Cluster Management Optimization: Design and manage cluster configurations (auto-scaling, instance types, pools)
• Optimize job performance, resource utilization, and execution times
• Troubleshoot cluster failures, memory issues, and job bottlenecks
• Implement best practices for Spark performance tuning
• Security Governance: Implement and enforce data security and access controls (RBAC, ACLs, IAM)
• Manage workspace security, secrets (Key Vault integration), and credential passthrough
• Ensure compliance with enterprise security and governance policies
• Implement data governance, auditing, and lineage tracking
• Azure Cloud Integration: Integrate Databricks with Azure services such as: Azure Data Lake Storage (ADLS Gen2), Azure Key Vault, Azure Data Factory (ADF), Azure Synapse Analytics
• Manage connectivity and networking (VNet, Private Endpoints, NSGs)

Educational Requirements

• BSc
• BTech
• BCA
• MTech
• MCA
• MSc

Technical and Professional Requirements

• Core Skills1015 years of IT experience with strong Azure Databricks administration
• Hands-on expertise in Apache Spark and Databricks platform management
• Strong experience with Azure cloud services (ADLS, ADF, Key Vault, Networking)
• Proven experience in production support and platform operations
• Deep knowledge of cluster configuration, job scheduling, and performance tuning
• Strong understanding of Spark architecture and optimization techniques
• Experience with Terraform / ARM / Bicep (IaC tools)
• Knowledge of security models, IAM, RBAC, and data governance
• Familiarity with CI/CD tools and DevOps practices

Preferred Skills

• Experience with Delta Lake and Lakehouse architecture
• Exposure to Python / PySpark / Scala
• Familiarity with monitoring tools (Azure Monitor, Splunk)
• Knowledge of multi-cloud environments (AWS/GCP)
• Experience in data engineering or ETL pipelines