Databricks Infrastructure & DevOps Engineer

About the Role

We are looking for an experienced Databricks Platform Engineer to help design, build, and operate a modern enterprise data platform. This role focuses on developing core platform automation, deployment frameworks, governance capabilities, and observability solutions that enable scalable, secure, and reliable data operations across the organization.

You will play a critical role in building and maintaining data contract-driven deployment systems, automation frameworks, monitoring infrastructure, and governance tooling. The ideal candidate combines deep Databricks expertise with strong Python development skills and software engineering best practices.

This position requires a hands-on engineer who can proactively identify complex design challenges, propose practical solutions, and contribute throughout the entire platform lifecycle, from architecture and implementation to operations and continuous improvement.

Responsibilities

Data Contract System & Deployment Pipeline

• Design and implement a contract-driven automation bundle generator that converts Open Data Contract Standard (ODCS) YAML files into deployable Databricks assets.
• Build and maintain deployment pipelines for Declarative Automation Bundles, including:
• Environment parameterization
• Deployment atomicity
• Rollback mechanisms
• Develop and operate Azure DevOps/Azure Pipelines CI/CD workflows, including:
• Multi-stage pipelines
• Environment approvals
• Variable groups
• Service connections
• Automate governance processes within Unity Catalog, including:
• Schemas
• Tables
• Tags
• Grants
• Develop integrations utilizing Databricks SDKs and REST APIs.
• Generate and execute SQL DDL statements for row-level filters and column-level masking based on data contract definitions.
• Design and maintain a contract index, including recovery and atomic maintenance mechanisms.
• Implement dead-letter asset deployment patterns and ensure deployment consistency.
• Establish and enforce Git workflows, including branching models, merge controls, and promotion gates.
• Develop schema validation frameworks, smoke testing processes, and idempotent deployment verification.

Platform Observability & Monitoring

• Build platform health dashboards and operational alerting systems using Databricks system tables.
• Design freshness monitoring using Delta transaction log analysis.
• Implement SLA/SLO monitoring and compliance validation against contractual data commitments.
• Develop dead-letter accumulation monitoring and other critical platform health indicators.
• Design monitoring and alerting for:
• SCIM synchronization latency
• Deployment ordering failures
• Emergency access ("break-glass") activations
• Develop cost attribution and reporting capabilities.
• Build data quality monitoring solutions using Unity Catalog quality constraints and expectation-based frameworks.

Platform Infrastructure

• Contribute to Databricks workspace administration and deployment activities.
• Collaborate with cloud platform and infrastructure teams to improve overall platform operations and scalability.

Required Qualifications

• 5+ years of experience in Data Engineering, Data Platform Engineering, or DataOps.
• 3+ years of hands-on experience with Databricks, including:
• Unity Catalog
• Workspace administration
• Databricks SDK
• REST APIs
• Strong Python programming skills, including:
• Automation development
• YAML processing
• Code generation
• API integrations
• Solid software engineering fundamentals, including:
• Modular architecture
• Code maintainability
• Testing methodologies
• Error handling
• Documentation
• Experience with Declarative Automation Bundles or similar infrastructure-as-code deployment frameworks in Databricks environments.
• Strong understanding of governance and access control models, including:
• RBAC
• Schemas
• Grants
• Column masking
• Row filtering
• Experience building CI/CD pipelines, preferably using Azure DevOps and Azure Pipelines.
• Experience designing and maintaining:
• Monitoring platforms
• Operational dashboards
• Alerting solutions
• Reporting systems

Preferred Qualifications

• Strong SQL skills.
• Familiarity with the Open Data Contract Standard (ODCS) or equivalent data contract frameworks.
• Experience working with Delta Lake internals, including:
• Transaction log analysis
• Audit trails
• Schema evolution
• Experience implementing data quality frameworks and SLA/SLO monitoring.
• Knowledge of:
• Azure Entra ID
• SCIM provisioning
• Workload identity federation
• Experience designing enterprise Git workflows and release management processes.
• Experience supporting IT operational domains such as:
• Asset management
• License management