Cloud Data Platform Engineer

Job Level: Senior Engineer / Lead Engineer

Function: Cloud Platform Engineering & Data Platform Operations

Primary Focus: Databricks Platform Design, Deployment, Administration, and AI Enablement

Position Summary

We are seeking a highly skilled Cloud Data Platform Engineer to design, deploy, administer, and optimize enterprise-scale Databricks platforms across cloud environments. This role will be responsible for building and

maintaining secure, scalable, and reliable data platforms that enable analytics, machine learning, and AI-driven business solutions.

The ideal candidate combines expertise in cloud infrastructure, platform engineering, data technologies, security, automation, and Databricks administration. Experience with Databricks AI capabilities, including

Generative AI, Vector Search, MLflow, and Mosaic AI, is highly desirable.

Key Responsibilities

Databricks Platform Engineering

l Design and deploy Databricks workspaces, clusters, and platform components in Azure and/or AWS.

l Build and maintain scalable, secure, and highly available Databricks environments.

l Establish platform standards, architecture patterns, and operational best practices.

l Configure and manage Unity Catalog, Delta Lake, and workspace governance frameworks.

l Implement platform lifecycle management, upgrades, and capacity planning.

Cloud Infrastructure & Automation

l Provision infrastructure using Infrastructure-as-Code tools such as Terraform.

l Automate platform deployments and configuration management.

l Design networking, private connectivity, and secure integration patterns.

l Implement backup, disaster recovery, and high-availability solutions.

l Optimize cloud resource utilization and manage platform costs.

Security & Governance

l Implement enterprise security controls and compliance requirements.

l Configure role-based access controls and least-privilege access models.

l Manage secrets, key vault integrations, and encryption standards.

l Establish monitoring, audit logging, and governance controls.

l Support regulatory and security audits.

Platform Operations & Reliability

l Monitor platform health, availability, and performance.

l Develop observability solutions for infrastructure and data workloads.

l Troubleshoot platform, networking, and workload issues.

l Create operational runbooks and support processes.

l Lead root-cause analysis and remediation efforts.

Collaboration & Technical Leadership

l Partner with Data Engineers, Data Scientists, AI Engineers, and Cloud Infrastructure teams.

l Provide technical guidance on platform architecture and operational excellence.

l Develop engineering standards and reusable automation patterns.

l Mentor engineers and promote cloud engineering best practices.

AI & Advanced Analytics Support (Preferred)

l Support Databricks AI and machine learning environments.

l Enable MLflow experimentation, model tracking, and deployment capabilities.

l Support Mosaic AI, Vector Search, Feature Store, and RAG architectures.

l Assist teams implementing Generative AI and Large Language Model solutions.

l Develop infrastructure and governance frameworks supporting AI workloads.

l Evaluate new Databricks AI capabilities and recommend adoption strategies.

Required Qualifications

Education

l Bachelor's degree in Computer Science, Information Technology, Engineering, or a related field.

Experience

l 5+ years of experience in cloud engineering, platform engineering, or infrastructure engineering.

l 3+ years of hands-on experience deploying and administering Databricks environments.

l Experience supporting enterprise cloud platforms in Azure and/or AWS.

l Experience designing highly available, secure, and scalable cloud solutions.

Technical Skills

Databricks

l Databricks Administration

l Unity Catalog

l Delta Lake

l Workspace Management

l Cluster Configuration and Optimization

l Databricks Workflows

l Job Scheduling

l Identity and Access Management

Cloud Platforms

l Microsoft Azure

l Amazon Web Services (AWS)

Infrastructure as Code & Automation

l Terraform

l Azure DevOps or GitHub Actions

l CI/CD Pipelines

l PowerShell, Python, or Bash scripting

Security & Networking

l RBAC

l Private Endpoints

l VNET/VPC Design

l Encryption and Key Management

l Identity Federation

l Secrets Management

Monitoring & Observability

l Databricks Monitoring

l Azure Monitor

l CloudWatch

l Log Analytics

l Performance Tuning and Capacity Management

Preferred Qualifications

l Databricks Certified Platform Administrator.

l Databricks Certified Data Engineer Professional.

l Azure Administrator Associate or Azure Solutions Architect certification.

l AWS Solutions Architect certification.

l Experience implementing enterprise AI and machine learning platforms.

l Experience supporting regulated environments such as Insurance or Financial Services.

Desired Knowledge of AI Features

The ideal candidate should have exposure to one or more of the following:

l Databricks Mosaic AI

l MLflow

l Databricks Model Serving

l Databricks Vector Search

l Feature Store

l Retrieval-Augmented Generation (RAG)

l Large Language Models (LLMs)

l AI Governance and Responsible AI Practices

l AI-enabled Data Engineering Patterns

Key Competencies

l Cloud Architecture

l Platform Engineering

l Infrastructure Automation

l Operational Excellence

l Security and Compliance

l Problem Solving

l Technical Leadership

l Communication and Collaboration

l Continuous Learning and Innovation

Success Measures

l Stable, secure, and high-performing Databricks platform operations.

l Increased platform automation and reduced operational overhead.

l Improved cloud cost efficiency and resource utilization.

l Successful adoption of enterprise data, analytics, and AI capabilities.

l Strong platform governance, compliance, and reliability.

l High customer satisfaction among data engineering and analytics teams.