Databricks Platform Engineer

Job Title: Databricks Platform Engineer
Location: Sydney

Job description:

Role Overview

The Databricks Platform Administrator is responsible for the administration, reliability, security, and performance of the enterprise Databricks platform. This role ensures that Databricks environments are scalable, available, cost‑efficient, and governed, while supporting data engineering, analytics, machine learning, and AI workloads across development, test, and production environments.

Key Responsibilities

• Administer and operate enterprise Databricks workspaces across development, QA, production, and disaster recovery environments.
• Manage Databricks workspace configuration including clusters, jobs, pools, policies, secrets, and init scripts.
• Design, configure, and enforce cluster policies to ensure security, stability, and cost control.
• Manage Databricks compute including autoscaling, instance pools, job clusters, and all‑purpose clusters.
• Administer Databricks Jobs, Workflows, and scheduling for batch and streaming workloads.
• Configure and manage Databricks security including workspace access controls, IAM integration, permissions, and secrets management.
• Implement and manage Databricks Unity Catalog for data governance, access control, and lineage.
• Configure and support integration with cloud storage (Azure Data Lake, S3, GCS) and external data sources.
• Administer Delta Lake features including table management, versioning, optimization, and retention policies.
• Monitor platform health, performance, and availability using logs, metrics, and alerts.
• Troubleshoot Spark workloads including job failures, performance degradation, memory issues, and scaling problems.
• Support ML and AI workloads using Databricks ML Runtime, Model Registry, Feature Store, and notebooks.
• Implement CI/CD processes for Databricks notebooks, jobs, and configurations using Git-based workflows.
• Manage Databricks REST APIs, CLI tools, and automation scripts for platform operations.
• Perform capacity planning, usage analysis, and cost optimisation across Databricks environments.
• Apply platform patches, runtime upgrades, and configuration changes following change management best practices.
• Act as the escalation point for Databricks production incidents and lead root cause analysis.
• Maintain operational documentation, standards, and platform runbooks.
• Collaborate with data engineers, data scientists, MLOps, cloud, and security teams.
• Provide on‑call production support through a rotation schedule.

Qualifications & Experience

• 5+ years experience administering data or analytics platforms in enterprise environments.
• Strong hands‑on experience administering Databricks workspaces and Spark‑based platforms.
• Deep understanding of Apache Spark architecture, execution, and performance tuning.
• Experience managing Databricks clusters, jobs, workflows, and autoscaling configurations.
• Strong understanding of Databricks security concepts including workspace access controls, IAM integration, and secrets management.
• Hands‑on experience with Unity Catalog and data governance concepts.
• Experience working with Delta Lake and lakehouse architectures.
• Strong knowledge of cloud platforms (Azure, AWS, or GCP) and cloud storage services.
• Proficiency in Python and Bash scripting.
• Good understanding of SQL and distributed data processing concepts.
• Experience implementing CI/CD pipelines for data platforms using Git‑based tooling.
• Strong troubleshooting, analytical, and problem‑solving skills.
• Strong communication and stakeholder engagement skills.
• Bachelor’s degree in Computer Science, Engineering, Data Engineering, or a related discipline.

Preferred Skills

• Databricks Administrator or Data Engineer certification.
• Experience supporting ML and AI workloads on Databricks.
• Experience with streaming technologies such as Spark Structured Streaming or Kafka.
• Familiarity with Infrastructure as Code tools such as Terraform for Databricks.
• Experience working in regulated or security‑sensitive environments.
• Familiarity with ITIL‑based enterprise service management processes.

About Us:

We are Delivery Centric, a dynamic technology company that is transforming the delivery of cloud implementations and cybersecurity. We are headquartered in Australia with a global presence. We are tenacious, future-thinking, and highly driven to achieve an ambitious vision to be the leading global provider of innovative technologies for companies that keep our world clean, safe, and secure.

Our fast-paced and supportive environment will offer you lots of benefits, including a competitive salary, a friendly team, a healthy work environment, and opportunities for training and development.

Please contact us at [email protected] for additional details

Job Type: Full-time

Work Authorisation:

• Australia (Preferred)

Work Location: In person