Databricks Tech Lead

Job Title: Databricks Tech Lead

Location: Sydney

Job Description:
We are seeking a seasoned Databricks Tech Lead to design, build, and scale our next-generation data ingestion and processing framework. In this role, you will lead a team of data engineers, champion best practices in software development, and build reusable, highly scalable pipelines to ingest and transform data from diverse enterprise sources into our Lakehouse platform.

Key Responsibilities

• Framework Development: Design and build metadata-driven, generic data ingestion frameworks (batch, streaming, and CDC) to automate the onboarding of new data sources.
• Architecture & Design: Define architectural standards for the Medallion architecture (Bronze, Silver, Gold layers) utilizing Delta Lake.
• Technical Leadership: Lead technical design discussions, mentor junior/mid-level engineers, and conduct rigorous code reviews.
• Pipeline Optimization: Optimize Spark jobs for performance, scalability, and cost reduction across the Databricks platform.
• CI/CD & DevOps: Implement and mature CI/CD pipelines (Git, automated testing) for data platforms and enforce DataOps best practices.
• Governance & Security: Integrate ingestion pipelines with Unity Catalog for enterprise-grade data governance and lineage tracking.

Required Qualifications

• Experience: 8+ years in Data Engineering, with at least 3–4 years of hands-on technical leadership experience leading engineering pods.
• Core Databricks & Spark: Advanced proficiency in PySpark and Databricks SQL.
• Ingestion Expertise: Proven experience building custom or highly configurable data ingestion frameworks for structured and unstructured data.
• Cloud Platforms: Strong experience working with AWS, Azure, or GCP cloud-native services (e.g., S3, ADLS).
• Engineering Standards: Deep understanding of data modeling (star/snowflake schemas), SCD (Slowly Changing Dimensions), and software engineering principles in a data context.
• Soft Skills: Excellent stakeholder management and the ability to translate complex technical architectures into business value.

Preferred Qualifications

• Experience with Kafka, dbt, or Delta Live Tables (DLT).
• Databricks Certified Data Engineer Associate/Professional credentials

Who we are.

We are Delivery Centric, a dynamic technology company that is transforming the delivery of cloud implementations and Cyber security. We are head-quartered in Australia with global presence. We are tenacious, future-thinking and highly driven to achieve an ambitious vision to be the leading global provider of innovative technologies for companies that keep our world clean, safe and secure. Our tagline "Excellence Delivered, Value Realized" is what keeps us going with our customers and team members.

Please contact me at [email protected] for additional details

Job Type: Full-time

Work Authorisation:

• Australia (Preferred)

Work Location: In person