Databricks Tech Lead

“We are HCLTech, one of the fastest-growing large tech companies in the world and home to 219,000+ people across 54 countries, supercharging progress through industry-leading capabilities centered around Digital, Engineering and Cloud.

The driving force behind that work, our people, are diverse, creative, and passionate, raising the bar for excellence on a regular basis. We, in turn, work hard to bring out the best in them as we strive to help them find their spark and become the best version of themselves that they can be.

Are you ready to be an important part of this ever-transformational journey?

Java Full Stack Developer

Location: Sydney

Role Overview

We are seeking a seasoned Databricks Tech Lead to design, build, and scale our next-generation data ingestion and processing framework. In this role, you will lead a team of data engineers, champion best practices in software development, and build reusable, highly scalable pipelines to ingest and transform data from diverse enterprise sources into our Lakehouse platform.

Key Responsibilities

• Framework Development: Design and build metadata-driven, generic data ingestion frameworks (batch, streaming, and CDC) to automate the onboarding of new data sources.
• Architecture & Design: Define architectural standards for the Medallion architecture (Bronze, Silver, Gold layers) utilizing Delta Lake.
• Technical Leadership: Lead technical design discussions, mentor junior/mid-level engineers, and conduct rigorous code reviews.
• Pipeline Optimization: Optimize Spark jobs for performance, scalability, and cost reduction across the Databricks platform.
• CI/CD & DevOps: Implement and mature CI/CD pipelines (Git, automated testing) for data platforms and enforce DataOps best practices.
• Governance & Security: Integrate ingestion pipelines with Unity Catalog for enterprise-grade data governance and lineage tracking.

Required Qualifications

• Experience: 8+ years in Data Engineering, with at least 3–4 years of hands-on technical leadership experience leading engineering pods.
• Core Databricks & Spark: Advanced proficiency in PySpark and Databricks SQL.
• Ingestion Expertise: Proven experience building custom or highly configurable data ingestion frameworks for structured and unstructured data.
• Cloud Platforms: Strong experience working with AWS, Azure, or GCP cloud-native services (e.g., S3, ADLS).
• Engineering Standards: Deep understanding of data modeling (star/snowflake schemas), SCD (Slowly Changing Dimensions), and software engineering principles in a data context.
• Soft Skills: Excellent stakeholder management and the ability to translate complex technical architectures into business value.

Preferred Qualifications

• Experience with Kafka, dbt, or Delta Live Tables (DLT).
• Databricks Certified Data Engineer Associate/Professional credentials

Why Us

We are one of the fastest-growing large tech companies in the world, with offices in 50+ countries across the globe and 219,000 employees

• Our company is extremely diverse with 165 nationalities represented
• We offer the opportunity to work with colleagues across the globe
• We offer a virtual-first work environment, promoting a good work-life integration and real flexibility
• We offer comprehensive benefits for all employees
• We are a certified great place to work and a top employer in 17 countries, offering a positive work environment that values employee recognition and respect

Equality & Opportunity for All

Representing 165 nationalities across the globe, we pride ourselves on being an equal opportunity employer, committed to providing equal employment opportunities to all applicants and employees regardless of race, religion, sex, colour, age, national origin, pregnancy, sexual orientation, physical disability or genetic information, military or veteran status, Aboriginal and Torres Strait Islander people or any other protected classification, in accordance with federal, state, and/or local law.

Candidate Data Privacy Notice | HCL Technologies

We are committed to respecting your privacy and for the protection of your personal data. Your personal data will be collected and processed in line with our candidate privacy notice: https://www.hcltech.com/candidate-privacy-notice. This privacy notice will help you to understand what personal data we collect about you, how we use this personal data, and what rights you have regarding your personal data. By replying to this email or submitting any personal data to HCLTech, you acknowledge that you have read and understood the candidate privacy notice and have provided your consent to the processing of your data for recruitment purposes as described in the privacy notice”.