Lead Data Engineer – GCP

Lead Data Engineer – GCP

Location: Sydney, NSW preferred | Melbourne, VIC also considered
Employment Type: 12-Month Fixed-Term Contract (FTC)
Industry: Financial Services / FSI
Work Rights: Australian Citizens or candidates with full working rights in Australia will be considered

About the Role

We are looking for an experienced Lead Data Engineer to join a large-scale Financial Services project focused on building and modernising a Transactions Ingestion Data Platform supporting both real-time and batch data processing.

This is a hands-on technical leadership role, combining solution architecture, data engineering, customer engagement and technical delivery.

Key Responsibilities

• Lead technical architecture and solution design for large-scale transaction data platforms.
• Design and develop real-time and batch data ingestion and processing pipelines.
• Build solutions using GCP Dataflow, BigQuery, Cloud Spanner, Pub/Sub and Confluent Kafka.
• Develop event-driven pipelines from Kafka/GCS into Spanner and BigQuery.
• Design batch ingestion pipelines from files, databases, APIs and other sources.
• Provide technical guidance and best-practice recommendations to customer data engineering teams.
• Define solution architecture, detailed designs and hands-on implementations.
• Perform pair programming and knowledge transfer with customer engineers.
• Lead sprint planning, backlog management, scope, risks and technical delivery.
• Work as a workstream or overall technical lead depending on project scale.
• Develop and maintain CI/CD pipelines using Cloud Build, Artifact Registry and Terraform.
• Develop Cloud Composer DAGs and orchestrate data workflows using Cloud Workflows.
• Optimise BigQuery queries and data processing performance.
• Support data governance initiatives using GCP governance tools such as Dataplex and Data Catalog.
• Contribute to enterprise Generative AI initiatives and integrations.

Essential Skills & Experience

• Strong hands-on experience with GCP data and analytics technologies.
• Experience designing large-scale transaction data platforms.
• Strong experience with:
• GCP Dataflow
• BigQuery
• Cloud Spanner
• Pub/Sub
• Cloud Composer
• Cloud Workflows
• Confluent Kafka
• Cloud Run
• Cloud Build
• Terraform
• Strong programming skills in Python and Java.
• Experience building both real-time/event-driven and batch data pipelines.
• Experience with cloud data platform migrations and/or greenfield GCP implementations.
• Experience with CI/CD and infrastructure-as-code for data workloads.
• Strong solution design and technical architecture capabilities.
• Excellent communication and customer-facing consulting skills.

Desirable Experience

• dbt or Dataform.
• Hadoop, Spark, Cloudera/Hortonworks and other open-source data platforms.
• HBASE or MongoDB.
• Spanner, Bigtable, Cloud SQL and database migration projects.
• Legacy code/data platform modernisation.
• Generative AI technologies and enterprise integrations.
• GCP Data Engineering certification.

What We're Looking For

We are looking for someone who is hands-on technically but also comfortable leading architecture, engaging directly with customers and guiding other engineers.

If you have strong GCP Data Engineering experience and have worked on large-scale transaction or financial data platforms, we’d like to hear from you.

Apply now with your updated CV to discuss the opportunity.

#GCP #GoogleCloud #DataEngineering #DataEngineer #LeadDataEngineer #BigQuery #Dataflow #CloudSpanner #PubSub #Kafka #Terraform #Python #Java #DataArchitecture #FinancialServices #FSI #SydneyJobs #MelbourneJobs #AustraliaJobs

Pay: $96,200.00 – $150,000.00 per year

Work Location: In person