Data Engineer

We are looking for a skilled and motivated Data Engineer to join our team supporting Commonwealth Bank of Australia (CBA) . The ideal candidate will have hands-on experience in designing, developing, and optimizing scalable data pipelines using modern big data technologies and cloud platforms. You will work closely with cross-functional teams to build reliable data solutions that support business intelligence, analytics, and regulatory reporting.

Key Responsibilities

• Design, develop, and maintain scalable ETL/ELT data pipelines.

• Develop and optimize Spark and Scala applications for large-scale data processing.

• Build and manage data solutions using Hadoop technologies including HDFS and Hive.

• Design, implement, and optimize Snowflake data warehouse solutions.

• Write high-performance SQL queries and perform query optimization.

• Develop cloud-native data pipelines using AWS services such as S3, Glue, Athena, and Redshift.

• Perform data validation, quality checks, and troubleshooting of production issues.

• Collaborate with business analysts, architects, and development teams to understand data requirements.

• Manage source code using GitHub and automate deployments through GitHub Actions and TeamCity.

• Create and maintain Shell scripts for automation and operational support.

• Participate in release management, production support, and continuous improvement initiatives.

• Follow banking security, governance, and compliance standards.

  Required Skills

• Strong experience with Spark and Scala

• Hands-on experience with Hadoop (HDFS, Hive)

• Experience working with Snowflake

• Advanced SQL skills including performance tuning and query optimization

• Experience with AWS services:

  • Amazon S3
  • AWS Glue
  • Amazon Athena
  • Amazon Redshift

• Experience in ETL/Data Pipeline development

• Version control using GitHub

• CI/CD using GitHub Actions and TeamCity

• Shell Scripting

• Experience working in the Banking domain

  Preferred Skills

• Experience in Financial Crime, AML, or Fraud Analytics

• AWS Certification

• Oracle Database

• Scheduling tools such as AutoSys or Apache Airflow

• Production Support and Release Management experience

  Required Experience

• 35 years of experience as a Data Engineer or Big Data Engineer.

• Strong understanding of data engineering best practices and cloud-based data platforms.

• Excellent analytical, troubleshooting, and communication skills.

Disclaimer: This job posting & Location has been aggregated from external source. Role details, content, and availability are subject to change. Applicants are advised to confirm the latest information directly on the company website before applying.