Associate-Data Engineer

Job purpose

We are seeking a Data Engineer for a multi-year strategic transformation program modernizing the global middle- and back-office technology platform through a cloud-native, event-driven architecture. The platform leverages AWS, Azure, Kafka-based event streaming, Databricks, modern workflow orchestration, API-led integration and scalable cloud infrastructure. We are seeking a highly motivated Analyst-level Data Engineer to support the design, development and implementation of modern cloud-based data platforms. The candidate will work closely with business stakeholders, solution and data architects, data engineers from other teams at SMBC, and application teams to build scalable data pipelines and streaming solutions that support strategic transformation initiatives. The ideal candidate will have hands-on experience with Databricks, cloud technologies (AWS and Azure) and real-time data processing frameworks; experience working with Apache Kafka, AWS MSK or Confluent Kafka is highly desirable.

Key responsibilities

Develop, maintain and optimize data pipelines using Databricks and cloud-native technologies.

Design and implement batch and real-time data ingestion frameworks.

Build and support streaming data solutions using Databricks Structured Streaming.

Integrate data sources with event-streaming platforms such as AWS MSK (Managed Streaming for Kafka) and Confluent Kafka.

Develop ETL/ELT processes to transform and load data into enterprise data platforms.

Participate in data modeling, data-quality validation and metadata management activities.

Collaborate with business, technology and architecture teams to understand data requirements and translate them into scalable solutions.

Support cloud migration and modernization initiatives across AWS and Azure environments.

Monitor, troubleshoot and optimize data pipelines for performance, reliability and scalability.

Participate in code reviews, testing, deployment and production-support activities.

Contribute to establishing engineering standards, best practices and reusable frameworks.

Key competencies

Bachelor's degree in Computer Science, Engineering, Information Systems or a related field.

25 years of experience in Data Engineering or related technical roles.

Hands-on experience with Databricks development and administration.

Experience developing data pipelines using PySpark and Spark SQL.

Knowledge of cloud platforms including AWS and/or Microsoft Azure.

Experience working with data lakes, distributed computing and large-scale data processing.

Understanding of real-time data streaming concepts and architectures, with familiarity with

Databricks Structured Streaming.

Working knowledge of Apache Kafka, AWS MSK or Confluent Kafka.

Experience with source-control systems such as Git, and strong SQL skills.

Strong analytical, problem-solving and communication skills.

Preferred qualifications

Experience with Delta Lake architecture and Lakehouse implementations.

Knowledge of event-driven architectures and messaging patterns.

Experience with CI/CD pipelines and DevOps practices.

Familiarity with Terraform, CloudFormation or other Infrastructure-as-Code frameworks.

Experience with financial services, capital markets or regulatory reporting platforms.

Understanding of trade lifecycle concepts across FX, Rates, Derivatives and Securities.

Exposure to enterprise integration technologies and API-based architectures.

Additional preferred tooling: Airflow, Docker, Kubernetes, Python, REST APIs and data modeling.

Disclaimer: This job posting has been aggregated from external source. Role details, content, and availability are subject to change. Applicants are advised to confirm the latest information directly on the company website before applying.