Data Engineer

Role & responsibilities

• Design, develop, and maintain scalable data pipelines and data engineering solutions.
• Develop data ingestion, transformation, and processing workflows using cloud-based technologies.
• Work with Snowflake and Databricks to build and optimize data processing and analytical solutions.
• Develop and maintain data pipelines using AWS Glue, S3, and Lambda.
• Implement data transformation, cleansing, validation, and integration processes.
• Develop efficient SQL queries and optimize data processing and warehouse workloads.
• Work with business and technical teams to understand data requirements and translate them into technical solutions.
• Participate in solution design, development, testing, and deployment activities.
• Develop reusable components and follow established data engineering and coding standards.
• Troubleshoot data pipeline and application issues and implement appropriate fixes.
• Participate in code reviews, technical discussions, and continuous improvement initiatives.
• Support deployment and release activities across environments.
• Monitor data pipelines and proactively identify issues affecting data processing or availability.
• Provide support for production issues when required, including troubleshooting and root cause analysis.
• Maintain technical documentation and operational procedures.

Preferred candidate profile

• 3-8 years of hands-on experience in data engineering and ETL/ELT development.

• Strong understanding of data engineering concepts, ETL/ELT, and data pipelines.

• Hands-on experience with Snowflake and/or Databricks.

• Experience building data pipelines and data transformation workflows.

• Strong SQL skills.

• Good understanding of data modelling and data warehousing concepts.

• Good experience with cloud-based data platforms and data pipelines.

• Hands-on experience with one or more of the following AWS services:

• AWS Glue

• Amazon S3

• AWS Lambda

• Experience with Databricks data engineering, including:

• Data transformation and processing

• Delta Lake

• PySpark and/or Spark

• Notebook-based development

• Experience with Snowflake development, including:

• Data loading and transformation

• SQL development

• Data warehouse concepts

• Performance optimization

• Experience with any of the following preferred/optional skills will be an advantage:

• ODI (Oracle Data Integrator)

• OAC (Oracle Analytics Cloud)

• PL/SQL

• Amazon MSK / Kafka

• Datadog

• Python

• PySpark

• CI/CD and DevOps practices

• Git

• Cloud monitoring and logging

• Oracle databases

• Bachelor's degree in Computer Science, Information Technology, Engineering, or a related discipline.

• Strong data engineering and analytical skills.

• Good understanding of cloud data platforms.

• Strong SQL and data transformation capabilities.

• Strong problem-solving and troubleshooting skills.

• Good communication and collaboration skills.

• Ability to work independently as well as part of a team.

• Strong focus on data quality and solution reliability.

• Ability to learn and work across new technologies and platforms.