Senior Data Engineer

The Senior Data Engineer will be part of the Data & Analytics Office (DAO) within the IT Services department, responsible for designing, developing, and maintaining scalable data pipelines, data models, and modern data solutions on the Microsoft Azure cloud platform.

The role involves integrating data from multiple sources, transforming raw data into high-quality datasets, and enabling data-driven decision-making across business functions. The candidate will collaborate with business stakeholders, data scientists, and reporting teams to deliver reliable, scalable, and analytics-ready data solutions.

Key Responsibilities

• Design, develop, maintain, and support scalable data pipelines and ETL/ELT workflows.
• Develop and optimize data models, database schemas, and data structures to support business intelligence and analytical requirements.
• Build and maintain data integration and orchestration workflows using Microsoft Azure data services.
• Work with data from multiple sources, including business applications, project systems, vessel operations, campaigns, and third-party data services.
• Develop efficient data processing solutions using SQL, Python, Spark, or other relevant technologies.
• Design and implement data models using ER modelling, UML, Star Schema, and Snowflake Schema techniques.
• Ensure data quality, consistency, reliability, and performance across data pipelines and datasets.
• Collaborate with business stakeholders to gather, analyze, and refine data requirements.
• Enable data scientists, analysts, and report developers to access reliable, well-structured, and analytics-ready datasets.
• Troubleshoot data pipeline failures, resolve data quality issues, and optimize data processing performance.
• Work with structured and unstructured data, including time-series and geospatial data, where applicable.
• Follow agile development practices and contribute to continuous improvement in data engineering processes and solutions.

Mandatory Technical Skills

• Microsoft Azure Data Stack: Hands-on experience with Azure Data Factory, Azure Synapse Analytics, Azure Databricks, Microsoft SQL Server, and Azure Analysis Services (AAS).
• SQL: Strong proficiency in writing, optimizing, and troubleshooting complex SQL queries.
• Data Engineering: Proven experience in developing and maintaining data pipelines, ETL/ELT processes, and data integration solutions.
• Programming: Hands-on experience with Python, Scala, Apache Spark, or other data-oriented programming technologies.
• Data Modelling: Strong understanding of entities, relationships, normalization, denormalization, and database design principles.
• Data Architecture: Experience designing database structures and analytical models using ER diagrams, UML, Star Schema, and Snowflake Schema.
• Data Orchestration: Experience implementing and managing data workflow scheduling, orchestration, and monitoring.

Good-to-Have Skills

• Experience with Machine Learning (ML), Artificial Intelligence (AI), or data science solutions.
• Familiarity with reporting and visualization tools such as Power BI, Tableau, or Grafana.
• Experience working with geospatial data, time-series data, and large or heterogeneous datasets.
• Knowledge of modern cloud data architectures, data platforms, and emerging data engineering technologies.
• Experience supporting data analytics, business intelligence, and data science teams.

Qualifications and Experience

• 6-10 years of relevant experience in data engineering, data integration, and data modelling.
• Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, or a related discipline.
• Mandatory hands-on experience with the Microsoft Azure data stack.
• Strong analytical, problem-solving, and troubleshooting skills.
• Good communication and stakeholder collaboration skills.
• Ability to work independently and collaboratively in an agile environment.
• Result-oriented approach with a willingness to learn, experiment with new technologies, and continuously improve data solutions.