Data Engineer

Data Engineer

Department: Software & Data Services (SDS)

Role Synopsis

The Data Engineer will be responsible for developing and maintaining our data infrastructure, as well as developing and implementing data pipelines to move data from our various sources into our data warehouse, leveraging Microsoft Fabric and related technologies. The Data Engineer will work closely with the other Data Engineers, Power BI Developers, and to help deliver data projects and BAU tasks.

Job Summary

We are looking for an experienced Data Engineer with strong hands-on expertise in Informatica, ETL, Data Warehousing, SQL, Python, and Azure Data Factory (ADF).

The Data Engineer will be responsible for designing, developing, and maintaining scalable data integration solutions and data pipelines. The role involves working with data from multiple sources, implementing data quality processes, supporting data warehouse solutions, and collaborating with Data Engineers, BI Developers, and business stakeholders to deliver reliable and high-quality data solutions.

Key Responsibilities

• Design, develop, and maintain robust ETL/data integration pipelines using Informatica.
• Develop and maintain data warehouse and data lake solutions.
• Design and implement data models to support analytical and reporting requirements.
• Develop, schedule, monitor, and troubleshoot Informatica workflows and ETL processes.
• Develop complex SQL queries, stored procedures, transformations, and data validation processes using SQL/MSSQL.
• Work with Azure Data Factory (ADF) for data integration and pipeline orchestration.
• Develop data processing solutions using Python and PySpark where required.
• Implement data quality, validation, reconciliation, and data cleansing processes.
• Identify and resolve data pipeline failures, performance issues, and data quality problems.
• Automate repetitive data processing and operational tasks.
• Work closely with Power BI Developers to provide reliable and optimized datasets for reporting and analytics.
• Collaborate with stakeholders to understand data requirements and translate them into scalable technical solutions.
• Support production data pipelines, troubleshoot issues, and ensure timely resolution of BAU activities.
• Follow best practices for data security, performance, scalability, and maintainability.
• Participate in deployment and CI/CD activities for data engineering solutions.

Required Technical Skills

Mandatory

• 8+ years of experience in Data Engineering
• Strong hands-on experience with Informatica
• Strong understanding of ETL and Data Integration
• Good experience in Data Warehousing and Data Modelling
• Strong proficiency in SQL / Microsoft SQL Server (MSSQL)
• Hands-on experience with Python
• Experience with Data Quality, Data Validation, and Data Reconciliation
• Experience with Azure Data Factory (ADF)
• Good understanding of data pipelines and ETL workflows
• Strong problem-solving and analytical skills

Good to Have

• Experience with PySpark
• Knowledge of pandas, NumPy, and Scikit-Learn
• Experience with Delta Lake
• Exposure to Azure data services
• Experience working with Power BI
• Exposure to CI/CD and DevOps practices for data engineering
• Knowledge of modern data lake/lakehouse architecture
• Exposure to Microsoft Fabric / Azure Synapse is an advantageRole & responsibilities

Preferred candidate profile

Perks and benefits