Data Engineer

THIS POSITION IS TO BE BASED IN MUTIARA DAMANSARA, PETALING JAYA

ROLE AND RESPONSIBILITIES

A.  DATA MANAGEMENT & TECHNICAL DEVELOPMENT

• Supports the end-to-end management of MS SQL Server, PostgreSQL and DuckDB (DuckLake) database systems, ensuring their stability, scalability and reliability.
• Monitor daily job statuses and perform troubleshoot and recovery when necessary.
• Develop scalable and low latency end-to-end data pipelines in accordance to best practices.
• Write, optimize and troubleshoot T-SQL queries, stored procedures, views, functions and triggers to support data transformation, loading, and validation logic.
• Orchestrate and schedule complex data workflows, ensuring timely and reliable data delivery.
• Implement and manage monitoring solutions to proactively address and prevent performance issues.
• Develop and maintain staging structures and data models (star and snowflake schemas), ensuring data is structured for efficient querying and reliable analytical consumption.
• Contribute to data quality validation, reconciliation checks, and exception handling across data pipelines, ensuring completeness, accuracy, and consistency of data.
• Document pipeline logic, data lineage, transformation rules, table definitions and data dictionary entries, ensuring technical artefacts are kept current and accessible to the team.

B.  OPTIMIZATION & PLATFORM TRANSFORMATION

• Basic understanding and assessment of SSIS data pipelines to refactor SQL scripts and migrate them to the new data lakehouse environment.
• Participate in discussions and debates on technical and architectural topics that are related to the data warehouse.
• Participate in peer reviews and knowledge-sharing sessions, contributes to a culture of continuous improvement and cross-training across both legacy and modern data tooling within the team.
• Identify opportunities to optimize data processes, reduce complexities and costs.
• Contribute to the establishment of a data governance framework. Support in driving data standards and strategy for maintaining a proper and traceable governance model for all end-to-end data assets.

C.  BUSINESS PARTNERSHIP & PROJECT MANAGEMENT

• Collaborate with stakeholders to understand business requirements and translate them into technical solutions.
• Assess, evaluate and prioritize new and change requests to ensure they are evaluated for strategic fit and feasibility before assigning them to the development pipeline.
• Provide clear and timely communication on project status, risks and progress, keeping business stakeholders and ITBS management well-informed throughout the delivery lifecycle.

Job Success Requirements

• Minimum of 1 years' experience in business solutioning, data / analytics consulting or a data warehouse delivery role, with a track record of partnering directly with business stakeholders.
• Understanding of data warehousing concepts, including dimensional modelling (star & snowflake schemas), slowly changing dimensions (SCD), and staging/ODS design patterns, with the ability to apply these principles to schema design and pipeline architecture decisions.
• Proficiency in MS SQL Server, including database design, T-SQL development (stored procedures, views, functions, triggers), query optimisation and performance tuning.
• Detail-oriented and methodical, with a structured approach to documentation, data quality validation, and issue escalation, combined with the ability to manage multiple concurrent deliverables and work effectively in a fast-paced, cross-functional team environment.

Technology Stack We Use

• Language: SQL (mandatory), PowerShell, Python (bonus)
• Orchestration: SSIS, ADF
• Transformation: SQL, SSIS
• Storage: SQL Server, ADLS2
• Modelling: SSAS
• Visualization: Excel, Power BI

Qualification

• Bachelor's degree in Information Technology, Computer Science, Data Analytics, Engineering or a related STEM field.

Additional Notes

• Maintains awareness of current developments in data and AI technologies, and able to evaluate their potential application within the organisation.