Data Engineer (Hybrid, BGC)

Overall Duties and Responsibilities

Does (The tasks / responsibilities that the role performs to address requirements in Key Result Areas).

1. Data Warehousing Design

• Design data warehouse and operational data marts following the Client standards.
• Prepare data model, data dictionary, and mapping documents

2. Data Pipeline design, development, testing, and support

• Design and develop data pipelines that will ingest the data from the respective source systems into the data warehouse
• Support and optimize existing data pipelines to ensure that the downstream SLAs are met
• Ensures the data engineering standards and practices are followed like code version history, readability, functionality, extendibility, test runs, deployments, and post deployment support
• Manage data storage, computing unit requirements, and data security within the data warehouse

3. Comply with Client's agile ways of working

• Responsible in attending agile ceremonies as part of the Agile Tribe set-up and delivering the different demands of the Tribe related to data ingestions to the data warehouse platform
• Gather data requirements coming from Tribe demands

4. Maintain high data quality in the data warehouse

• Ensures data quality within the data warehouse tables by implementing data validation rules and data quality checks within the data pipeline

5. Incident Management Support

• Supports L2 data operations team during production incidents by troubleshooting data pipelines and understanding data behaviors
• Supports stakeholders of the data warehouse platform by solving data related issues and conducting data investigations
• Displays (The knowledge, skills and behaviors indicating how tasks / responsibilities will be performed).
• Good understanding of data warehousing concepts and data modeling
• Proficient in SQL and optimizing queries
• Good communication and problem solving skills
• Delivers (The specific outputs / tangible results produced by the role; resources responsible for)
• Data Delivery Documentations – Data Dictionary, Data Mapping Documents, Data Models
• Optimized data pipelines
• Root cause analysis for production related issues and incidents

Qualifications (Minimum requirements in terms of experience and background, skills and competencies for sourcing purposes)

• Graduate of any IT related courses like BS Computer Science, Information Technology or the likes
• 4-6 years of experience in IT project delivery or data development delivery particularly in Data Warehousing or Business Intelligence
• Proficient in ETL development related to Data Warehousing, Business Intelligence, or other Data Delivery projects.
• Proficient in optimizing data pipelines in order to meet SLA requirements and save cost
• Proficient in SQL, Python, R, Java, Scala
• Proficient in Agile delivery methodologies like Scrum or Kanban
• Proficient in data modeling, data integrations, data quality, and data architecture is a plus
• Competent in Big Data, Hadoop, EMR, Hive, and cloud environments is a plus
• Competent in using Alibaba Cloud products particularly Dataworks, Maxcompute, OSS, Hologres Competent in Kafka, Confluent, Real-time processing, Data Streaming is a plus
• Proficient in using AWS Cloud products particularly S3, Glue, Athena, Data Sync, Lake Formation, Airflow, Secrets Manager, IAM, KMS, Cloudwatch
• Competent in using Terraform, Gitlab