Data Engineer-Team Lead

Job Summary:

The Team Lead Data Engineer will be a foundational member of our team, responsible for designing, building, and maintaining scalable and robust data pipelines and infrastructure. This role will be critical in integrating vast and diverse datasets from across our group businesses into a unified data platform. The Senior Data Engineer will ensure data quality, accessibility, and performance, enabling our Data Analysts to derive actionable insights that drive business growth.

Key Responsibilities:

• Design, develop, and maintain robust ETL/ELT pipelines to ingest, transform, and load data from various source systems into our central data warehouse.

• Build and optimize data models within the data warehouse (e.g., BigQuery) to support analytical and reporting needs. • Ensure data quality, consistency, and reliability across all integrated datasets.

• Implement and manage data governance best practices, including data lineage, metadata management, and access controls.

• Collaborate closely with Data Analysts to understand their data requirements and optimize data structures for analytical performance.

• Manage and optimize cloud data infrastructure, ensuring scalability, cost-efficiency, and security.

• Troubleshoot data-related issues and implement solutions to ensure continuous data flow.

• Evaluate and recommend new data technologies and tools to enhance our data platform capabilities.

Qualifications:

• Bachelor's or Master's degree in Computer Science, Software Engineering, Data Engineering, or a related technical field.

• Minimum of 7-10 years + of extensive experience in data engineering, building and maintaining large scale data pipelines.

• Expert proficiency in SQL and Python for data manipulation, scripting, and pipeline development.

• Deep hands-on experience with cloud data platforms, specifically Google Cloud Platform (GCP), including BigQuery, Cloud Storage, Dataflow, Cloud Composer (Airflow), and Pub/Sub.

• Strong understanding of data warehousing concepts, data modeling (e.g., Kimball, Inmon), and schema design. • Experience with various data integration patterns (batch, streaming) and tools.

• Familiarity with data governance principles and data security best practices.

• Excellent problem-solving skills and ability to work with complex, diverse datasets.

• Experience with distributed computing frameworks (e.g., Spark) is a plus