Lead Data Engineer

GENERAL RESPONSIBILITIES:

The Lead Data Engineer is responsible for designing, developing, and optimizing data infrastructure to ensure reliable pipelines, scalable architecture, and high-quality data for analytics and business needs. The role includes leading a team of data engineers, collaborating with cross-functional stakeholders, and implementing best practices in data transformation, validation, governance, and cloud-based solutions.

DUTIES AND RESPONSIBILITIES:

• Lead and mentor a team of data engineers, fostering collaboration, innovation, and continuous improvement.
• Define and enforce data engineering standards, coding practices, and architectural guidelines.
• Partner with data scientists, analysts, and business stakeholders to translate requirements into scalable solutions.
• Design and maintain high-performance ETL/ELT pipelines and workflows.
• Build and optimize data lake, data warehouse, and streaming data solutions.
• Ensure systems support structured, semi-structured, and unstructured data sources.
• Develop data transformation and preparation processes to deliver clean, reliable, analytics-ready datasets.
• Implement validation, anomaly detection, and automated quality checks to ensure accuracy and consistency.
• Define and execute data remediation strategies to resolve issues with minimal business disruption.
• Implement and manage cloud-based data platforms (AWS, Azure, or GCP).
• Apply modern frameworks (e.g., Spark, Kafka, Airflow, DBT) for efficient data processing.
• Oversee data modeling, schema design, and database optimization (SQL/NoSQL).
• Ensure accuracy, consistency, and availability of data across platforms.
• Implement governance, lineage, validation frameworks, and security best practices.
• Monitor, troubleshoot, and optimize pipelines to meet SLAs and performance benchmarks.
• Stay current with emerging data engineering and analytics technologies.
• Recommend improvements to systems, tools, and processes to enhance scalability, reliability, and quality.
• Other job-related activities that may be assigned from time to time.

JOB SPECIFICATIONS:

• Education

  –    At least graduate with a Bachelor’s or Master's Degree in IT, Computer Science, Engineering, or any related course.

• Related Work Experience –

• 7+ years of experience in data engineering, with at least 2–3 years in a leadership or senior technical role.

• Proven expertise in building and managing enterprise-level data pipelines and architectures.

• Strong experience with one or more cloud platforms (AWS, Azure, GCP).

• Knowledge

  –   Knowledgeable in the following:

• Proficiency in SQL and at least one programming language (Python, Java, Scala).

• Hands-on experience with big data tools (Spark, Hadoop, Kafka).

• Strong knowledge of data transformation, preparation, validation, and remediation techniques.

• Familiarity with data warehousing solutions (Snowflake, Redshift, BigQuery, Synapse).

• Experience with workflow orchestration (Airflow, Prefect, Luigi).

• Solid understanding of DevOps and CI/CD for data pipelines.

• Skills

• Excellent communication and interpersonal skills to collaborate effectively with developers, analysts, and business stakeholders.

• Strong ability to document data requirements and translate them into technical solutions.

• Proven leadership and mentoring capabilities to guide and develop team members.

• Ability to work seamlessly across technical and non-technical teams.

• Strong problem-solving mindset with focus on detail, data quality, and scalability.

• Ability to thrive in a fast-paced environment and meet tight SLAs.