Lead Data Engineer

Lead Data Engineer

Location: Noida, UP, India

Job Description

• Must Have - Python & Advanced SQL
• Spark / distributed processing / PySpark
• Cloud platforms (AWS) Modules like Lambda, Glue, EMR, Dynamo DB, S3, Redshift, EC2, RDS, DMS, Step Functions, Cloud Watch, Systems Manager, Event Bridge, DMS, Boto3 APIs
• Data pipeline automation
• Data governance & quality. Preferred if worked on AWS Data Zone.
• Any orchestrator like Airflow, Step Functions
• Knowledge of any git repository like GitHub, GitLab, Bitbucket.
• NICE TO HAVE - Kafka/Kinesis streaming
• Basic knowledge of AI
• Power BI
• Postgre SQL
• DBT
• Experience with CI/CD implementation using Terraform or TypeScript.
• Job Summary - We are seeking a Lead Data Engineer who is able to develop and maintain ETL Pipelines preferably on AWS cloud. Should be able to lead a project that includes working with client and come up with suggestions as and when required. Oversee data pipelines, data lakes, data warehouses, and ETL/ELT processes. Establish data quality standards, monitor system health and drive continuous improvements. Should be proficient in Python, advanced SQL and PySpark. Sound understanding of Spark architecture. Basic understanding of AI is essential. Knowledge of DBT, Postgre database and Power BI would be an added advantage.

Key Responsibilities

• Hands on in Python and advanced SQL.
• Data modelling with good understanding of dimensions and facts.
• Good understanding of Spark eco-system and PySpark programming.
• Able to convert existing SSIS workflows to ETL pipelines running on AWS. That includes knowledge of Lambda, Glue or EMR, Systems Manager, Event Bridge, EC2, DMS
• Knowledge of Dynamo DB, Redshift, RDS (Preferably Postgre)
• Sound knowledge of AWS Boto3 APIs
• Awareness of reporting tool like Power BI.
• Monitor and troubleshoot migration processes, resolving issues in a timely manner.
• Work closely with business stakeholders, developers, and administrators to understand data requirements.
• Good understanding of a Data Lakehouse
• Understanding of CI/CD process with exposure to TypeScript and any Git repository like GitHub, GitLab or Bitbucket
• Document migration procedures, processes, and best practices.

Required Skills and Qualifications

• Bachelors degree in Computer Science, Information Technology, or related field.
• 8+ years of experience in data migration or similar roles.
• Strong proficiency in SSIS.
• Proven experience in ETL tools like SSIS and data migration pipelines build on AWS.
• Familiarity with data modeling and schema design principles.
• Ability to analyze and resolve complex data-related issues.
• Strong analytical and problem-solving abilities.
• Excellent communication and collaboration skills.
• Ability to work independently and in a team-oriented environment.

Preferred

Familiarity with Agile project management methodologies.

Mandatory Competencies

• Data Science and Machine Learning - Data Science and Machine Learning - Python
• Data Science and Machine Learning - Data Science and Machine Learning - Apache Spark
• Data & AI - Data Engineering - Data Quality & Validation
• Big Data - Big Data - Pyspark
• Database - Database Programming - SQL
• Data Science and Machine Learning - Data Science and Machine Learning - Databricks
• Beh - Communication and collaboration