Lead Data Engineer (AWS)

Technology

DGS India - Pune - Indiqube Orchid

About the job

Business Title

Technical Lead

Years of Experience

Min 6 and max upto 10.

Job Description

Looking for a hands‑on AWS Technical Lead – Data Engineering with 6 to 10 years of total experience to design and deliver scalable, secure, and high‑performance data platforms on AWS.

This role focuses on strong individual contribution with technical ownership, working closely with global teams, architects, and clients to deliver enterprise‑grade data engineering solutions. The position requires deep expertise in AWS data services, SQL, and Python, and the ability to build and optimize reliable data pipelines for analytics and business use cases.

Key responsibiltes

Data Platform Design & Development

Design and implement AWS‑based data engineering solutions aligned to enterprise standards

Build and optimize batch and streaming data pipelines using AWS native and open‑source tools

Develop SQL‑driven transformations and Python‑based data pipelines for analytics use cases

Design efficient data models for performance, scalability, and cost effectiveness

Delivery & Quality Ownership

Own data engineering deliverables from development through production support

Perform performance tuning, cost optimization, and capacity planning

Troubleshoot complex data pipeline and production issues, including root‑cause analysis

Ensure solutions meet requirements for security, reliability, and scalability

Collaboration & Client Engagement

Work closely with architects, product owners, and client stakeholders

Translate business and analytics requirements into robust AWS data engineering solutions

Provide clear technical inputs, estimates, and implementation trade‑offs

Contribute to solution discussions and technical design reviews

Engineering Best Practices

Follow and contribute to coding standards, documentation, and data engineering best practices

Participate in code reviews and continuous improvement initiatives

Ensure adherence to AWS, security, and compliance guidelines

Must have skills

Cloud & Data Engineering (AWS)

Strong hands‑on experience with AWS data services, including:

Amazon S3, AWS Glue, Athena, Redshift

Experience designing cloud‑native data lakes and data warehouse architectures on AWS

Deep understanding of batch and streaming data pipelines

Experience building scalable, fault‑tolerant data ingestion and transformation workflows

SQL & Python (Mandatory)

Strong SQL expertise

Writing complex SQL for transformations, aggregations, performance tuning, and analytics

Hands‑on experience handling large‑scale datasets in Redshift / Athena

Strong Python programming skills (mandatory) for data engineering use cases

PySpark / Spark‑based processing

Building reusable ETL components, utilities, and data pipelines

Strong understanding of data modeling, transformations, and performance optimization

Data Processing & Engineering

Proven hands‑on experience with distributed processing frameworks such as Spark / PySpark

Experience working with structured, semi‑structured, and unstructured data

Solid understanding of schema design, partitioning, and query optimization

DevOps & Platform Engineering

Experience with Infrastructure as Code using Terraform and/or CloudFormation

Hands‑on experience building and maintaining CI/CD pipelines for data platforms

Exposure to containerized workloads (Docker, ECS/EKS where applicable to data workloads)

Collaboration & Ownership

Strong ownership mindset for solution quality, performance, and production stability

Excellent communication skills to collaborate with architects, DevOps, QA, and business stakeholders

Good to have skills

Experience with real‑time/streaming technologies (Kinesis, Kafka, MSK)

Exposure to Lakehouse architectures and modern data platform patterns

Experience integrating AWS data platforms with BI and analytics tools

Knowledge of data governance, data quality, and metadata management

Familiarity with FinOps practices for optimizing AWS data platform costs

Exposure to marketing, customer, or analytics data domains (CDP / MarTech)

Experience working in Agile delivery models with global delivery exposure

Education Qulification

1. Bachelor’s or Master’s degree in Computer Science, Information Systems, Data Engineering, or a related field.

Certification If Any

AWS Data Analytics / Solutions Architect

Any two of the above

Databricks, Snowflake, or other cloud data platform certifications are a plus.

Shift timing

12 PM to 9 PM and / or 2 PM to 11 PM - IST time zone