Data Engineer

JOB ID:QT-AJH-08-604

Job Summary

We are looking for an experienced Data Engineer with strong expertise in building scalable data pipelines, lakehouse architectures, and data transformation workflows. The ideal candidate must have hands-on experience with Apache Airflow, Data Lake, SQL, Apache Iceberg, dbt, and PySpark.

Mandatory Skills

• Apache Airflow
• Data Lake / Data Lakehouse
• Advanced SQL
• Apache Iceberg
• dbt (Data Build Tool)
• PySpark

Key Responsibilities

• Design, develop, and maintain scalable ETL/ELT data pipelines using Apache Airflow and PySpark.
• Build and optimize data processing workflows on Data Lake/Lakehouse platforms.
• Develop and manage Apache Iceberg tables, including partitioning, schema evolution, and data optimization.
• Write complex and optimized SQL queries for data transformation and analytics.
• Develop modular, reusable data transformation models using dbt.
• Create and maintain Airflow DAGs with proper scheduling, dependency management, retries, monitoring, and alerting.
• Implement data quality, validation, and reconciliation checks across pipelines.
• Optimize PySpark jobs for performance, scalability, and efficient resource utilization.
• Work with large-scale structured and semi-structured datasets.
• Implement incremental data loading and CDC-based processing where required.
• Collaborate with Data Scientists, Analysts, Architects, and Application teams to understand data requirements.
• Troubleshoot pipeline failures, performance issues, and data inconsistencies.
• Follow best practices for data security, governance, version control, and CI/CD.

Technical Requirements

• Strong hands-on experience with Apache Airflow and workflow orchestration.
• Excellent knowledge of SQL, including complex joins, CTEs, window functions, and query optimization.
• Strong experience with PySpark and distributed data processing.
• Practical experience working with Apache Iceberg and lakehouse data architectures.
• Strong experience with dbt, including models, tests, macros, snapshots, and incremental models.
• Good understanding of Data Lake concepts, partitioning, file formats, and data lifecycle management.
• Experience with Git and CI/CD practices.
• Understanding of data modeling, ETL/ELT patterns, and data quality frameworks.

Good to Have

• Experience with AWS, Azure, or GCP data platforms.
• Experience with cloud storage such as S3, ADLS, or GCS.
• Knowledge of Kafka or other streaming technologies.
• Experience with Databricks or similar lakehouse platforms.
• Knowledge of Delta Lake or other table formats.
• Experience with Docker/Kubernetes.
• Familiarity with data governance and metadata management.

Preferred Candidate

• Strong problem-solving and analytical skills.
• Ability to design scalable and reliable data solutions.
• Good communication and collaboration skills.
• Experience working in Agile development environments.

Disclaimer: This job description has been sourced from a public domain and may have been modified by Naukri.com to improve clarity for our users. We encourage job seekers to verify all details directly with the employer via their official channels before applying.