Data Engineer

We are looking for a Data Engineer with 1-4 yrs years of experience to design, build, and maintain scalable data pipelines and data platforms across cloud and big data ecosystems. The candidate should have hands-on experience with Hadoop, Spark/PySpark, Snowflake, data modeling, and workflow orchestration tools.

Key Responsibilities

Design, develop, and maintain ETL/ELT data pipelines.

Work with large-scale distributed systems like Hadoop and Spark.

Develop data pipelines using PySpark and Spark SQL.

Build and optimize data warehouses in Snowflake.

Perform data modeling (Star Schema, Snowflake Schema, SCD Types).

Implement workflow orchestration using tools like Airflow.

Work with cloud platforms such as AWS / Azure / GCP.

Optimize data pipelines for performance and cost.

Ensure data quality, data validation, and monitoring.

• Work with structured and semi-structured data (JSON, Parquet, Avro, CSV).
• Collaborate with Data Analysts, Data Scientists, and Business Teams.

Required Technical Skills

Big Data & Hadoop Ecosystem

• Hadoop (HDFS, YARN, MapReduce)
• Apache Spark
• PySpark
• Hive
• HBase (optional)
• Kafka (basic knowledge)

Cloud Platforms (At least one required)

• AWS: S3, Glue, EMR, Redshift, Lambda, Athena
• Azure: ADLS, Azure Data Factory, Synapse, Databricks
• GCP: GCS, BigQuery, Dataproc, Dataflow, Composer

Data Warehousing

• Snowflake
• Redshift / BigQuery / Synapse (nice to have)

Data Modeling

• Star Schema
• Snowflake Schema
• Slowly Changing Dimensions (SCD Type 1, 2, 3)
• Fact & Dimension tables

Orchestration Tools

• Apache Airflow
• Azure Data Factory / AWS Step Functions / Prefect / Luigi

Programming

• Python
• PySpark
• SQL (Advanced SQL required)
• Shell Scripting (basic)

File Formats

• Parquet
• Avro
• ORC
• JSON
• CSV

Version Control & CI/CD

• Git
• CI/CD pipelines (GitHub Actions / Jenkins / Azure DevOps)

Preferred Skills

• Databricks
• Delta Lake
• Iceberg
• Data Lake Architecture
• Data Governance & Security
• Docker
• Kubernetes (basic)