Data Engineer

Job Description – Data Engineer (PySpark / Databricks)

Position

Data Engineer – PySpark / Databricks

Experience

4–8 Years

Location

UAE – Dubai / Abu Dhabi

Employment Type

Full-Time / Contract

Work Mode

Onsite / Hybrid – As per client requirement

Role Overview

We are looking for an experienced Data Engineer with strong expertise in PySpark, Python, SQL, Databricks, and cloud data platforms to design, develop, and maintain scalable data pipelines and data processing solutions.

The ideal candidate should have hands-on experience working with large datasets, developing ETL/ELT pipelines, implementing data transformations, optimizing Spark workloads, and building cloud-based data platforms.

Key Responsibilities

• Design, develop, and maintain scalable data pipelines and ETL/ELT workflows.
• Develop high-performance data processing solutions using PySpark and Apache Spark.
• Build data pipelines to ingest data from databases, APIs, files, applications, and other sources.
• Perform data cleansing, transformation, aggregation, and validation.
• Develop and optimize Spark SQL and PySpark jobs.
• Work with structured, semi-structured, and unstructured data.
• Implement data pipelines using Databricks and cloud-based data platforms.
• Optimize Spark jobs for performance, scalability, memory utilization, and processing time.
• Implement data quality, validation, monitoring, and error-handling mechanisms.
• Work with data lakes, lakehouses, and data warehouse environments.
• Develop reusable and scalable data engineering frameworks.
• Collaborate with Data Architects, Data Scientists, Business Analysts, and application teams.
• Troubleshoot production data pipeline issues and perform root-cause analysis.
• Participate in code reviews, testing, deployment, and production support.
• Maintain technical documentation for data pipelines and data architecture.

Mandatory Skills

• 4–8 years of experience in Data Engineering.
• Strong hands-on experience with PySpark and Apache Spark.
• Strong programming experience in Python.
• Strong proficiency in SQL.
• Experience developing ETL/ELT pipelines.
• Hands-on experience with Databricks.
• Strong understanding of Spark architecture, DataFrames, Spark SQL, transformations, and actions.
• Experience working with large-volume datasets.
• Good understanding of data warehousing and data lake concepts.
• Experience with relational and NoSQL databases.
• Knowledge of data pipeline performance tuning and optimization.
• Experience working with at least one major cloud platform – Azure, AWS, or GCP.

Cloud & Data Technologies

Candidates with experience in one or more of the following will be preferred:

Azure

• Azure Data Factory
• Azure Databricks
• Azure Data Lake Storage
• Azure Synapse
• Microsoft Fabric

AWS

• AWS Glue
• Amazon EMR
• Amazon S3
• Redshift
• Kinesis

GCP

• BigQuery
• Dataflow
• Dataproc
• Cloud Storage

Good-to-Have Skills

• Delta Lake / Lakehouse architecture
• Apache Kafka
• Apache Airflow
• Snowflake
• dbt
• Hadoop / Hive
• Data modelling
• Data governance and data quality
• Docker and Kubernetes
• CI/CD
• Git / GitHub / GitLab
• Terraform
• Experience with real-time / streaming data pipelines
• Exposure to Generative AI / ML data pipelines

Preferred Technology Stack

Programming: Python, SQL

Big Data: Apache Spark, PySpark, Kafka

Data Platform: Databricks, Delta Lake, Snowflake

Cloud: Azure / AWS / GCP

Orchestration: Airflow / Azure Data Factory

DevOps: Git, CI/CD, Docker, Kubernetes

Candidate Profile

The ideal candidate should:

• Have strong analytical and problem-solving skills.
• Be experienced in designing scalable data solutions.
• Have strong debugging and troubleshooting capabilities.
• Understand data architecture and data lifecycle concepts.
• Be comfortable working with large and complex datasets.
• Have good communication and stakeholder-management skills.
• Be able to work independently as well as collaboratively in a multicultural environment.
• Have experience working in enterprise or large-scale data environments.

Education

Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, Data Science, or a related discipline.

Desired Candidate Profile

Relevant Job Titles for Sourcing

• Data Engineer
• Senior Data Engineer
• PySpark Developer
• PySpark Data Engineer
• Databricks Data Engineer
• Big Data Engineer
• Cloud Data Engineer
• Data Platform Engineer
• ETL Developer
• Data Engineering Specialist