Data Engineer (AWS, Databricks, PySpark)

We are seeking an experienced Data Engineer with strong expertise in AWS, Databricks, Python, PySpark, and SQL to build and optimize scalable data platforms and modern data pipelines. The ideal candidate will have hands-on experience in cloud-native data engineering, ETL/ELT development, and large-scale data processing across AWS environments.

Key Responsibilities

• Design, develop, and maintain scalable data ingestion, transformation, and integration pipelines.
• Build and optimize ETL/ELT workflows using Python, PySpark, SQL, and Databricks.
• Develop cloud-native data solutions on AWS platforms.
• Create and maintain data pipelines for structured and unstructured data processing.
• Work with AWS services including Glue, Redshift, S3, Lambda, EMR, and Databricks.
• Collaborate with architects, analysts, and business stakeholders to design data models and integration frameworks.
• Implement data quality checks, monitoring, and governance controls.
• Develop reusable frameworks and components for orchestration, automation, and monitoring.
• Optimize Spark workloads and data processing performance.
• Troubleshoot data pipeline issues and ensure reliability, scalability, and security.
• Support Agile delivery and contribute to continuous improvement initiatives.

Required Skills

• 6-10 years of experience in Data Engineering.

• Strong expertise in Python programming.

• Hands-on experience with PySpark and Apache Spark.

• Strong SQL development and optimization skills.

• Experience with Databricks platform.

• Experience building ETL/ELT pipelines.

• Strong AWS experience including:

• AWS Glue

• Amazon Redshift

• Amazon S3

• AWS Lambda

• Amazon EMR

• Knowledge of data modelling and data integration concepts.

• Experience working with large-scale data platforms and cloud-native architectures.

Preferred Skills

• Data Lake and Data Warehouse implementations.
• Workflow orchestration tools.
• Real-time data processing experience.
• Data Governance and Data Quality frameworks.
• CI/CD and DevOps practices for Data Engineering.

Desired Candidate Profile

• Strong analytical and problem-solving skills.
• Excellent communication and stakeholder management capabilities.
• Ability to work independently and within Agile teams.
• Passion for building scalable, high-performance data solutions.
• Strong focus on automation, reliability, and continuous improvement.