Senior Data Engineer

Role Overview:

We are seeking an experienced Data Engineer with strong expertise in Databricks, AWS, and modern data engineering practices. The ideal candidate will have a proven track record of designing and implementing scalable data platforms, building robust ETL/ELT pipelines, and optimizing data processing workflows. Exposure to AI and Machine Learning data pipelines and MLOps is highly desirable.
You will collaborate with data scientists, analytics teams, and software engineers to deliver reliable, high-performance data solutions that support business intelligence and AI-driven initiatives.

A highly motivated Data Engineer who is passionate about building scalable cloud-native data platforms, has deep expertise in Databricks and AWS, excels in ETL development, and has practical exposure to AI/ML data engineering. Candidate thrives in a collaborative environment, enjoys solving complex data challenges, and is eager to contribute to modern analytics and AI initiatives.

Key Responsibilities:

• Design, develop, and maintain scalable data pipelines using Databricks and Apache Spark.
• Build and optimize ETL/ELT workflows to ingest, transform, and curate structured and unstructured data from multiple sources.
• Develop and maintain cloud-native data solutions on AWS.
• Implement data lake and lakehouse architectures using Delta Lake and Databricks.
• Optimize Spark jobs for performance, scalability, and cost e ciency.
• Develop data models to support analytics, reporting, and machine learning workloads.
• Build reliable orchestration workflows using tools such as Apache Airflow or AWS Step Functions.
• Ensure data quality through validation, monitoring, testing, and governance practices.
• Collaborate with data scientists to prepare feature datasets and productionize ML pipelines.
• Support AI/ML initiatives by developing feature engineering pipelines and integrating model outputs into enterprise data platforms.
• Implement CI/CD pipelines and infrastructure-as-code for data engineering deployments.
• Troubleshoot production issues and continuously improve platform reliability and performance.
• Work closely with cross-functional stakeholders to translate business requirements into scalable data solutions.

Skills & Abilities:

• Bachelor s or Master s degree in Computer Science, Information Technology, Engineering, or a related field.

• 4+ years of hands-on experience in Data Engineering.

• Strong expertise with Databricks, Apache Spark (PySpark/Scala), and Delta Lake.

• Extensive experience designing and implementing enterprise ETL/ELT pipelines.

• Strong experience with AWS services including:

• S3

• Glue

• EMR

• Lambda

• Redshift

• Athena

• IAM

• CloudWatch

• Step Functions

• Kinesis

• Strong SQL skills and experience with relational and NoSQL databases.

• Proficiency in Python and PySpark.

• Experience with data orchestration tools such as Apache Airflow or similar.

• Experience with Git, CI/CD pipelines, and DevOps best practices.

• Strong understanding of data warehousing, dimensional modeling, and modern Lakehouse architecture.

• Experience implementing data quality, lineage, and governance practices.

• Strong analytical, troubleshooting, and communication skills.

Key Skills:

• Databricks
• Apache Spark
• PySpark
• Delta Lake
• AWS
• ETL/ELT
• Python
• SQL
• Data Lakehouse
• Apache Airflow
• MLflow
• AI/ML Data Pipelines
• Data Modeling
• AWS Glue
• Redshift
• S3
• CI/CD
• Git
• Terraform
• Data Governance
• Performance Optimization

Preferred Qualifications:

• Experience supporting AI/ML workloads and data preparation for model training and inference.
• Familiarity with MLflow, Databricks Model Registry, or MLOps Practices.
• Experience with vector databases, Retrieval-Augmented Generation (RAG), or Generative AI data pipelines is a plus.
• Exposure to large language model (LLM) applications and AI data engineering workflows.
• AWS Certifications (Solutions Architect, Data Analytics, or Developer) are preferred.
• Databricks Certified Data Engineer certification is a plus.
• Experience working in Agile/Scrum environments.

Nice to Have:

• Experience with Kafka or other streaming technologies.
• Knowledge of Terraform or CloudFormation.
• Experience with Snowflake or other cloud data warehouses.
• Experience with Kubernetes and Docker.
• Familiarity with monitoring and observability tools.

Interview Process

• Assessment
• 2 Technical Rounds

Disclaimer : This job posting has been aggregated from external source. Role details, content, and availability are subject to change. Applicants are advised to confirm the latest information directly on the company website before applying.