Data Engineer (AWS + Databricks)

Data Engineer (AWS + Databricks)

Experience: 812 years Location: Pan India(All LTM locations) Grade: P4

Role Summary

We are seeking a highly skilled Lead Data Engineer with strong expertise in AWS and Databricks. The role involves designing and developing largescale data processing solutions, building robust pipelines, and collaborating with crossfunctional teams to deliver secure, scalable, and highperformance data platforms.

Key Responsibilities

• Handson Databricks development for largescale data processing and analytics.
• Build and maintain ETL/ELT pipelines for batch and near realtime ingestion.
• Work with Apache Spark, PySpark, Spark SQL, Python, Scala, Power BI.
• Implement and optimize Delta Lake features (ACID transactions, schema evolution, performance tuning).
• Design and implement analytical and dimensional data models (Star/Snowflake).
• Leverage AWS cloud storage systems for scalable data solutions.
• Ensure adherence to coding standards, documentation, and data engineering best practices.
• Collaborate with architects, BI teams, and data science teams for endtoend delivery.
• Exposure to ML/AI workloads on Databricks, including feature engineering.
• Familiarity with MLflow or similar model tracking tools.
• Experience with Databricks CI/CD and Gitbased version control.
• Strong understanding of data governance, security, and compliance.
• Work in Agile/Scrum delivery environments.
• Domain experience in Insurance preferred.

Mandatory Skills

• AWS (cloud storage, services).
• Databricks (development, pipelines, Delta Lake).
• Apache Spark / PySpark / Spark SQL.
• SQL and Data Modelling.

Good to Have Skills

• Exposure to ML/AI workloads on Databricks.
• Familiarity with MLflow.
• Experience with Power BI.
• Domain knowledge in Insurance.