Databricks Data Engineer

Advaiya is a leading C&I (consulting and implementation) services company, providing business applications and analytics solutions to organizations across the world. We help businesses gain advantage by enabling digital workplace—through identifying, architecting, building, implementing, integrating, and ensuring adoption of relevant technology solutions and innovations. We are seeking enthusiastic Databricks Data Engineer to join us growing team.

About the role

We are seeking a skilled Databricks Data Engineer with 4+ years of experience in designing, building, and optimizing modern data platforms on Azure. The ideal candidate will have strong expertise in Databricks, PySpark, SQL, and cloud-based data engineering practices, and will be responsible for creating scalable, reliable, and analytics-ready data solutions.

Key responsibilities

• Design, develop and maintain scalable data pipelines using Azure Databricks.
• Build ETL/ELT solutions to ingest data from diverse sources including databases, APIs, files, SaaS platforms and streaming systems.
• Develop data transformation logic using PySpark and Spark SQL.
• Implement and maintain Medallion Architecture (Bronze, Silver, Gold).
• Design and manage Delta Lake based data solutions.
• Optimise Spark jobs, cluster utilisation and query performance.
• Collaborate with business stakeholders, analysts and reporting teams to deliver analytics-ready datasets.
• Implement data quality checks, validation frameworks and monitoring processes.
• Support data governance, security and compliance requirements.
• Participate in code reviews, deployment automation and CI/CD practices.
• Troubleshoot data issues and ensure timely resolution of production incidents.
• Contribute to solution architecture discussions and technical estimations.

Required skills

Core

• 4+ years of experience in Data Engineering.
• Strong hands-on experience with Azure Databricks.
• Strong PySpark development experience.
• Advanced SQL skills.
• Azure Data Lake Storage (ADLS Gen2).
• Delta Lake.
• ETL/ELT design and implementation.
• Data modelling concepts.

Azure ecosystem

• Azure Data Factory (ADF) or Microsoft Fabric.
• Azure Synapse Analytics (good to have).
• Azure Event Hub or streaming technologies (good to have).

Engineering practices

• Git version control.
• CI/CD implementation.
• Performance tuning and troubleshooting.
• Data quality and monitoring frameworks.

Preferred qualifications

• Databricks Certified Data Engineer Associate.
• Microsoft Azure Data Engineer certification.
• Experience with Microsoft Fabric.
• Experience with Snowflake, Power BI, or Synapse Analytics.
• Experience working in Agile delivery environments.

Soft skills

• Strong analytical and problem-solving skills.
• Effective stakeholder communication.
• Ability to work independently in client-facing environments.
• Strong ownership and accountability.
• Ability to translate business requirements into scalable technical solutions.