Senior Databricks Engineer

Job Summary

We are seeking a highly experienced Senior Databricks Engineer to design, build, and optimize large-scale data engineering solutions on the Databricks Lakehouse Platform. The ideal candidate will have deep expertise in big data processing, cloud architecture, and end-to-end data pipeline development, along with strong leadership skills to guide technical teams.

Key Responsibilities

• Design, develop, and optimize scalable data pipelines and ETL/ELT workflows using Databricks (PySpark, Spark SQL, Delta Lake)
• Architect and implement Lakehouse solutions integrating data lakes and data warehouses Lead the migration of legacy data platforms to Databricks on Azure/AWS/GCP
• Optimize Spark jobs for performance, cost, and scalability
• Implement data governance, security, and access control using Unity Catalog
• Collaborate with data scientists, analysts, and business stakeholders to understand data requirements
• Design and manage CI/CD pipelines for Databricks notebooks and jobs
• Mentor junior engineers and provide technical leadership on best practices
• Troubleshoot performance bottlenecks and ensure high availability of data pipelines
• Work with structured, semi-structured, and unstructured data at scale

Required Skills & Qualifications

• 8–10 years of overall experience in data engineering/big data, with strong hands-on expertise in Databricks
• Proficiency in PySpark, Spark SQL, and Delta Lake
• Strong experience with at least one major cloud platform (Azure Databricks, AWS Databricks, or GCP)
• Solid understanding of data warehousing concepts, data modeling, and ETL/ELT design patterns
• Experience with Unity Catalog, Delta Live Tables, and Databricks Workflows
• Proficiency in Python and SQL
• Knowledge of Azure Data Factory (ADF) and Medallion Architecture
• Experience with orchestration tools (Airflow, ADF, or similar)
• Familiarity with CI/CD tools (Git, Azure DevOps, Jenkins)
• Strong understanding of data governance, security, and compliance frameworks
• Excellent problem-solving, communication, and stakeholder management skills

Good to Have

• Databricks Certified Data Engineer Professional/Associate certification
• Experience with MLflow and machine learning pipelines
• Knowledge of Kafka or other streaming technologies
• Experience working in the US healthcare industry is a plus

Education

• Bachelor's/Master's degree in Computer Science, Information Technology, or a related field

Pay: ₹1,500,000.00 - ₹1,800,000.00 per year

Work Location: Remote