Databricks Engineer

Databricks Engineer

Experience: 7 to 12 years

Locations: Chennai, Bengaluru, Kolkata, Hyderabad, Pune

Employment Type: Full-time

We are looking for an experienced Databricks Engineer to design, develop, and optimize scalable data-engineering solutions using Databricks, Apache Spark, PySpark, Python, and SQL. The ideal candidate should have strong experience in Lakehouse architecture, Delta Lake, batch and streaming data pipelines, data governance, and cloud-based data platforms.

Key Responsibilities

• Design and implement scalable Databricks Lakehouse solutions.
• Build, maintain, and optimize Spark, PySpark, SQL, batch, and streaming data pipelines.
• Develop Delta Lake ingestion, transformation, curation, and consumption layers.
• Create reusable ingestion frameworks for structured and semi-structured data.
• Implement ETL and ELT pipelines, CDC processes, and medallion architecture.
• Configure and manage Databricks Workflows, Jobs, Databricks SQL, and Unity Catalog.
• Implement data-quality checks, reconciliation, schema management, lineage, metadata management, and automated testing.
• Optimize Spark jobs, clusters, pipeline reliability, performance, and cost.
• Automate Databricks deployments using Git, CI/CD, and Terraform.
• Monitor production pipelines, troubleshoot failures, and resolve incidents.
• Collaborate with business, architecture, analytics, data science, QA, and operations teams.
• Prepare technical documentation, operational procedures, and runbooks.

Mandatory Skills

• 7 to 12 years of experience in data engineering.
• Strong hands-on expertise in Databricks, Apache Spark, PySpark, Spark SQL, Python, and SQL.
• Experience with Delta Lake, Unity Catalog, Databricks SQL, Workflows, Jobs, and Lakehouse architecture.
• Strong understanding of medallion architecture, MERGE, time travel, and schema evolution.
• Experience building batch, streaming, ETL, ELT, CDC, ingestion, transformation, and data-curation pipelines.
• Strong knowledge of Spark performance optimization and cluster optimization.
• Experience in data modeling, data warehousing, data quality, reconciliation, lineage, metadata, and access controls.
• Experience with Git, CI/CD, Databricks deployment automation, and Terraform.
• Exposure to AWS, Azure, or GCP data services and cloud-storage platforms.
• Strong communication, analytical thinking, and problem-solving skills.

Good-to-Have Skills

• MLflow and MLOps
• DBT
• Kafka
• Airflow
• Azure Data Factory
• AWS Glue
• Snowflake
• Data Mesh
• Streaming analytics