Walk-in || Senior Databricks Engineer

Senior Databricks Engineer:

• 9-12 years of overall experience in IT, Data Engineering, and a minimum of 5 years of hands-on experience with Databricks in production environments, in cloud-based data engineering, with a strong background in designing, implementing, and supporting modern data lakehouse architectures.
• Design and build Databricks Lakehouse architecture specific to the client's context: Unity Catalog hierarchy, cluster policies, network configuration (VNET injection or Private Link), and medallion layer design justified against their security, compliance, and budget constraints
• Implement Databricks Unity Catalog governance: three-level namespace design (metastore/catalogue/schema), column-level lineage, dynamic data masking, row-level security, and Delta Sharing for cross-cloud data access
• Build AI-ready data pipelines: embedding generation workflows, document chunking and indexing pipelines, and semantic search indices using Databricks Vector Search
• Implement end-to-end monitoring: Databricks job failure alerts, data freshness SLA checks, row count anomaly detection, data quality expectation failures via DLT

Primary Skills: Databricks Lakehouse Platform — Delta Lake, Unity Catalog, Delta Live Tables, Databricks Workflows, Databricks SQL, Multi-Cloud Data Engineering — Azure (Synapse, ADF, ADLS Gen2) or AWS (S3, Glue, Redshift) or GCP (BigQuery, Dataflow);

Python — PySpark, Pandas, Boto3, Delta API ; SQL Secondary Skills: Apache Kafka / Azure Event Hubs / AWS Kinesis, Databricks Structured Streaming, Databricks MLflow, Databricks AI Assistant, Snowflake, OpenLineage / Databricks Unity Catalog lineage, Azure DevOps / GitHub Actions, Cloud Cost Management / Databricks DBU optimisation, Docker / Kubernetes, REST / GraphQL API Integration, Data Governance Frameworks, GitHub Copilot.