Senior Databricks Engineer

Role & responsibilities

Design, implement, and manage data pipelines on Databricks (PySpark, SQL, Delta Lake) hosted on AWS.
 Develop ETL/ELT workflows for structured and unstructured data using Databricks, AWS Glue, Lambda, and Step Functions.
Build and optimize data lakes (S3 + Delta Lake) and data warehouses (Redshift, Databricks SQL).
Enable data democratization via Notebooks, APIs, and self-service analytics.
Implement data governance, quality, and lineage frameworks using AWS Lake Formation, Databricks Unity Catalog, or similar tools.
Collaborate with Data Scientists and Analysts to deliver ML-ready datasets and advanced analytics.
Tune clusters, jobs, and queries for performance and cost-efficiency.
Manage CI/CD pipelines for data engineering workloads (Terraform, GitHub/GitLab CI/CD, Databricks CLI).
Ensure security, compliance, and best practices in data handling, access controls, and monitoring.
Lead troubleshooting and incident management for large-scale data pipelines.

Preferred candidate profile