Lead Data Engineer (Databricks)

Job Title: Lead Data Engineer (Databricks)

Employment Type: Fulltime

Location: [Remote]

Job Summary

We are seeking an experienced Lead Data Engineer (Databricks) with a strong background in IT consulting and professional services to lead end-to-end cloud data enterprise initiatives. In this role, you will act as a trusted technical advisor to client stakeholders while leading the architecture, design, and execution of scalable Lakehouse platforms on AWS using Databricks.

The ideal candidate possesses exceptional client engagement abilities, deep hands-on expertise with the Databricks Medallion Architecture , Unity Catalog , and Delta Live Tables (DLT) , along with a proven track record of migrating legacy data warehouses to modern AWS data ecosystems.

Key Responsibilities

1. Solution Architecture & Technical Leadership

• Lakehouse Architecture: Architect, design, and implement scalable, enterprise-grade data platforms on AWS using the Databricks Lakehouse architecture.
• Medallion Architecture: Establish robust data architecture patterns across Bronze, Silver, and Gold layers for both real-time streaming and batch processing pipelines.
• Data Governance & Security: Implement fine-grained access control, governance, data lineage, and audit trails using Databricks Unity Catalog, AWS IAM, and KMS.
• Performance & Cost Optimization: Optimize cluster configurations, auto-scaling parameters, and Delta Lake performance techniques (Z-Ordering, Liquid Clustering, compaction).
• Ecosystem Integration: Seamlessly integrate AWS-native services (S3, AWS Glue, Lambda, EventBridge, Redshift, Athena, Secrets Manager) with Databricks.

2. Client Engagement & Strategic Advisory (Consulting)

• Stakeholder Management: Lead discovery workshops, technical assessments, and architecture reviews with C-level executives, VPs of Engineering/Data, and Enterprise Architects.
• Technical Roadmaps: Translate complex business requirements into solution blueprints, high-level architecture diagrams, technical roadmaps, and Statements of Work (SOWs).
• Legacy Modernization: Drive client migration strategies from legacy data warehouses (e.g., Teradata, Netezza, Snowflake, On-Prem Hadoop) to AWS Databricks.
• Best Practices Guidance: Educate clients on cloud data best practices, FinOps/cost optimization strategies, and modern Lakehouse paradigms.

3. Team Leadership, CI/CD & DevOps

• Technical Mentorship: Guide, direct, and mentor cross-functional teams of Data Engineers, Analytics Engineers, and DevOps Engineers.
• Infrastructure as Code (IaC): Enforce IaC best practices using Terraform or CloudFormation for automated enterprise workspace and cluster deployments.
• DevOps & Automation: Build and maintain CI/CD pipelines for Databricks notebooks, Delta Live Tables (DLT), and Spark jobs across multi-environment setups (Dev, Test, Prod).

Qualifications & Requirements

• Education: Bachelor’s or Master’s degree in Computer Science, Data Engineering, Information Technology, or a related field.
• Technical Expertise:
• 8+ years in Data Engineering with at least 3+ years in a Lead/Architect capacity.
• Deep experience with Databricks , PySpark/Scala, Delta Lake, Unity Catalog, and Delta Live Tables (DLT).
• Hands-on experience with AWS Cloud Data Ecosystem (S3, Glue, Lambda, EMR, Redshift, IAM).
• Proficiency in Python , SQL , and Spark SQL performance tuning.
• Strong experience with Infrastructure-as-Code ( Terraform ) and CI/CD tools (GitHub Actions, Azure DevOps, Jenkins).
• Consulting & Soft Skills:
• Exceptional verbal and written communication skills with the ability to articulate complex technical ideas to executive audiences.
• Strong problem-solving skills and client-facing advisory experience.