Lead Data Engineer

Lead the design, development, and optimization of scalable enterprise-grade data platforms on Microsoft Azure using PySpark, Azure Databricks, Delta Lake, and streaming/time-series data technologies.

Key Responsibilities

Design and implement Lakehouse architectures, lead enterprise adoption of Medallion Architecture and Data Mesh principles, develop optimized ETL/ELT pipelines, implement real-time streaming solutions, optimize Spark workloads and Azure Databricks cluster utilization, enforce data governance and quality, and lead a team while collaborating with stakeholders.

Requirements

PySpark, Azure Databricks, Delta Lake, Medallion Architecture, Spark SQL, Kafka, Azure Event Hubs, Spark Structured Streaming, Delta Live Tables, ADLS Gen2, Azure Data Factory, Azure Event Hub, Azure Functions, Azure Synapse Analytics, Azure Key Vault, Azure Monitor, Lakehouse architectures, Data Mesh, Data Platform Architecture, ETL/ELT data pipelines, Event-driven Architectures, Batch and streaming workload optimization, Partitioning Strategies, Shuffle Operations, Caching, Broadcast Joins, Adaptive Query Execution, Delta File Optimization, Spark workload optimization, Team leadership, Architecture reviews, Code reviews, Mentoring, Stakeholder collaboration