Lead Data Platform enginner

Job Summary

We are seeking an experienced Lead Data Platform Engineer to lead the design, development, and optimization of an enterprise-scale data platform. The ideal candidate will drive technical strategy, establish engineering best practices, and collaborate with cross-functional teams to deliver scalable, secure, and high-performing data solutions across the Hadoop ecosystem and Azure, including Databricks.

Key Responsibilities

• Lead the architecture, development, and enhancement of large-scale data platforms supporting analytics, machine learning, and operational workloads.
• Design and optimize big data solutions using Hadoop ecosystem technologies including HDFS, Hive, Spark, YARN, and related components.
• Develop and manage data pipelines and transformations using Azure Data Lake Storage, Azure Data Factory, Azure Synapse, and Azure Databricks.
• Implement and enforce robust data governance, security, and data quality frameworks across all data layers.
• Partner with data engineering, analytics, product, and infrastructure teams to translate business requirements into scalable technical solutions.
• Drive performance tuning, capacity planning, and cost optimization across on-premises and cloud-based data platforms.
• Mentor and technically guide data engineers while establishing engineering standards, reusable patterns, and best practices.
• Oversee CI/CD, deployment, and monitoring for data workflows.
• Evaluate emerging technologies and contribute to long-term data platform modernization and technology strategy.

Required Qualifications

• 8+ years of experience in data engineering or data platform roles, including 3+ years in a technical lead or architect capacity.
• Strong hands-on experience with the Hadoop ecosystem, including HDFS, Hive, Spark, Oozie, Ranger, Airflow, and related technologies.
• Deep expertise in Azure Data Services, including:
• Azure Data Lake Storage
• Azure Data Factory
• Azure Synapse
• Azure Functions
• Azure Key Vault
• Advanced hands-on experience with Databricks, including:
• Apache Spark optimization
• Delta Lake
• Unity Catalog
• Strong proficiency in Python and SQL and distributed data processing frameworks.
• Experience with DevOps, CI/CD pipelines, and Infrastructure as Code, such as Terraform or ARM.
• Strong understanding of data modeling, storage formats, and data governance, including Parquet, ORC, and Delta.
• Proven ability to lead technical teams, communicate effectively, and influence architecture decisions.

Preferred Qualifications

• Experience migrating on-premises Hadoop workloads to cloud platforms, preferably Azure Databricks.
• Knowledge of real-time data processing technologies, including:
• Kafka
• Azure Event Hubs
• Spark Streaming

Pay: $55.00 - $65.00 per hour

Work Location: In person