LEAD DATA ENGINEER - Databricks

Job Summary

As a Lead Data Engineer, you will leverage Azure technologies to drive data transformation, analytics, and machine learning.

Responsibilities

• You will design scalable Databricks data pipelines using PySpark, transforming raw data into actionable insights.
• Your role includes building, deploying, and maintaining machine learning models using MLlib or TensorFlow while optimizing cloud data integration from Azure Blob Storage, Data Lake, and SQL/NoSQL sources.
• You will execute large-scale data processing using Spark Pools, fine-tuning configurations for efficiency.

Requirements

• The ideal candidate holds a Bachelors or masters in computer science, Data Science, or a related field, with 7+ years in data engineering and 5+ years specializing in Azure Databricks, PySpark, and Spark Pools.
• Proficiency in Python PySpark, Pandas, NumPy, SciPy, Spark SQL, DataFrames, RDDs, Delta Lake, Databricks Notebooks, and MLflow is required, along with hands-on experience in Azure Data Lake, Blob Storage, and Synapse Analytics.

Skills

• Azure
• Databricks
• PySpark
• Spark Pools
• MLlib
• TensorFlow
• Azure Blob Storage
• Data Lake
• Synapse Analytics

Disclaimer: This job posting has been aggregated from external source. Role details, content, and availability are subject to change. Applicants are advised to confirm the latest information directly on the company website before applying.