Senior Databricks / Machine Learning Engineer

• Support Window: 6:00 PM-10:00 AM
• Assignment: 3 Hours Daily
• Mode: Remote / Part-Time
• Focus: Databricks + Machine Learning

ROLE OVERVIEW

• We are looking for an experienced Senior Databricks / Machine Learning Engineer with 8+ years of experience for a part-time job support assignment.
• The ideal candidate must have strong hands-on experience in Azure, Azure Databricks, Apache Spark, PySpark and Machine Learning, with experience building data ingestion, processing, ML experimentation and deployment workflows.
• Candidates must be available for a 3-hour assignment within the 6:00 PM
• 10:00 AM support window.
• MUST-HAVE TECHNOLOGIES - Cloud: Microsoft Azure Data Ingestion: Azure Synapse Pipelines Storage: Azure Data Lake Storage (ADLS) Data Processing: Azure Databricks Apache Spark PySpark Data Formats: Delta Lake / Parquet Machine Learning: Python Keras Machine Learning model development Hyperparameter Tuning: Optuna Experiment Tracking: MLflow Model Registry: MLflow Model Registry Workflow Orchestration: Databricks Jobs YAML Configurations Source Control: GitHub Artifact Management: JFrog Artifactory Documentation: Confluence Operational / AI Analysis: Genie Agents Future custom-agent ecosystem Deployment: Plant Servers Supporting Systems: DTR BOPH / Equipment Metadata Plant Telemetry Systems

KEY RESPONSIBILITIES

• Develop and support Azure Databricks data and ML solutions.
• Build scalable data processing pipelines using Spark/PySpark.
• Work with Azure Synapse Pipelines for data ingestion.
• Process and manage data in ADLS using Delta/Parquet formats.
• Develop ML solutions using Python and Keras.
• Implement hyperparameter optimization using Optuna.
• Manage ML experiments and model lifecycle using MLflow.
• Maintain MLflow Model Registry.
• Configure and manage Databricks Jobs and YAML-based workflows.
• Use GitHub for source control and JFrog Artifactory for artifact management.
• Support deployment of models/solutions to plant servers.
• Work with plant telemetry, equipment metadata, DTR and BOPH systems.
• Support operational analysis involving Genie agents and future custom-agent ecosystems.
• Troubleshoot data, ML and deployment issues.
• Maintain technical documentation in Confluence.

CANDIDATE MUST HAVE

• 8+ years overall IT experience
• Strong Azure Databricks experience
• Strong Machine Learning experience
• Strong Python + PySpark + Apache Spark
• Azure Data Lake Storage experience
• Azure Synapse Pipelines experience
• Keras / ML model development experience
• Optuna / hyperparameter tuning experience
• MLflow + MLflow Model Registry experience
• Databricks Jobs + YAML experience
• GitHub experience
• Experience with Delta / Parquet
• Good understanding of ML lifecycle and deployment
• Experience working with production/operational systems
• Available for 3 hours daily
• Available within 6 PM-10 AM support window.