Lead Data Engineer

We're looking for a Lead Data Engineer to construct an MLOps platform that supports automated model delivery, deployment, and large-scale feature engineering. In this role, you'll advance the featureOps stack by leveraging Azure Databricks, Azure Data Factory, and Azure DevOps to implement CI/CD pipelines feeding into the Databricks Feature Store for Model Lab applications.

Responsibilities

• Transition existing Spark feature engineering pipelines over to Azure
• Work alongside the team to design and develop a forecasting model
• Support the deployment of the machine learning model into a production environment
• Contribute to designing and building the CI/CD pipeline that supplies the Databricks Feature Store
• Work in close coordination with Data Scientists and engineers to align feature engineering with ML model requirements
• Ensure data pipelines remain reliable and scalable across the entire platform
• Troubleshoot and enhance current Spark-based workflows during the migration process

Requirements

• A minimum of 5 years of practical PySpark experience
• Strong foundational knowledge in core data science principles
• Demonstrated success in building feature engineering and machine learning pipelines
• Hands-on background in moving data pipelines into cloud-based environments
• B2 (Upper-Intermediate) English proficiency or above

Nice to have

• Familiarity with MLflow and Docker
• Working knowledge of Azure Kubernetes Services, Azure Data Factory, and Azure Databricks
• Understanding of best practices for deploying ML models

We offer

• International projects with top brands
• Work with global teams of highly skilled, diverse peers
• Healthcare benefits
• Employee financial programs
• Paid time off and sick leave
• Upskilling, reskilling and certification courses
• Unlimited access to the LinkedIn Learning library and 22,000+ courses
• Global career opportunities
• Volunteer and community involvement opportunities
• EPAM Employee Groups
• Award-winning culture recognized by Glassdoor, Newsweek and LinkedIn

EPAM is an Equal Opportunity Employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, national origin, age, sexual orientation, gender identity or expression, disability, protected veteran status, or any other characteristic protected by applicable law.