Lead Databricks Engineer (Databricks)

Job Title: Lead Databricks Engineer (Databricks)

KPI Partners is seeking a skilled and experienced Lead Databricks Engineer to join our dynamic team. The ideal candidate will have a strong background in Databricks, Pyspark, CI/CD practices, and MLOps. This role will be pivotal in developing data solutions that drive business insights and enhance operational efficiency.

Key Responsibilities:

• Lead the design and implementation of Databricks-based data solutions.
• Develop, test, and deploy data processing pipelines using Pyspark.
• Collaborate with data scientists and analysts to implement MLOps practices for deploying machine learning models.
• Establish and maintain CI/CD pipelines to automate the deployment process and ensure high-quality code delivery.
• Optimize data workflows and improve system performance within the Databricks environment.
• Mentor and guide junior engineers and team members in best practices and emerging technologies.
• Work closely with stakeholders to understand business requirements and translate them into technical specifications.

Required Skills:

• Proficient in Databricks and its ecosystem.
• Strong experience with Pyspark for data processing and analysis.
• Solid understanding of CI/CD practices and tools.
• Familiarity with MLOps concepts and their application in real-world scenarios.
• Excellent problem-solving skills and attention to detail.
• Ability to work collaboratively in a fast-paced, team-oriented environment.

Qualifications:

• Bachelor’s degree in Computer Science, Engineering, or a related field.
• 5+ years of experience in data engineering or a related role, with at least 2 years of experience specifically in Databricks.
• Proven experience in leveraging Pyspark for data processing.
• Experience implementing CI/CD practices in data projects.
• Knowledge of machine learning concepts and MLOps practices is a plus.

If you are passionate about data engineering and looking to take your career to the next level with KPI Partners, we encourage you to apply for the Lead Databricks Engineer position. Join us in driving innovation through data!