Senior Databricks and AI Engineer

Job Summary:

We are seeking a highly experienced Senior Databricks and AI Engineer to join our growing data engineering team. The ideal candidate will have a proven track record in designing, building, and deploying scalable data solutions on the Databricks platform. A critical requirement for this role is demonstrated experience in migrating and modernizing SQL Server environments to Databricks, including data transformation, performance optimization, and ensuring data integrity. This role will also involve developing and implementing AI/ML solutions leveraging Databricks capabilities.

Key Responsibilities:

1. Databricks Platform Expertise

• Design, develop, and maintain robust, scalable ETL/ELT pipelines on Databricks using Spark, Python, Scala, and SQL.
• Optimize Databricks cluster performance, job execution, and cost efficiency.
• Implement data governance, security, and access control best practices on the Databricks Lakehouse.
• Work with Delta Lake, Delta Live Tables, and other Databricks data management features.

2. SQL Server to Databricks Migration

• Lead and execute the migration of data and workloads from on-premises SQL Server databases to the Databricks Lakehouse.
• Develop strategies for schema conversion, data ingestion, and transformation from SQL Server to Delta Lake format.
• Identify and implement performance tuning strategies for migrated SQL workloads on Databricks.
• Ensure data accuracy, consistency, and completeness throughout the migration process.
• Develop and execute comprehensive testing plans to validate migrated data and functionality.

3. AI/ML Development

• Collaborate with data scientists and stakeholders to operationalize AI/ML models on Databricks.
• Develop and deploy machine learning pipelines, including data preprocessing, feature engineering, model training, and inference.
• Utilize Databricks MLflow for experiment tracking, model management, and deployment.
• Explore and implement advanced AI/ML techniques relevant to business needs.

4. Collaboration and Leadership

• Mentor junior engineers and contribute to the team's technical growth.
• Work closely with data architects, data scientists, analysts, and business stakeholders to understand requirements and deliver solutions.
• Provide technical guidance and documentation for data engineering processes and solutions.
• Troubleshoot and resolve complex data-related issues.

5. Qualifications

• Bachelor's or Master's degree in Computer Science, Engineering, Data Science, or a related quantitative field.
• 5+ years of experience in data engineering, with a strong focus on cloud-based data platforms.
• Mandatory: Proven, hands-on experience migrating complex SQL Server environments to cloud-based data warehouses or lake houses, specifically Databricks.
• Mandatory: Deep expertise in Databricks platform, including Spark, Delta Lake, SQL Analytics, and Databricks Runtime.
• Proficiency in programming languages such as Python and SQL. Experience with Scala is a plus.
• Strong understanding of data warehousing concepts, data modeling, ETL/ELT best practices, and database performance tuning.
• Experience with AI/ML workflows, MLOps, and tools like MLflow.
• Familiarity with cloud platforms (AWS, Azure, or GCP) and their data services.
• Excellent problem-solving, analytical, and communication skills.
• Ability to work independently and collaboratively in a fast-paced environment.

6. Preferred Skills

• Databricks certifications.
• Experience with containerization technologies (Docker, Kubernetes).
• Knowledge of other database technologies.
• Experience with CI/CD practices for data pipelines.

Please send your resumes to [email protected].