Senior Databricks AI Engineer

We are HCLTech , one of the fastest-growing large technology companies in the world, with more than 225,000 professionals across 60 countries. Through our expertise in Digital, Engineering, Cloud, Infrastructure, and AI, we help leading organizations transform their businesses and create value through technology.

Our success is driven by our passionate people. We foster an environment where innovation, collaboration, and continuous learning empower individuals to grow and make a meaningful impact.

Overview

We are looking for a Databricks expert with strong experience in platform setup, architecture, and AI/ML capabilities across Azure and/or GCP . The ideal candidate should bring hands-on expertise in building Databricks-based AI solutions , not just traditional data pipelines.

Key Responsibilities

• Design and implement Databricks architecture on Azure and/or GCP (workspace, clusters, networking, security).
• Lead platform setup, governance, and optimization (Unity Catalog, cost control, access management).
• Build and deploy AI/ML solutions using Databricks (MLflow, Feature Store, Model Serving).
• Develop GenAI / LLM solutions (RAG pipelines, embeddings, vector search).
• Integrate LLMs (Azure OpenAI / Vertex AI / other APIs) within Databricks.
• Deliver scalable Lakehouse architecture solutions .
• Collaborate with engineering and business teams for end-to-end AI use cases .
• Ensure performance tuning, reliability, and security of workloads.

Required Skills

• Strong experience in Databricks setup & architecture (must-have) .
• Hands-on with Databricks AI stack :
• MLflow, AutoML
• Feature Store
• Model Serving
• RAG / vector search / embeddings
• Experience with PySpark, Spark optimization, Delta Lake .
• Strong exposure to cloud platforms :
• Azure Databricks + Azure services
• GCP Databricks + GCP ecosystem (BigQuery, GCS, Vertex AI)
• Experience integrating LLMs / GenAI solutions .
• Knowledge of Unity Catalog, governance, and security .

Preferred Skills

• Experience in GenAI applications on Databricks
• Familiarity with LangChain / vector databases
• Exposure to real-time/streaming pipelines
• CI/CD and DevOps practices

Ideal Candidate

• Strong in Databricks platform + AI (not just data engineering)
• Hands-on in cloud setup (Azure/GCP) and AI delivery

Able to design end-to-end AI solutions