AWS Databricks Architect

Greetings from TCS!!

About AWS:

The TCS AWS Business Unit is a dedicated, full-stack group within Tata Consultancy Services launched to help global enterprises accelerate cloud migration, application modernization, and AI-driven digital transformation using Amazon Web Services

Role: AWS Databricks Architect

We are hiring an experienced AWS Databricks Architect with 12+ years of overall IT experience and strong hands-on background in designing and delivering enterprise-scale Data & AI platforms on AWS using Databricks.

The ideal candidate will own the end-to-end Databricks Lakehouse architecture on AWS, drive Data Engineering, Data Science, and Generative AI solutioning, provide technical leadership across multiple engagements, and engage with clients and business stakeholders as an SME.

Responsibility:

• Own end-to-end architecture, design, and delivery of Databricks Lakehouse platforms on AWS across ingestion, processing, storage, consumption, and governance layers.
• Define reference architectures and blueprints using Delta Lake, Unity Catalog, and the Medallion (Bronze/Silver/Gold) framework.
• Lead design of scalable batch and streaming ETL/ELT pipelines using Databricks, PySpark, Spark SQL, and Structured Streaming.
• Architect ML and Data Science workloads using Databricks ML, MLflow, and feature stores; drive MLOps best practices.
• Design and deliver GenAI solutions including RAG pipelines, vector search, prompt engineering, and LLM integrations using LangChain / LlamaIndex.
• Drive Spark performance tuning, cluster sizing, cost optimization, and FinOps for large workloads.
• Define data governance, security, lineage, monitoring, and CI/CD standards using Unity Catalog, IAM, Terraform, and DevOps toolchains.
• Engage with clients in pre-sales, solution discussions, RFP/RFI responses, PoCs, and workshops.
• Provide technical leadership, mentor engineers and leads, review solutions, and enforce architecture best practices.
• Collaborate with AWS partner teams, product engineering, and business stakeholders to align architecture with business outcomes.

Must Have Skills:

• Databricks on AWS – workspace design, cluster policies, jobs, workflows, and administration.
• Delta Lake, Unity Catalog, Databricks SQL, and Lakehouse architecture patterns.
• PySpark, Spark SQL, Python, advanced SQL, and Spark performance tuning at scale.
• AWS Services: S3, Glue, IAM, Lambda, Athena, Redshift, Kinesis, CloudWatch, VPC, KMS.
• Batch and streaming pipeline design (Structured Streaming, Kinesis, Kafka).
• MLflow, model lifecycle management, and MLOps on Databricks.
• Generative AI / LLM integration – RAG architecture, prompt engineering, embeddings.
• LangChain / LlamaIndex and vector databases (e.g., Pinecone, OpenSearch, FAISS).
• Infrastructure as Code – Terraform; CI/CD using Git, GitHub Actions / Jenkins / Azure DevOps.
• Data governance, security, RBAC, encryption, and audit logging.

Good to have:

• Docker / Kubernetes (EKS) exposure.
• MLOps / LLMOps practices at enterprise scale.
• Experience leading large enterprise modernization and cloud migration programs.
• Exposure to multi-cloud (AWS) Databricks deployments.

Location: Pan India

Experience: 12 to 20 years

Qualification: B.E/B.Tech , BSC, BA, MBA, M.Tech, BCA, MSC

Certification:

• Databricks Certified Data Engineer / ML / Architect and AWS Solutions Architect certifications.