Senior Architect LLM Evaluation & Automation Frameworks

Senior Architect LLM Evaluation & Automation Frameworks

Role Overview

We are seeking a visionary and hands-on Senior Architect to design, scale, and lead our next-generation testing and evaluation ecosystem. In this role, you will bridge the gap between traditional UI/API automation and cutting-edge GenAI/LLM validation.

You will own the architecture for evaluating Large Language Model (LLM) applications using Ragas and DeepEval, while maintaining and modernizing our enterprise Java Selenium test automation suites. The ideal candidate has deep expertise in Python for AI/ML workflows and a proven track record of designing robust enterprise automation frameworks from scratch.

Key Responsibilities

1. LLM Evaluation & GenAI Architecture

• Design and implement robust evaluation pipelines for LLM-powered applications (RAG pipelines, agents, and chatbots) using Python, Ragas, and DeepEval.
• Define core GenAI metrics including faithfulness, answer relevance, context recall, context precision, and hallucination detection.
• Architect automated, continuous evaluation systems integrated directly into CI/CD pipelines to monitor LLM performance regression.
• Establish guardrails and synthetic data generation strategies to stress-test LLM application robustness and safety.

2. Enterprise UI/API Automation & Modernization

• Architect and scale enterprise-grade web automation frameworks using Java and Selenium WebDriver.
• Design hybrid automation frameworks (BDD/Cucumber, TestNG/JUnit, Page Object Model) that support parallel execution and cross-browser testing.
• Lead the migration or integration efforts between legacy Java-based testing infrastructure and modern Python-based AI evaluation tools.
• Optimize locator strategies, handle dynamic web elements, and minimize flaky tests across large-scale regression suites.

3. Strategy, Leadership & DevOps

• Provide technical leadership, mentorship, and architectural governance to QA, AI, and DevOps engineering teams.
• Integrate evaluation and testing frameworks with CI/CD platforms (Jenkins, GitLab CI, GitHub Actions) and cloud environments (AWS, Azure, or GCP).
• Collaborate closely with Data Scientists, ML Engineers, and Product Managers to define acceptance criteria for AI products.

Required Skills & QualificationsTechnical Core

• Programming: Advanced, production-level proficiency in both Python (for AI/ML workflows), Playwright(Typescript) and Java (for enterprise automation).
• LLM Evaluation: Hands-on experience architecting frameworks with Ragas, DeepEval, or TruLens. Deep understanding of prompt engineering and RAG trilemma.
• UI Automation: Expert-level knowledge of Java Selenium WebDriver, Grid, and advanced automation patterns.
• APIs & Data: Strong experience testing RESTful/GraphQL APIs and handling data serialization (JSON, XML).
• CI/CD & DevOps: Experience with Docker, Kubernetes, and setting up automated testing pipelines.

Experience & Leadership

• 12+ years of total experience in software test architecture, automation engineering, or SDET roles.
• 3+ years of dedicated experience working with GenAI, LLM validation, or MLOps frameworks.
• Proven ability to author architectural blueprints, technical roadmaps, and whitepapers for enterprise stakeholders.
• Experience with vector databases (ChromaDB, Pinecone, Milvus) and LLM orchestration tools like LangChain or LlamaIndex.
• Familiarity with alternative modern testing tools like Playwright.

Location- Chennai/Coimbatore/Hyderabad/Bangalore/Pune

Notice Period- Any