AI Quality Engineer

Job highlights

• 3+ years QA/testing with API and performance focus, automation (Selenium, Playwright), Python/TypeScript programming, CI/CD and cloud basics
• Lead testing of AI applications including LLMs, RAG systems, agent workflows; design and execute functional, regression, API, performance, and security tests; integrate automated tests into CI/CD; monitor and optimize AI system performance

Role: AI Quality Engineer (Tester)
Experience: 3-4 years
Location: Pune

Purpose:
Lead testing and evaluation of AI applications (LLMs, RAG, agent workflows) to ensure quality, safety, performance, and business alignment.

Key Responsibilities:

• Design & execute functional/regression test cases (SIT/UAT)
• Evaluate LLMs using frameworks (e.g., RAGAS, PromptFoo)
• Test RAG systems: retrieval accuracy, hallucination, grounding
• Validate agent workflows: multi-step logic, tool usage, structured outputs
• Perform API, performance, and load testing (latency, throughput, cost)
• Conduct security testing (prompt injection, data privacy, guardrails)
• Integrate automated tests into CI/CD pipelines
• Monitor, report, and optimize AI system performance
• Collaborate with cross-functional teams and support release readiness

Required Skills:

• 3+ years in QA/testing (API + performance testing)
• Experience with automation tools (Selenium, Playwright)
• Programming knowledge (Python/TypeScript)
• CI/CD, version control, and cloud basics (AWS/Azure/GCP)

Preferred:

• AI/ML and MLOps understanding
• Experience with LLM evaluation tools (LangSmith, RAGAS, TruLens, etc.)
• Knowledge of AI security testing and observability tools

Tools:
Playwright, Selenium, Postman, REST Assured, k6, JMeter, CI/CD tools

Traits:
Strong communication, problem-solving, and quality-focused mindset