Senior Test Automation Engineer(AI Testing + python) (0-45 days joiners)

Responsibilities

● Design and execute test plans for agent behavior across reasoning, tool usage, multi-turn flows, and goal completion.

● Create prompt regression suites and synthetic test data for 50+ use cases.

● Build and maintain automated testing frameworks for agent workflows using Python.

● Evaluate agent output using a combination of rubric-based, LLM-based, and scenario-based techniques.

● Test for hallucinations, prompt injection, bias, PII leakage, and response safety.

● Work with observability tools (e.g., LangSmith, Trulens, logging pipelines) to analyze agent execution.

● Collaborate with engineers to iterate on prompt design, tool chaining, and fallback logic.

● Contribute to the development of agent test harnesses and benchmarking pipelines.

️ Skills & Qualifications

Required:

● 3+ years in QA, software testing, or NLP/ML product evaluation.

● Strong Python scripting and test automation experience.

● Experience testing REST APIs, workflows, or backend logic.

● Familiarity with LLMs and basic concepts in prompt engineering.

● Understanding of hallucination, context limits, and RAG-based agents.

Preferred:

● Experience with LangChain, CrewAI, or custom agent frameworks

● Experience with LLM evaluation tools like LangSmith, Promptfoo, Trulens, Ragas

● Exposure to guardrails, moderation APIs, or adversarial testing

● Prior experience in generative AI/LLM-based applications or chatbots

🎯 Bonus Points

● Experience with synthetic data generation

● Familiarity with CI/CD, version control (Git), and Agile QA practices

● Ability to write or grade evaluation prompts using GPT-4 or similar models

📦 What We Offer

● Opportunity to work on cutting-edge GenAI platform used across 50+ enterprise use cases

● Work with cross-functional AI-first teams (MLEs, PEs, PMs)

● Competitive compensation and benefits

● A collaborative, fast-paced environment where quality is a first-class citizen