Artificial Intelligence Test Engineer

Company: Zyoin

Website: Visit Website

Business Type: Enterprise

Company Type: Product

Business Model: B2B

Funding Stage: Series A

Industry: SAAS

Job Description This is a Permanent role with Series A funded startup - A valued client of Zyoin. Location: Bangalore (Client Location) - 11 AM - 8 PM - Monday - Friday

Mode of Work: Work from Office

Role Overview

• Automation Tester with hands-on experience testing AI products
• Experience working on and contributing to product testing strategy for AI-driven systems

Key Responsibilities & Skills

• Automation Tester who has tested AI Products
• Product Testing Strategy - someone who has worked upon
• API testing, Automation Framework, BDD, Python + Behave, Git Version Control
• Evaluation reporting and communication, CI/CD Integration
• LLM Evaluation and Testing for Accuracy, relevance, and context of responses or outputs
• Feedback mechanism and continuous validation
• Testing for Bias, Safety and non-subjective response handling / fall back
• Basic performance testing including p50 / p95, response time SLA, and load testing
• Data storage, traceability, and auditability
• Handling edge cases, harmful prompts, etc.
• Any legal framework application for testing before production

Nice To Have

• Experience working with Generative AI or LLM-based applications in production environments
• Exposure to AI governance, responsible AI, or model risk management practices
• Familiarity with cloud platforms (AWS, GCP, or Azure) and cloud-based CI/CD pipelines
• Experience collaborating closely with data scientists and ML engineers
• Understanding of prompt engineering and prompt versioning concepts
• Prior experience in regulated domains such as fintech, healthcare, or enterprise SaaS