Automation QA Engineer

The Automation QA Engineer will define, implement, and continuously improve quality assurance practices for AI-enabled and agent-based solutions. The role will ensure that AI workflows meet agreed standards for accuracy, reliability, performance, safety, compliance, and operational readiness through automated testing, evaluation, and continuous monitoring.

Key Responsibilities

• Define and execute QA and testing strategies for AI-enabled applications, agent workflows, and automated processes.
• Establish quality criteria covering accuracy, latency, reliability, safety, compliance, and operational risk.
• Build and maintain automated evaluation frameworks to assess agent performance, including:
• Hallucination and response accuracy
• Tool misuse and workflow errors
• Policy or safety violations
• Task completion and success rates
• Regression and quality degradation
• Develop and maintain automated test suites using Playwright for UI and end-to-end testing.
• Create Python-based scripts for API, workflow, and agent validation.
• Apply LLM evaluation frameworks such as DeepEval, RAGAS, and Evidently AI to measure output quality, regression, and system drift.
• Implement continuous production monitoring to identify anomalies, quality degradation, and emerging safety or reliability concerns.
• Develop dashboards and reports to communicate QA metrics, trends, defects, and quality insights to engineering teams and stakeholders.
• Develop and maintain runbooks for common failure scenarios and support incident response activities.
• Collaborate with developers, product teams, and other technical stakeholders to improve prompts, tool definitions, agent workflows, and overall system quality.
• Ensure testing, logging, monitoring, and documentation practices support data privacy, auditability, risk management, and regulatory requirements.
• Contribute to continuous improvement of QA processes, automation frameworks, and AI quality standards.

Must-Have Qualifications

• 3+ years of experience in QA, Test Automation, Software Testing, or DevOps.
• 2+ years may be considered for candidates with direct hands-on experience testing AI/ML-enabled applications or agent-based systems.
• Strong Python skills for test automation, evaluation frameworks, API testing, and basic data analysis.
• Hands-on experience with Playwright or a comparable UI automation framework.
• Experience with API testing and workflow validation using automated scripts.
• Hands-on exposure to AI/LLM evaluation, including tools such as:
• DeepEval
• RAGAS
• Evidently AI
• Understanding of automated testing, regression testing, quality monitoring, and defect analysis.
• Strong analytical and problem-solving skills with high attention to detail.
• Ability to work effectively in a fast-changing technical environment and collaborate with engineering and product teams.

Technical Skills

Required:

• Python – Test Automation & Data Analysis
• Playwright – UI/E2E Automation
• API & Agent Workflow Testing
• LLM/AI Evaluation
• Production Quality Monitoring
• Regression & Performance Testing
• Anomaly Detection

Nice to Have:

• Pytest or equivalent testing frameworks
• SQL for querying logs, metrics, and evaluation datasets
• Prometheus, Grafana, or similar monitoring platforms
• Knowledge of hallucination detection and AI safety patterns
• Experience with CI/CD pipelines and Git-based workflows
• Exposure to cloud environments and observability tools

Application Question(s):

• When is your earliest availability to start?
• What is your expected salary in Philippine Peso (PHP)?
• Are you open to a 6-month project-based role, with a possibility of extension to 9–12 months, depending on your performance and business requirements?
• Are you willing to work full onsite in Taguig?

Education:

• Bachelor's (Preferred)

Experience:

• QA / Test Automation: 3 years (Preferred)
• Python automation: 2 years (Preferred)
• Playwright / UI automation: 2 years (Preferred)
• AI/LLM testing: 1 year (Preferred)
• DeepEval, RAGAS, Evidently AI, or similar LLM: 1 year (Preferred)
• CI/CD & Git-based workflows: 2 years (Preferred)

Work Location: In person