Senior AI QA / Evaluation Engineer

About DataSpecta

DataSpecta is a Data & AI consulting and engineering company helping organizations transform data into measurable business value through advanced analytics, modern data platforms and enterprise AI solutions.

We design and deliver solutions across both on-premise and cloud environments, working closely with enterprise customers from discovery through production.

Alongside customer projects, we build our own AI products such as CogniSpecta (enterprise knowledge & cognitive search) and CodeSpecta (AI-powered software engineering).

We're looking for a Senior AI QA / Evaluation Engineer based in Baku, Azerbaijan , with strong test automation expertise and hands-on experience evaluating LLM and ML systems.

You will own quality assurance for production AI systems in a regulated enterprise environment, acting as the quality gate for every production release.

Location & Work Model

• Location: Baku, Azerbaijan
• Work model: [Hybrid / On-site]

What You Will Work On

• Evaluation harnesses and golden datasets for enterprise AI agents
• Evaluation of RAG assistants and report generation agents
• Adversarial and security testing for LLM applications
• Audit evidence and quality governance

Responsibilities

• Build versioned golden datasets and evaluation harnesses
• Gate production releases on evaluation baselines in CI
• Revalidate agents at each model version change
• Run quality control on recurring reports
• Run adversarial tests for prompt injection, data leakage and unsafe tool use
• Triage support and agent output incidents
• Maintain the audit evidence pack

Required Qualifications

• Based in Baku or willing to relocate
• 4+ years in QA or test engineering, including test automation ownership
• Proven experience evaluating LLM or ML outputs
• Strong Python skills for evaluation tooling
• Experience with formal defect management
• Fluent English

Technical Skills

• Evaluation frameworks: Ragas, DeepEval, promptfoo or equivalent
• Metrics: Groundedness, faithfulness, context precision/recall, answer correctness, tool call accuracy
• Statistics: Repeated sampling, bootstrapped confidence intervals
• Automation: Python, pytest, API testing, Playwright, CI
• Adversarial testing: OWASP Top 10 for LLMs, garak, PyRIT
• Observability: Langfuse, Arize Phoenix or equivalent
• Multilingual: Azerbaijani and mixed-language evaluation

Preferred Qualifications (a plus)

• Audit evidence experience in a regulated sector
• Model risk management frameworks

What We Value

• Curiosity about solving business problems with AI
• Comfort across business, data and technology
• End-to-end ownership
• Continuous learning

Follow us:

• LinkedIn: https://www.linkedin.com/company/innovance-consultancy
• LinkedIn: https://www.linkedin.com/company/dataspecta
• Instagram: https://www.instagram.com/innovanceconsultancy
• Instagram: https://www.instagram.com/dataspecta