DevOps Research Engineer — AI Incident Investigation

EYT3 Limited (“EYT3”) is a Pure-Play Cybersecurity and FinTech / Regulatory Technology (RegTech) company based in Hong Kong. The group established in 2016. We specialize in helping organizations to transform to the new digital era through client centric approach and long-term trusted advisor. We are experts in niche areas such as Security Detection and Response, Cyber Threat Protection for Industrial Automation and IoT and Data Analytics for Financial Crime and Fraud Investigation.

Role Overview

A full-time, permanent applied-research position. The successful candidate will lead hands-on research and development of a read-only AI investigation agent for the company's software platform — one that correlates alerts, logs, health checks, and release notes, forms root-cause hypotheses, and cites the evidence behind every recommendation. The role centres on building and evaluating this investigation capability in a laboratory setting; the platform's DevOps environment serves as the research testbed.

Responsibilities

Research Environment & Signal Engineering (R&D Testbed)

• Engineer and maintain the laboratory research environment for the project: the CI/CD pipelines, Kubernetes-based testbed, and deployment tooling used to generate, capture, and replay operational signals for experimentation.
• Research and map the platform's operational signals — alerts, application and deploy logs, health checks, and release/change notes — identifying owners, formats, and access paths for each source as inputs to the investigation agent.
• Study the platform's infrastructure failure modes and compile a failure-mode catalogue (target ≥12 modes, including deploy regression, certificate/TLS failure, dependency timeout, resource pressure, post-release health degradation, and alert storms).
• Conduct a structured survey of published Agentic SRE and incident-agent approaches (including hyperscaler SRE-agent systems) and maintain a capability matrix comparing their methods.

Agent Research & Laboratory Prototype

• Research, design, and implement a read-only investigation agent: a tool-calling LLM workflow operating under a strict allow-list (fetch alert, search logs by service and time window, get health status, get last release, write investigation record), with no shell access and no production mutation APIs.
• Build the signal ingestion and normalisation layer for the four read-only signal types, including desensitisation and synthetic incident generation for laboratory use.
• Develop an evidence-citation mechanism so that every hypothesis and recommendation carries pointers to retrieved artefacts (log lines, health JSON fields, release identifiers).
• Produce structured investigation briefs following a defined output schema: timeline, suspected service, likely change event, evidence citations, recommended next step, confidence, and audit id.

Experimentation & Evaluation

• Curate and label a synthetic incident pack of at least 12 scenarios across six fixed failure families, with seeded ground truth.
• Design and execute the evaluation protocol — measuring evidence-citation rate, faulty-service identification rate, and audit completeness — and analyse results across agent configurations, retrieval windows, and hypothesis-ranking strategies.
• Implement the audit schema (alert id, queries run, documents retrieved, hypothesis text, recommendation, model/tool versions, timestamp) and verify that every investigation brief is fully reconstructable from the audit log.

Trials, Governance & Knowledge Transfer

• Re-run the scenario pack in a controlled sandbox environment that more closely resembles operations, and analyse changes in agent performance.
• Support practitioner review sessions of investigation briefs and write up consultation notes.
• Draft runbook templates the agent is permitted to suggest (e.g. restart instance, roll back to last good version, renew certificate, silence duplicate alert) — templates only, for human execution.
• Collaborate with product, platform, and operations teams to specify how the research outcomes sit beside the existing product as an internal operations copilot.
• Document the architecture, evaluation protocol, and policy matrix; produce research reports, an implementation guide, and a module specification.
• Stay up to date with research in agentic AI, LLM tool-use, and incident-response automation, and incorporate relevant advances into the investigation workflow.

Skills and Qualifications

• Bachelor's degree or above in Computer Science, Software Engineering, or a related field; a Master's degree or equivalent research experience is advantageous.
• 2+ years of experience in backend, platform, SRE, or DevOps engineering; candidates with less experience will be considered for a junior position.
• Proficiency in Python and scripting for log/data processing; experience building against LLM APIs (tool-calling / function-calling patterns) is highly desirable.
• Knowledge of CI/CD pipelines and experience with tools like Jenkins, GitLab CI/CD, or similar, applied here to building the project's experimental testbed.
• Experience with Kubernetes is required; OpenShift experience is advantageous.
• Familiarity with Infrastructure-as-Code (IaC) tools such as Terraform, and with cloud platforms, is advantageous.
• Working knowledge of observability and operations tooling: log pipelines, alerting systems, health-check endpoints, and release processes.
• Understanding of prompt engineering, retrieval-augmented generation, or agent evaluation methods is a strong plus.
• A rigorous, evidence-first research working style: comfortable designing experiments with labelled ground truth and measurable success criteria.
• Strong technical writing skills — the role produces labelled datasets, evaluation protocols, audit schemas, and formal research reports.
• Strong problem-solving skills, teamwork, and communication skills.
• Fluent in English and Cantonese.

What We Offer

• 5-day work week
• Flexible working hours and work-from-home policy
• Performance bonus
• Medical insurance
• Job-related course sponsorship
• The opportunity to conduct applied research on agentic AI for operations, working in a dedicated laboratory environment with direct mentorship from senior technical leadership

Interested candidates, please send your resume, including your current and expected salary, as well as your availability, by clicking "Quick apply".