Android AI Operations & Debug Specialist

ROLE SUMMARY

We are seeking a detail-oriented and analytical Android AI Operations & Debug Specialist to join our team. In this role, you will bridge the gap between hardware orchestration, automated testing, and deep-dive data analysis. You will be responsible for preparing Android devices from scratch, executing automated test scripts, generating and analyzing user action trajectories.

Your insights will directly shape the reliability and intelligence of our platform, ensuring the agent executes complex tasks across locally relevant apps with high accuracy, total precision, and strict compliance with safety policies.

KEY RESPONSIBILITIES

1. Device Provisioning & Configuration

• Flash and configure Android devices to precise build versions based on specific testing requirements.
• Install, configure, and maintain a diverse suite of consumer applications across multiple test devices.
• Ensure environment consistency to guarantee reproducible test results.

2. Test Execution & Device Orchestration

• Run automated scripts to simulate and execute complex user queries on physical or virtual Android devices.
• Manage and execute diverse test suites containing golden & underspecified queries.

3. Trajectory Generation & Analysis

• Generate detailed execution trajectories (logs, UI states, and action sequences) from test runs.
• Analyze trajectories to verify if all user actions and target apps were correctly identified, mapped, and executed by the system.
• Maintain, apply, and iteratively update objective rating criteria to evaluate execution success.

4 . Output Evaluation

• Evaluate automated script execution results, UI action sequences, and AI agent outputs against strict Guidelines to ensure rigorous quality control and compliance.
• Systematically identify, flag, and classify execution failures or trajectory deviations according to specific criterias (e.g., incorrect app selection, wrong action order, UI navigation errors, or policy breaches).
• Document evaluation findings clearly to provide actionable data for engineering teams, contributing to the continuous refinement of evaluation and guideline alignment.
• Verify that the agent correctly parses complex user intent and executes target actions on local apps without violating safety policies or diverging from baseline guidelines.

REQUIRED QUALIFICATION & SKILLS

Technical Skills

• Android Ecosystem Mastery: Strong experience with Android OS, including flashing ROMs/builds, working with adb (Android Debug Bridge), and managing Android environments.
• Code-Level Debugging: Ability to read, interpret, and debug Python/Bash scripts and log files to diagnose errors.
• Data & Trajectory Analysis: Experience analyzing structured logs, JSON outputs, or UI trees to evaluate system behavior.

Experience & Competencies

• Experience with mobile automation tools
• Familiarity with evaluating AI agents, LLM-based tool-use, or complex intent-parsing systems is a massive plus.
• Strong analytical mindset with a rigorous approach to data validation and benchmarking (handling edge cases, negative testing, and ambiguous inputs).
• Excellent documentation skills for updating evaluation rubrics and writing clear bug reports.

Nice to Have

• Experience working with mobile app profiling tools.
• Background in AI/LLM evaluation or benchmarking frameworks.
• LLM Tool-Use Knowledge: Conceptual understanding of how LLM agents interact with third-party tools, APIs, and mobile app interfaces.
• Ability to describe app glitches or unexpected behavior clearly.
• Prior experience with quality checking or rating content.