Senior Data Annotation (Austin)

Senior Data Annotation (Austin) — Remote (US) Rex.zone is hiring for a full-time Senior Data Annotation role aligned to the Austin market, with day-to-day work performed remotely within the US. You will lead end-to-end data annotation and evaluation workflows that support AI/ML training pipelines, including RLHF, prompt-response evaluation, and rubric-based scoring for large language model evaluation.

What You Will Do

• Execute and review data labeling for NLP and (as needed) computer vision datasets
• Perform RLHF evaluations (preference ranking, pairwise comparisons, structured scoring)
• Run QA evaluation audits for accuracy, consistency, and policy alignment
• Conduct prompt evaluation for helpfulness, correctness, and safety
• Apply named entity recognition tagging and taxonomy mapping where required
• Support content safety labeling using policy-based classification
• Write/refine annotation guidelines, edge-case notes, and decision logs
• Track dataset health metrics (agreement rate, defect density, rework rate) and propose fixes
• Collaborate with engineering on tooling feedback (labeling UI, workflow automation, sampling strategy)

Required Qualifications

• Experience in data annotation, data labeling, or QA evaluation for AI/ML systems
• Familiarity with RLHF and large language model evaluation methodologies
• Ability to follow and improve annotation guidelines compliance under ambiguity
• Strong attention to detail and consistent rubric-based judgment
• Experience with NLP tasks such as classification, summarization evaluation, and named entity recognition
• Professional communication for documenting edge cases and explaining labeling decisions

Work Model This is a Remote role for candidates located in the US , aligned to Austin hiring intent. You will work asynchronously with distributed teams and participate in calibration sessions to maintain training data quality and rubric consistency.

Compensation Hourly base pay range: $30–$50 per hour , depending on experience and scope of QA ownership.

How To Apply Apply via Rex.zone and include a short summary of your data annotation, RLHF, and QA evaluation experience, plus examples of guidelines you have followed or improved (no confidential data).