Remote Data Annotator Jobs Madrid

About The Role As a Remote Data Annotator, you will perform end-to-end data annotation and AI/ML evaluation work that powers training datasets for NLP, computer vision, and large language model evaluation. You will follow detailed annotation guidelines, document edge cases, and contribute to training data quality initiatives that improve model performance.

What You Will Do

• Create and validate labels for text and image data
• Perform named entity recognition and taxonomy tagging
• Run content safety labeling using policy-based classification
• Conduct prompt evaluation and response grading for LLM behavior
• Execute RLHF ranking and pairwise comparisons
• Complete QA checks, audits, and improve inter-annotator agreement
• Write clear rationales for complex or subjective judgments
• Collaborate asynchronously with distributed teams and project leads

Required Qualifications

• Mid-to-senior experience in data annotation, data labeling, or QA evaluation
• Strong written reasoning and ability to maintain guideline adherence
• Familiarity with LLM concepts, RLHF-style ranking, and prompt evaluation
• Comfort using labeling tools, spreadsheets, and structured workflows

Quality Standards Success is measured by audit scores, guideline adherence, low rework rates, strong inter-annotator agreement, consistent throughput, and clear rationales that improve training data quality.

This is a Remote , FULL_TIME role.