Senior Data Annotation Specialist

Role Overview

Rex.zone is seeking a Senior Data Annotation Specialist to remotely lead sophisticated data labeling and evaluation tasks that ensure high-quality training inputs for AI and machine learning models. This role focuses on improving large language model performance through detailed annotation, preference ranking, and quality assurance processes.

Key Responsibilities

• Lead advanced annotation and review of diverse data types including text, images, and multi-modal datasets.
• Conduct reinforcement learning from human feedback (RLHF) preference rankings, label rationales, and adhere to evaluation rubrics.
• Implement comprehensive quality assurance procedures including spot checks, secondary reviews, adjudication, and categorization of errors.
• Evaluate prompts based on criteria such as helpfulness, accuracy, safety, and adherence to instructions.
• Ensure strict compliance with annotation guidelines, suggest clarifications and establish decision protocols.
• Monitor dataset quality metrics like accuracy, inter-annotator agreement, and coverage, driving continual improvements.
• Manage dataset governance activities like version documentation, issue tracking, and resolving ambiguous cases.
• Collaborate closely with operations, quality assurance, and engineering teams to support training pipeline enhancements.

Required Qualifications

• Proven professional background in data annotation, data labeling, or quality assurance evaluation within AI/ML environments.
• Experience applying rubric-based assessments and advanced reasoning for complex or edge cases.
• Knowledge of large language model evaluation techniques and RLHF workflow processes.
• Exceptional written communication skills for creating rationales, updating guidelines, and documenting audits.
• High precision and consistent enforcement of policies and annotation standards.
• Comfortable using structured annotation tools, queue management, and quality sampling methodologies.

Applicable Domains

• Natural Language Processing tasks including named entity recognition, classification, and summarization.
• Computer vision annotation such as bounding box, polygon, and attribute labeling.
• Content safety assessment aligned with defined policies and safety criteria.

Compensation & Benefits

The position offers an hourly pay rate ranging from $30 to $50. Additional benefits may include healthcare options, paid leave, and remote work assistance subject to engagement specifics and geographical location.