Data Scientist QA Lead

Role overview

This remote contractor position is focused on quality leadership for data science AI training work. The main goal is to make sure AI-produced data science content, as well as trainer and QA outputs, meet strict standards for accuracy, clarity, consistency, and usefulness.

You will review technical work against detailed project rubrics, identify gaps, and share clear written feedback so contributors can improve. The role supports a fast-growing AI data services business that creates training data for major AI companies and foundation-model labs. Your contribution will help ensure the data science material is statistically sound, reproducible, well explained, and aligned with client needs.

This is an hourly remote contractor role. There is currently no immediate project available, but suitable candidates may be contacted first when related opportunities come up in the future. Joining this talent network also gives you access to future projects.

Selection process

The hiring process includes an AI interview, a subject-specific task, and a discussion with a recruiter.

Responsibilities

• Review samples of data science work, identify quality problems, and send precise feedback through direct messages.
• Check AI-generated explanations, Python, R, and SQL snippets, modeling steps, statistical interpretations, dashboards, experiment plans, and reasoning flows.
• Escalate repeated issues or serious concerns when contributor quality drops below the expected standard.
• Keep trainers and QAs informed in Discord about guideline updates, workflow changes, and data-science quality expectations.
• Answer questions on statistical assumptions, metrics, model choice, data leakage, validation, coding decisions, reproducibility, and rubric interpretation.
• Reach out to inactive contributors, encourage them to resume work, monitor follow-ups, and raise availability concerns.
• Build and update style guides, trackers, FAQs, examples, honeypots, calibration tasks, and onboarding documents.
• Run onboarding and training sessions to explain project standards, review workflows, and evaluation rubrics.
• Identify misleading, overly confident, statistically unsound, or non-reproducible outputs and flag them quickly.
• Help improve QA systems by spotting recurring issues and supporting more scalable review processes.

Requirements

• A bachelor’s, master’s, or PhD in Data Science, Statistics, Computer Science, Machine Learning, Mathematics, Economics, Engineering, or a closely related quantitative field.
• At least 3 years of professional experience in data science, analytics, machine learning, statistical modeling, experimentation, data engineering, technical review, or data science teaching.
• Strong command of English for following instructions, collaborating with teams, and giving clear technical feedback.
• Solid understanding of statistics, probability, cleaning data, exploratory analysis, feature engineering, supervised and unsupervised learning, model evaluation, experimentation, regression, classification, clustering, and validation.
• Ability to judge data science content against detailed rubrics and catch problems such as leakage, weak assumptions, wrong metrics, poor methods, non-reproducible code, hallucinated libraries or APIs, and misleading conclusions.
• Preferred familiarity with Python, pandas, NumPy, scikit-learn, SQL, Jupyter, matplotlib, R, Spark, Git, MLflow, notebooks, dashboards, and cloud or data platforms.
• Experience coordinating or guiding remote teams of trainers, annotators, analysts, data scientists, engineers, educators, or QAs is highly preferred.
• Comfort using Discord, Google Sheets, Google Docs, trackers, dashboards, GitHub, and project management tools.
• Strong organizational habits and the ability to maintain style guides, trackers, FAQs, onboarding resources, honeypots, calibration tasks, and quality documentation.
• Background in AI training, data annotation, LLM evaluation, data science QA, or rubric-based technical review is a strong advantage.

Additional information

This role is described as an hourly, remote contractor assignment. It is not an immediate active project, but qualified experts may be contacted first when matching opportunities become available. The work is part of an expert network for future assignments.

Selection and workflow notes

The process includes an AI interview, a domain task, and a recruiter interview. Contributors may need to be activated if they are not working regularly, and communication will often happen through Discord and other collaboration tools.