Data Science Expert

About the Role

We seek experienced data science professionals to assess the effectiveness of AI systems by evaluating their performance on practical data science tasks. This involves defining clear and precise grading criteria for specific assignments and providing thorough written justifications for scoring AI-generated and human work samples.

Key Responsibilities

• Create detailed, task-specific grading rubrics for various real-world data science outputs such as analyses, predictive models, dashboards, experimental summaries, and strategic recommendations.
• Assess and score completed work—both AI-produced and human—using these criteria, ensuring each score is supported by comprehensive written reasoning.
• Consistently apply objective, evidence-based judgment to guarantee scoring is reliable, repeatable, and can withstand scrutiny.
• Quickly incorporate structured feedback from senior reviewers and revise evaluations accordingly.

Candidate Profile

• Minimum of 5 years of professional experience in data science within industry settings.
• Experience in business operations, product analytics, or growth data science roles at leading technology firms.
• Expertise in experiment design including A/B testing, defining relevant metrics, proficiency in SQL and Python for data analysis, and effectively communicating results to senior management.
• Exceptional written communication skills.
• Detail-oriented personality with commitment to consistency and openness to peer reviews and calibration exercises.
• Prior involvement with AI training, evaluation, or projects combining human data is highly advantageous.