AI Scoring and Evaluations Architect

Role Overview

This position focuses on designing architectures for AI-driven scoring and evaluation systems. The role involves translating rubric criteria into automated scoring logic, developing prompts and frameworks to support assessment accuracy, and implementing pipelines for speech, acoustic, and language analysis.

Key Duties and Responsibilities

• Develop AI architectures that convert rubrics into scoring logic.
• Create scoring prompts and comprehensive evaluation frameworks.
• Design and maintain speech, acoustic, and natural language processing pipelines.
• Implement machine learning components to support scoring evaluation.
• Construct psychometric scoring models and calibration logic.
• Manage benchmark, annotation, and calibration datasets including human-rated references.
• Produce detailed reports on model accuracy, consistency, and analyze errors.
• Perform assessments of bias, edge cases, and potential failure modes.
• Develop regression testing frameworks to ensure system robustness.
• Draft API specifications for scoring and feedback functionality.
• Document scoring model versions and maintain comprehensive records.

Ideal Candidate Profile

• 3 to 5 years of research-driven experience in fields related to AI scoring.
• Background may include expertise in speech recognition, speech analytics, NLP evaluation, machine learning validation, automated essay or response evaluation, LLM evaluation frameworks, acoustic or phonetic analysis, as well as recommendation or classification systems.

Educational Qualifications

• Advanced degrees (M.Tech., M.S. or Ph.D.) in Artificial Intelligence, Machine Learning, Data Science, Computer Science, Computational Linguistics, or Natural Language Processing.
• Alternatively, degrees in Signal Processing, Electronics and Communication Engineering, Speech Technology, or Audio Processing combined with relevant experience are acceptable.
• Postgraduate or doctoral research in Computational Linguistics, particularly focusing on language model evaluation, automated scoring, text analytics, human annotation, or linguistic feature extraction.
• Strong applied AI researchers or machine learning engineers showing demonstrable experience in speech, language, evaluation systems, or assessment technologies, regardless of formal degree background.