AI Product Analyst (Developer Community)

About AI Verify Foundation

AI Verify Foundation operates as an open-source initiative and is a wholly owned subsidiary under the Infocomm Media Development Authority (IMDA). Our mission is to promote trustworthy AI by developing and maintaining open-source tools to evaluate large language model (LLM) applications through Project Moonshot and predictive machine learning models using the AI Verify Toolkit. We empower developers, data scientists, and organizations to create responsible, transparent, and dependable AI systems.

Role Overview

We are seeking a Product Analyst at the Manager level to enhance the quality and usefulness of our AI product offerings. This role serves as a nexus between cutting-edge AI research and practical evaluation needs, directing the creation and assurance of quality for datasets and evaluators in our toolkit. Your efforts will drive the success of our developer community through improved AI evaluation tools.

Key Responsibilities

• Design and execute offline A/B testing to optimize evaluation pipelines for Project Moonshot benchmark tests, including LLM-based jury systems.
• Support the development and upkeep of benchmark datasets, focusing on producing realistic test cases relevant to the Singapore context.
• Investigate emerging best practices and open-source tools for generative AI evaluation, distilling insights to guide product roadmaps and build actionable prototypes for engineering teams.
• Engage and educate the developer community about benchmark testing challenges and advocate for the solutions provided within our open-source library.

Candidate Profile

• Educational background in Statistics, Data Science, or a related technical discipline.
• Between one and three years’ professional experience as a data scientist, demonstrating proficiency in experimental design and analytical methods.
• Strong command of Python and/or R programming languages is essential; prior experience with AI testing is advantageous.
• Combination of deep intellectual rigor with a practical and user-focused approach.
• Exceptional communication skills, capable of storytelling through data and simplifying complex ideas for diverse audiences.

Additional Information

The offered position's level and compensation will correspond to each candidate's qualifications and prior experience. Shortlisted applicants will be contacted; others will not receive notifications.