- Erfahrung
- Beliebig
- Gehalt
- —
- Stellenangebote
- 1
- Veröffentlicht
- vor 4 Stunden
- Arbeitsmodus
- Im Büro
- Ausbildung
- PhD or equivalent research experience
- Wieder aufnehmen
- Bewerbung erforderlich
Stellenbeschreibung
About OxSci
OxSci is pioneering a transformative approach to scientific peer review by developing a certification system that integrates AI technologies with expert evaluations. This hybrid model aims to rapidly and reliably assess the quality of scientific research, addressing longstanding issues of speed, transparency, and reliability in traditional peer review.
Role Overview
Join OxSci's founding team as an AI Research Scientist focusing on evaluating AI-based peer review systems. This position offers meaningful equity and the opportunity to shape foundational standards that will influence the future of AI-assisted scientific review.
Key Responsibilities
- Lead research on the evaluation of AI reviewers by continuously monitoring cutting-edge studies in AI-science interaction, automated review, and natural language processing.
- Design and conduct meta-evaluations that go beyond agreement metrics, identifying specific failure points such as factual inaccuracies or hallucinations in AI reviews.
- Manage large-scale studies involving expert annotations, developing protocols, rubrics, and performing rigorous statistical analyses like inter-annotator agreement to validate AI vs. human reviewer performance.
- Create and maintain a detailed taxonomy of AI reviewer errors and establish regression benchmarks to prevent performance regressions over time.
- Develop and calibrate quality scoring rubrics to integrate human and AI judgments into reliable composite ratings used by academic institutions and publishers.
- Implement feedback loops to improve AI review agents based on benchmark outcomes, enhancing retrieval methods, context management, and model orchestration.
Candidate Profile
- Possess a PhD (or near completion) in Computer Science, Machine Learning, Natural Language Processing, or related areas, or demonstrate equivalent research excellence.
- Exhibit deep interest in the interplay between AI and human reviewers, aiming to delineate their respective strengths and limitations methodically.
- Have a strong background in rigorous evaluation frameworks for ML/LLM systems, including designing benchmarks, study protocols, and metrics for accuracy, hallucination, and uncertainty assessments.
- Be adept with evaluation methodologies and statistical techniques such as sampling strategies, inter-annotator agreement calculation, and significance testing.
- Proficient in Python programming with practical experience building evaluation pipelines and integrating large language models through retrieval-augmented generation (RAG), orchestration, or tool integration.
- Bonus qualifications include published work in NLP/ML evaluation, contributions to open-source benchmarks, or experience handling scholarly corpora at scale.
What We Provide
- Founding team position with significant equity participation and direct collaboration with company founders.
- Ownership of a landmark research challenge with encouragement to publish and present findings publicly.
- Access to a continuous network of expert reviewers serving as a scalable annotation resource.
- Exclusive datasets comprising paired human and AI peer review records of actual scientific submissions.
- Competitive salary with openly discussed equity and a generous daily token budget for large language model work.
- Flexible working schedule fostering rapid development and broad responsibility from the outset.
Additional Information
- This position is hybrid, based in London.
- Work visa sponsorship is available for qualified candidates.