Freelance Agent Evaluation Engineer

Role overview

Mindrift offers project-based AI work for specialists who want to help test, assess, and improve advanced AI systems for leading technology companies. This is not a permanent employment role; participation is tied to individual projects.

Project focus

The assignment centers on creating a dataset used to measure how well AI coding agents perform on realistic development work. The goal is to mirror genuine engineering situations and determine whether a model can handle them accurately.

What you will do

• Set up believable developer environments, including a virtual company setup with a codebase, infrastructure, and supporting context such as tickets, documentation, and conversations.
• Create tasks from intermediate states of those environments by writing the prompt, defining the success criteria, and confirming that an AI agent can complete the task.
• Develop tests that check whether the agent’s output is correct, allowing every valid solution while filtering out incorrect ones without being overly strict.
• Refine tasks and test cases using QA feedback, reviewing agent attempts, diagnosing failures, and improving the work until the evaluation is reliable and balanced.

What this role is not

• This is not a data annotation position.
• This is not a prompt-writing role.
• You will not be building the entire solution from scratch; the agent performs most of the coding while you guide the task and judge the results.

Profile sought

The ideal candidate has strong software development experience, a solid grasp of testing, and the ability to assess complex coding outputs in English. You should be comfortable working with realistic engineering scenarios and identifying where AI systems tend to succeed or fail.

Working style and compensation

The expected effort for each task is around 20 hours, depending on complexity. This is only an estimate and does not define a fixed schedule. You can choose when and how to work, but submissions must be completed by the deadline and satisfy the acceptance criteria to be approved. Compensation can be as high as the equivalent of $50 per hour, depending on your level and pace.

Application notes

Please send your CV in English and clearly mention your English proficiency level.

Additional information

This opportunity is project-based rather than a continuous employment arrangement. The process is: apply, complete the qualification stage if required, join a project, complete the assigned tasks, and receive payment.

Requirements

• At least 5 years of experience in software development.
• Hands-on familiarity with Python, especially FastAPI.
• Experience with JavaScript or TypeScript, including React.
• Working knowledge of Docker, PostgreSQL, Kafka, and Redis.
• Practical experience creating functional and integration tests.
• English level of B2 or higher.

Compensation details

The role is paid at an hourly equivalent rate of up to $50, based on experience and working pace. The project tasks are estimated to take about 20 hours each.

Important note

Tasks must meet the stated acceptance standards to be accepted, and the final workload depends on task complexity.