Research Engineer

About Judgment Labs

Judgment Labs specializes in Agent Behavior Monitoring (ABM) infrastructure. Unlike traditional observability that tracks exceptions and latency, ABM identifies behavioral irregularities such as instruction deviation and loss of context retrieval in large-scale production environments.

Hundreds of teams deploying autonomous agents rely on Judgment Labs to gain insights into system behavior after deployment. Rather than simply reacting to incidents, they analyze conversation clusters and workflows, link regressions to specific interaction types, and detect reliability failures within the operational context.

The company has secured over $30 million in funding across two rounds in the last five months from notable investors including Lightspeed, SV Angel, Valor Equity Partners, Nova Global, and several industry leaders.

Role Overview

We seek Research Engineers capable of creating AI systems that leverage agent interaction data to assess agent behavior, conduct large-scale evaluations, and enhance performance via learning and feedback mechanisms. Your work will directly impact real-world operations, working hands-on with agent data, applying advanced methodologies in production, and quickly deploying your solutions to end-users.

You will handle projects with considerable independence and collaborate closely with the team to develop self-improving agent systems serving critical sectors such as finance, legal, and operations.

Key Responsibilities

• Develop systems to collect, index, and analyze vast quantities of agent interaction data to derive actionable evaluation metrics.
• Construct agent-focused analytical tools for assessing complex, long-term behaviors.
• Create post-training and optimization pipelines aimed at refining agent performance.
• Build internal platforms and infrastructure facilitating quick experimentation, data analysis, and model training.

Desired Qualifications

• Passion for data integrity, system evaluation, and benchmarking, with experience handling imperfect data.
• Background in developing or operating agent systems in practical or production environments.
• Strong foundation in reinforcement learning, agent methodologies, or core machine learning principles.
• Comfortable spanning infrastructure domains including training processes, data pipelines, and model deployment.
• Proven ability to collaborate across teams, converting research insights into customer-driven products while managing real-world constraints.
• Aptitude for transforming ambiguous challenges into structured and actionable plans.

What Makes Judgment Labs Unique?

• We provide the critical monitoring infrastructure enabling agents to self-correct — addressing hallucinatory and drift issues common in production agents.
• Our compact yet dynamic team operates at high velocity, boasting members with achievements in olympiads, debates, and competitive sports who bring fierce dedication to building the company.
• Engineers have rapid advancement opportunities, interacting with clients, deploying code into production environments, and directly shaping the company's roadmap. Most team members are founders or aspiring founders.
• We prioritize our team's well-being and performance, offering comprehensive benefits, access to premium fitness facilities, and private chef services. While we work intensively, we also cherish camaraderie through activities like Smash and Mario Kart tournaments.

Work Location

This role requires working onsite at our San Francisco office.