Data Scientist

About the Role

Zof AI is hiring a full-time Data Scientist based in San Francisco, CA to play a critical role in evaluating the performance of its AI agent fleets that detect and resolve software defects. The position involves owning the key metrics such as detection precision, recall, remediation success rates, and ensuring statistical soundness in experiment design to differentiate true improvements from noise.

Key Responsibilities

• Develop and maintain metrics that accurately assess the effectiveness of AI agents in identifying and fixing defects.
• Evaluate detection precision and recall using customer codebases as benchmarks.
• Quantify how successful remediation efforts are and validate the robustness of such evidence.
• Design and analyze experiments to verify if product or model updates produce measurable enhancements.
• Collaborate with engineering teams to create consistent evaluation frameworks for agent performance over time.
• Build reusable and trustworthy analysis pipelines using Python and SQL.
• Communicate insights clearly to cross-functional teams including engineering, product management, and leadership.
• Manage reporting to substantiate claims about the product's impact with data-backed evidence.

Required Qualifications

• Proficiency in Python programming and SQL querying.
• Strong foundation in statistics, including experimental design and inferential analysis.
• Experience handling unstructured, real-world datasets to extract feasible conclusions.
• Capability to transform abstract questions into actionable metrics.
• Ability to reason with causal inference, bias recognition, and confounding factors.
• Effective written and oral communication skills.
• Comfort working in agile, fast-paced environments.
• Demonstrated impact of analytical insights on decision-making.

Preferred Skills

• Previous exposure to assessing the quality of machine learning or AI systems.
• Knowledge of causal inference techniques or Bayesian statistics.
• Experience with data warehouse technologies, dbt, or promoting code notebooks to production workflows.
• Familiarity with software testing methodologies, continuous integration pipelines, and metrics related to developer tools.

Work Environment and Benefits

• Company-provided MacBook Pro and access to premium AI development software tools such as Cursor Ultra, Claude Code Ultra, OpenAI Codex Max or equivalent.
• Engagement in a high-performance AI product environment with direct collaboration with leadership, engineering teams, and customers.
• Opportunity to contribute significantly to product outcomes within San Francisco's thriving AI ecosystem.
• Support for wellness and productivity tailored to individual needs.
• Competitive startup culture encouraging high ownership and impact.