Evals Engineer, Applied Ai

Scale AI

San Francisco, CA, US
$179,400—$224,250 usd; not specified; comprehensiv...
On-site
Large language models (llms)
Genai evaluation suite
Human-rated datasets and expert rubrics
This high-impact role is critical to our mission of delivering the industry's leading GenAI Evaluation Suite

Job Summary

  • This high-impact role is critical to our mission of delivering the industry's leading GenAI Evaluation Suite.
  • Partner with Scale’s Operations team and enterprise customers to translate ambiguity into structured evaluation data, guiding the creation and maintenance of gold-standard human-rated datasets and expert rubrics that anchor AI evaluation systems.
  • Scale employees in eligible roles are also granted equity based compensation, subject to Board of Director approval.

Matching Summary

This high-impact role is critical to our mission of delivering the industry's leading GenAI Evaluation Suite.

Salary

$179,400—$224,250 USD; Not specified; Comprehensive health, dental and vision coverage, retirement benefits, a learning and development stipend, and generous PTO.

Skills & Requirements

Must-have

  • Large Language Models (LLMs)
  • GenAI Evaluation Suite
  • human-rated datasets and expert rubrics
  • LLM-as-a-Judge autorater frameworks
  • Python and major ML frameworks
  • frontier model evaluation methodologies

Nice-to-have

  • passion for tackling complex evaluation challenges
  • thrives in a dynamic, fast-paced research environment
  • integrating novel research ideas into workflows
  • published research in leading ML or AI conferences
  • ML research engineering expertise

Key Requirements

  • Bachelor’s degree in Computer Science or related field
  • 2+ years of experience in Machine Learning or Applied Research
  • Hands-on experience with Large Language Models (LLMs)
  • Proficiency in Python and major ML frameworks
  • Advanced degree (Master’s or Ph.D.) preferred
  • Published research in leading ML or AI conferences preferred

Work Rights

Not specified

Tailored Resume

Cover Letter