Research Engineer, Model Evaluations

Anthropic

Remote
Remote
Strong python programming skills
Experience with distributed systems
Reliable data pipeline infrastructure
Anthropic is seeking a Research Engineer for Model Evaluations to develop and implement evaluations that clarify the capabilities of their AI system, Claude. The role involves designing metrics, building infrastructure for reliable evaluations, and working closely with research teams to ensure accurate assessments of AI performance

Job Summary

  • The role focuses on turning ambiguous notions of intelligence into clear, defensible metrics for researchers and the public.
  • Candidates will build and harden a distributed evaluation platform capable of running hundreds of tests reliably against live training checkpoints.
  • The team aims to make Anthropic the leader in extremely well-characterized AI systems with exhaustively measured performance.

Matching Summary

Match Score: 85

Anthropic is seeking a Research Engineer for Model Evaluations to develop and implement evaluations that clarify the capabilities of their AI system, Claude. The role involves designing metrics, building infrastructure for reliable evaluations, and working closely with research teams to ensure accurate assessments of AI performance.

Skills & Requirements

Must-have

  • Strong Python programming skills
  • Experience with distributed systems
  • Reliable data pipeline infrastructure
  • Clear technical communication skills
  • Production support on-call capacity

Nice-to-have

  • Hands-on LLM prompting experience
  • Data visualization dashboard background
  • Robust evaluation metric development
  • Observability and monitoring systems
  • Statistics and experimental design
  • Large-scale dataset curation
  • ML training infrastructure support
  • Pair programming preference

Key Requirements

  • Strong Python programming skills
  • Experience building reliable distributed systems
  • Comfort operating in production-support capacity

Work Rights

Not specified

Tailored Resume

Cover Letter