Ai Evaluations Lead

CRESTA

Remote
Base salary + bonus + equity; competitive location...
Remote
5+ years quality engineering experience
Systems thinking with llm knowledge
End-to-end technical project leadership
Cresta is seeking an AI Evaluations Lead to oversee the quality assurance of their AI Agent product line, merging human psychology with machine logic. The ideal candidate should have extensive experience in quality engineering, particularly within AI or SaaS environments, and possess strong leadership and operational skills

Job Summary

  • Cresta is on a mission to turn every customer conversation into a competitive advantage by unlocking the true potential of the contact center using AI and human intelligence.
  • As the Ai Evaluations Lead, you will own the end-to-end quality strategy, designing complex test plans for non-deterministic LLMs and building scalable testing environments.
  • The company offers comprehensive medical, dental, and vision coverage along with flexible PTO, paid parental leave, and a remote work setup budget.

Matching Summary

Match Score: 85

Cresta is seeking an AI Evaluations Lead to oversee the quality assurance of their AI Agent product line, merging human psychology with machine logic. The ideal candidate should have extensive experience in quality engineering, particularly within AI or SaaS environments, and possess strong leadership and operational skills.

Salary

Base salary + Bonus + Equity; Competitive location-based pay reflecting market and individual skillset; Not specified

Skills & Requirements

Must-have

  • 5+ years Quality Engineering experience
  • Systems thinking with LLM knowledge
  • End-to-end technical project leadership
  • Manual UAT and voice-call testing
  • Building automated testing frameworks

Nice-to-have

  • Experience with CCaaS or telephony
  • Background in Conversation Design
  • SDET role experience
  • Direct team management experience
  • High empathy and consultative mindset

Key Requirements

  • 5+ years in Quality Engineering or Technical QA
  • Strong technical intuition regarding LLMs and RAG
  • Proven ability to lead large E2E technical projects
  • Experience leading a pod of QA analysts

Work Rights

Not specified

Tailored Resume

Cover Letter