Agent Evaluation Engineer

Xumo LLC

Base: $142,651.46 - $213,977.19; bonus: eligible f...
**
Ai agent evaluation pipelines
Conversational ai quality metrics
Ci/cd integration for ai
** Xumo LLC is seeking an Agent Evaluation Engineer to develop and implement evaluation pipelines for AI agents, ensuring accuracy and reliability before deployment. The role emphasizes collaboration with engineering and product teams, along with the mentoring of junior engineers, in a fully remote work environment. **

Job Summary

  • The team is responsible for building frameworks and metrics to validate AI agent behavior, accuracy, and reliability before release.
  • Employees will design evaluation pipelines across development, staging, and production environments while integrating them into CI/CD workflows.
  • Comcast offers a culture that recognizes, rewards, and invests in employee ideas while allowing individuals to bring their authentic selves to work.

Matching Summary

Match Score: 75

** Xumo LLC is seeking an Agent Evaluation Engineer to develop and implement evaluation pipelines for AI agents, ensuring accuracy and reliability before deployment. The role emphasizes collaboration with engineering and product teams, along with the mentoring of junior engineers, in a fully remote work environment. **

Salary

Base: $142,651.46 - $213,977.19; Bonus: Eligible for non-sales bonus plan; Benefits: Best-in-class physical, financial, and emotional support options

Skills & Requirements

Must-have

  • AI Agent Evaluation Pipelines
  • Conversational AI Quality Metrics
  • CI/CD Integration for AI
  • Automated and Human-in-the-loop Testing
  • Dataset Curation and Annotation Workflows

Nice-to-have

  • Customer Support AI Experience
  • Responsible AI Knowledge
  • A/B Testing and Experimentation
  • Mentoring Junior Engineers
  • Curious Mindset

Key Requirements

  • Bachelor's Degree or equivalent experience
  • 5-7 Years of relevant professional experience

Work Rights

Not specified

Tailored Resume

Cover Letter