Agent Evaluation Engineer

Comcast

Base: $142,651.46 - $213,977.19; bonus: eligible f...
Fully remote
Ai agent evaluation pipelines
Conversational ai quality metrics
Ci/cd integration for ai
Comcast is seeking an Agent Evaluation Engineer to join their remote technology team, responsible for testing and improving AI agents. The role involves designing evaluation pipelines, defining metrics, and ensuring consistent agent performance before release, while collaborating with engineering and product teams

Job Summary

  • The team is responsible for building frameworks and test cases to validate AI agent behavior, accuracy, and reliability before release.
  • Employees will design evaluation pipelines across development, staging, and production environments while standardizing metrics for conversational AI quality.
  • Comcast offers a culture that recognizes, rewards, and invests in employee ideas while ensuring they can bring their authentic selves to the workplace.

Matching Summary

Match Score: 85

Comcast is seeking an Agent Evaluation Engineer to join their remote technology team, responsible for testing and improving AI agents. The role involves designing evaluation pipelines, defining metrics, and ensuring consistent agent performance before release, while collaborating with engineering and product teams.

Salary

Base: $142,651.46 - $213,977.19; Bonus: Eligible for non-sales bonus plan; Benefits: Best-in-class benefits including physical, financial, and emotional support options

Skills & Requirements

Must-have

  • AI Agent Evaluation Pipelines
  • Conversational AI Quality Metrics
  • CI/CD Integration for AI
  • Automated Testing Systems
  • Dataset Curation and Annotation

Nice-to-have

  • Customer Support AI Experience
  • Responsible AI Knowledge
  • A/B Testing and Experimentation
  • Mentoring Junior Engineers
  • Curious Mindset

Key Requirements

  • Bachelor's Degree or equivalent experience
  • 5-7 Years of relevant professional experience

Work Rights

Not specified

Tailored Resume

Cover Letter