Senior Product Manager - Tech, GenAI, Amazon Rufus

Amazon

London, GB
On-site
Llm-as-a-judge governance frameworks
International marketplace localization
Model evaluation and debugging workflows
This role owns the end-to-end governance framework for a portfolio of dozens of LLM judges that power critical evaluation metrics

Job Summary

  • This role owns the end-to-end governance framework for a portfolio of dozens of LLM judges that power critical evaluation metrics.
  • You will drive the localization of judges from en-US to five or more international marketplaces while ensuring quality standards are met.
  • The team moves fast and believes that if you cannot measure quality automatically, you cannot improve it at scale.

Matching Summary

This role owns the end-to-end governance framework for a portfolio of dozens of LLM judges that power critical evaluation metrics.

Skills & Requirements

Must-have

  • LLM-as-a-Judge governance frameworks
  • International marketplace localization
  • Model evaluation and debugging workflows
  • Automated governance tool development
  • Cross-functional team leadership

Nice-to-have

  • Experience with LLM evaluation metrics
  • Strong data-driven problem solving skills
  • Background in natural language processing
  • Ability to drive high-autonomy decisions

Key Requirements

  • Senior Product Manager experience
  • Technical background in AI/ML evaluation
  • Proven track record in operational excellence

Work Rights

Not specified

Tailored Resume

Cover Letter