B2b saas product management for non-technical users
Experience shipping qa tooling or admin consoles
This role owns the end-to-end product strategy for a Testing & Observability suite designed to let admins simulate conversations and measure AI agent quality before deployment
Job Summary
This role owns the end-to-end product strategy for a Testing & Observability suite designed to let admins simulate conversations and measure AI agent quality before deployment.
The successful candidate will define how simulation works end-to-end, including scenario generation from real conversation patterns and automated pass/fail evaluation.
Zendesk is an equal opportunity employer committed to fostering global diversity, equity, and inclusion while offering a hybrid work environment.
Matching Summary
This role owns the end-to-end product strategy for a Testing & Observability suite designed to let admins simulate conversations and measure AI agent quality before deployment.
Skills & Requirements
Must-have
AI/ML product experience with reliability focus
B2B SaaS product management for non-technical users
Experience shipping QA tooling or admin consoles
Understanding of LLM evaluation and traditional testing limitations
Ability to ship platform capabilities through user-facing surfaces
Integration of acquired or adjacent products into unified experiences
Coordination across multiple engineering teams
Nice-to-have
Background in conversational AI or chatbot platforms
Experience building simulation or synthetic data products
Familiarity with human-in-the-loop scoring methods
Experience with A/B testing infrastructure at scale
Track record turning internal prototypes into customer-facing products
Knowledge of automated rubrics and AI-as-judge approaches
Key Requirements
Several years of product management experience
2+ years building for non-technical users in complex technical domains
Experience shipping AI/ML products where evaluation was a priority
Proven track record coordinating across 3+ engineering teams