Senior Applied Ai Engineer (observability)

Komodo Health

San Francisco, CA, United States
On-site
Production ai system development experience
Llm prompt engineering and agent orchestration
Ai observability and evaluation framework design
The role focuses on building the foundation that makes AI systems trustworthy at scale within a newly formed AI Platform team

Job Summary

  • The role focuses on building the foundation that makes AI systems trustworthy at scale within a newly formed AI Platform team.
  • Candidates will own problems end-to-end from early exploration through production deployment of logging, tracing, and evaluation pipelines.
  • The position requires defining metrics for LLM performance including hallucinations, drift, latency, and failure modes in production.

Matching Summary

The role focuses on building the foundation that makes AI systems trustworthy at scale within a newly formed AI Platform team.

Skills & Requirements

Must-have

  • Production AI system development experience
  • LLM prompt engineering and agent orchestration
  • AI observability and evaluation framework design
  • Python backend engineering and distributed systems
  • A/B testing and benchmarking implementation

Nice-to-have

  • Experience in ambiguous fast-moving environments
  • Strong communication and mentorship skills
  • Background in healthcare data ecosystems
  • Ability to drive experimentation across teams

Key Requirements

  • Experience building production AI systems end-to-end
  • Strong expertise with LLMs and tool/function calling
  • Hands-on experience with debugging production AI behavior

Work Rights

Not specified

Tailored Resume

Cover Letter