Ai Quality Engineer

Community Brands

Not specified; not specified; medical, dental & vi...
Fully remote
Hands-on experience with llms or agentic ai systems
Proficiency in python for scripting and test automation
Experience designing evaluations for generative ai features
Community Brands is seeking an AI Quality Engineer to design and implement evaluation frameworks for AI systems, focusing on quality metrics and automated testing. The role requires strong experience with LLMs, Python scripting, and software testing principles, along with a collaborative mindset in a fully remote work environment

Job Summary

  • The role focuses on building robust evaluation frameworks to assess the accuracy, safety, and consistency of LLM and agentic AI systems before they reach production.
  • Candidates will collaborate with engineers and product managers to identify edge cases and define quality metrics such as hallucination rates and tool-use accuracy.
  • Momentive Software offers a purpose-driven culture with benefits including flexible paid time off, employer-paid parental leave, and remote work flexibility.

Matching Summary

Match Score: 85

Community Brands is seeking an AI Quality Engineer to design and implement evaluation frameworks for AI systems, focusing on quality metrics and automated testing. The role requires strong experience with LLMs, Python scripting, and software testing principles, along with a collaborative mindset in a fully remote work environment.

Salary

Not specified; Not specified; Medical, Dental & Vision Benefits; 401(k) Savings Plan with Company Match; Flexible Planned Paid Time Off; Generous Sick Leave; Employer-Paid Parental Leave; Remote Work Flexibility

Skills & Requirements

Must-have

  • Hands-on experience with LLMs or agentic AI systems
  • Proficiency in Python for scripting and test automation
  • Experience designing evaluations for generative AI features
  • Solid understanding of unit, integration, and regression testing
  • Familiarity with agentic frameworks like tool use and RAG

Nice-to-have

  • Experience with prompt engineering and systematic evaluation
  • Familiarity with AI safety, alignment, and responsible AI concepts
  • Exposure to agentic orchestration frameworks like LangChain or AutoGen
  • Experience with vector databases and RAG pipelines
  • Knowledge of observability tools like LangSmith or Weights & Biases

Key Requirements

  • Bachelor's degree in Computer Science, Engineering, or equivalent practical experience
  • 3–5 years of professional software engineering or quality engineering experience
  • Eligibility to work in the United States without sponsorship

Work Rights

Must be eligible to work in the US without sponsorship

Tailored Resume

Cover Letter