Ai Qa Engineer – Large Language Models

Meltplan

Bengaluru, India
Competitive comp; meaningful equity; not specified
On-site
Design and implement test strategies for llm-based applications
Evaluate llm outputs for accuracy, consistency, bias, and hallucinations
Develop automated testing frameworks for ai/ml systems
MeltPlan is seeking an AI QA Engineer specialized in Large Language Models to ensure the quality and reliability of AI-driven applications. The role involves designing test strategies, evaluating model outputs, and collaborating with cross-functional teams to enhance model performance. Candidates should possess a relevant degree, experience in QA/testing within AI/ML environments, and strong analytical skills

Job Summary

  • MeltPlan is building the planning engine for the $14 Tn construction industry to optimize decisions before construction begins.
  • The role involves designing test strategies for LLM-based applications and validating model outputs for accuracy and safety.
  • Candidates will enjoy high ownership from day one in a small team with zero bureaucracy and meaningful equity.

Matching Summary

Match Score: 85

MeltPlan is seeking an AI QA Engineer specialized in Large Language Models to ensure the quality and reliability of AI-driven applications. The role involves designing test strategies, evaluating model outputs, and collaborating with cross-functional teams to enhance model performance. Candidates should possess a relevant degree, experience in QA/testing within AI/ML environments, and strong analytical skills.

Salary

Competitive comp; Meaningful equity; Not specified

Skills & Requirements

Must-have

  • Design and implement test strategies for LLM-based applications
  • Evaluate LLM outputs for accuracy, consistency, bias, and hallucinations
  • Develop automated testing frameworks for AI/ML systems
  • Create test datasets, prompts, and evaluation benchmarks
  • Proficiency in Python programming language
  • Experience with API testing tools like Postman
  • Understanding of NLP concepts such as tokenization

Nice-to-have

  • Experience working in construction or on project sites
  • Startup experience preferred
  • Ability to write code to prototype solutions
  • Enjoy being close to the field not just behind a desk
  • Familiarity with cloud platforms AWS GCP or Azure

Key Requirements

  • Bachelor's degree in Computer Science or Engineering
  • 3–6 years of experience in QA/testing
  • Experience testing AI/ML models or data pipelines
  • Experience with prompt engineering and prompt testing

Work Rights

Not specified

Tailored Resume

Cover Letter