Test automation framework architecture development
This role involves designing end-to-end evaluation frameworks for AI platforms including LLM-based chat and voice bots to ensure accuracy and relevance
Job Summary
This role involves designing end-to-end evaluation frameworks for AI platforms including LLM-based chat and voice bots to ensure accuracy and relevance.
The company is a global healthcare leader headquartered in Indianapolis dedicated to discovering life-changing medicines and improving disease management.
Candidates will work hands-on with production-grade automation code in Python, Java, and JavaScript to build scalable test infrastructure integrated into CI/CD pipelines.
Matching Summary
This role involves designing end-to-end evaluation frameworks for AI platforms including LLM-based chat and voice bots to ensure accuracy and relevance.
Skills & Requirements
Must-have
AI platform evaluation framework engineering
Python Java JavaScript TypeScript proficiency
Test automation framework architecture development
LLM hallucination detection and benchmarking
CI/CD pipeline integration with GitHub Actions
Performance load testing with JMeter
Cloud-based test execution on AWS Kubernetes
Nice-to-have
Experience with LangChain or LangGraph orchestration
Multi-modal AI application testing expertise
Speech recognition quality metrics knowledge
GxP HIPAA compliance testing in regulated environments
Open-source contribution to test automation tooling
Self-healing automation and autonomous testing agents
Key Requirements
10+ years of Quality Engineering experience
Bachelor's or Master's degree in Computer Science or Engineering
Hands-on experience building AI evaluation frameworks
Deep expertise in pytest Selenium WebdriverIO Appium
Strong understanding of cloud computing and containerized testing