Ai Evaluations Engineer

BMO

New York, NY, United States
$122,400.00 - $228,000.00; not specified; health i...
Llm evaluation methods
Ml system evaluation
Evaluation datasets
The AI Evaluation Scientist role is focused on delivering the data science stream of AI evaluations, including designing, implementing, and productionizing evaluation methods, metrics, and datasets

Job Summary

  • The AI Evaluation Scientist role is focused on delivering the data science stream of AI evaluations, including designing, implementing, and productionizing evaluation methods, metrics, and datasets.
  • You will work hands-on with complex models, particularly LLMs and deep learning systems, developing rigorous empirical analyses that surface model weaknesses, performance trends, and risk signals.
  • BMO also offers health insurance, tuition reimbursement, accident and life insurance, and retirement savings plans.

Matching Summary

The AI Evaluation Scientist role is focused on delivering the data science stream of AI evaluations, including designing, implementing, and productionizing evaluation methods, metrics, and datasets.

Salary

$122,400.00 - $228,000.00; Not specified; health insurance, tuition reimbursement, accident and life insurance, and retirement savings plans

Skills & Requirements

Must-have

  • LLM evaluation methods
  • ML system evaluation
  • evaluation datasets
  • Python and SQL proficiency
  • PyTorch or TensorFlow
  • reproducible experimentation

Nice-to-have

  • interpretability techniques
  • fairness techniques
  • research contributions
  • open-source contributions

Key Requirements

  • 7+ years data science/ML/AI experience
  • 3+ years evaluation/safety/reliability focus
  • Master's or PhD or equivalent experience
  • Experience building evaluation pipelines
  • Experience with RAG systems

Work Rights

Not specified

Tailored Resume

Cover Letter