AI Harness Engineers

AI Harness Engineers Overview

In this role, you will be responsible for ensuring the trustworthiness of AI products within a regulated domain by managing the evaluation evidence for a specific product line. You will work closely with domain experts to develop golden datasets that define correct behavior, implement gates to prevent regressions, and monitor quality in production.

Key Responsibilities

• Own the evaluation evidence for your product line, ensuring its integrity and reliability.
• Develop and maintain golden datasets in collaboration with domain experts.
• Establish gates to prevent regressions from reaching clients, utilizing statistical methods.
• Integrate evaluation processes into CI systems as merge gates and implement drift monitors.
• Analyze production failures to create permanent regression cases.
• Compile evaluation evidence packs for customer model-risk validation.
• Maintain the local harness for your product line, ensuring alignment with platform standards.

Requirements

• Experience in machine learning or data engineering with a focus on rigorous measurement.
• Strong understanding of statistics, including cohort design and sample sizing.
• Hands-on experience in evaluation engineering for language models.
• Proficiency in Python and familiarity with CI/CD systems.
• Willingness to develop domain knowledge through collaboration with experts.
• Experience using AI coding assistants in your workflow.

Benefits

• Flexible working arrangements with a hybrid model.
• Access to extensive training and development opportunities.
• Supportive work environment focused on employee wellbeing.
• Participation in a peer recognition program.

Degree Requirement: Bachelor’s

Visa Sponsorship May be