Sr AI QE Engineer

Full Description

Job Title: Senior AI QE Engineer

Location: Canada

Role Summary

We are looking for a Senior AI Quality Engineering Engineer with strong automation expertise and hands-on experience validating LLMs, GenAI workflows, and AI-driven applications.

This role involves building automated test suites, validating AI outputs, ensuring system reliability, and contributing to the quality strategy for next-generation AI features.

Core Experience

• 6 10 years as a Software Engineer, SDET, or Automation Engineer.

• Strong coding skills in Python, TypeScript, or Java.

• Hands-on experience developing automation scripts, tools, or frameworks.

• Practical experience using LLMs, prompt engineering, and evaluating AI-generated outputs.

• Familiarity with agentic AI systems and exposure to tools like LangGraph, AutoGen, CrewAI.

• Basic understanding of Model Context Protocol (MCP) or context-aware workflow automation (nice to have).

AI / ML Technologies

• Practical experience with AI/ML frameworks: LangChain, Hugging Face, GPT models, vector databases, RAG pipelines.

• Experience with ML/DL libraries such as:

Scikit-learn, PyTorch, TensorFlow, Keras, Transformers, OpenCV.

• Ability to work with embeddings, similarity search, and content evaluation metrics.

GenAI & AI Agent Development

• Ability to integrate or build GenAI components-including RAG pipelines or agent-based workflows.

• Support model evaluation tasks such as:

• Output quality checks

• Hallucination detection

• Prompt validation

• Regression checks for model updates

Automation & Quality Engineering

• Experience building automation using Python, PyTest, Selenium, Playwright, or API testing libraries.

• Ability to design and execute automated tests for:

• Functional

• Integration

• API

• Basic performance & reliability testing

• Hands-on experience testing RESTful APIs and building automated API suites.

Cloud, DevOps & CI/CD

• Exposure to deploying AI or automation solutions on AWS.

• Working knowledge of CI/CD pipelines, including:

• Automated testing

• Model validation steps

• Versioning & artifact management

SDLC & Collaboration

• Strong understanding of the software development lifecycle, including requirements, development, testing, and defect analysis.

• Ability to collaborate with Developers, Data Scientists, and QE teams, clearly communicating progress, risks, and results.

• Skilled in documenting and tracking defects and participating in defect triage.