Description

Seeking an experienced QA Lead with deep expertise in AI systems testing to join a contract-based team in Montreal, Quebec. The role involves owning end-to-end test strategy for complex AI products, defining quality standards, and working at the intersection of quality engineering and artificial intelligence.

Required Skills

LLM Behavior Analysis
RAG Pipeline Validation
Modern AI Orchestration Frameworks
Test Strategy Design
Prompt Engineering
Data Quality Validation
Retrieval Accuracy Assessment
Vector Database Mechanics
LangChain
LangGraph
Generative AI Evaluation Metrics
Written and Verbal Communication in English

Additional Information

Required Previous Experiences: Proven QA leadership experience designing and executing test strategies for AI/ML systems or LLM-powered applications.
Strong understanding of LLM internals: tokenization, embeddings, attention mechanisms, and inference behavior to anticipate and diagnose failure modes.
Hands-on experience with prompt engineering — constructing effective prompts, detecting hallucinations, and evaluating outputs across accuracy, tone, coherence, and bias dimensions.
Experience testing RAG pipelines and knowledge base integrations, including validation of data quality and retrieval accuracy as they impact model outputs.
Familiarity with vector database mechanics: similarity search thresholds, embedding drift, near-duplicate documents, and sparse vs. dense embeddings.
Practical experience with LangChain and/or LangGraph — able to read chain/graph construction code, identify failure points, and write test harnesses.
Ability to validate MCP (Model Context Protocol) integration points, including tool availability and error-handling scenarios.
Proficiency applying generative AI evaluation metrics and establishing quality thresholds appropriate for production AI systems.
Preferred Qualifications: Experience with bias detection and safety testing frameworks for AI systems.
Exposure to performance and scalability testing of vector databases under high load.
Familiarity with CI/CD pipelines for ML model deployment and automated regression testing.
Knowledge of responsible AI principles and AI governance frameworks.
Contributions to or experience with open-source AI testing or evaluation tooling (e.g., DeepEval, Ragas, PromptFlow).
Background in data engineering or data quality practices relevant to AI pipeline inputs.
Cloud platform experience (AWS, Azure, or GCP) in the context of deploying or testing AI workloads.

About Jay Analytix

Jay Analytix

data analytics, performance engineering, AI, and IT staffing11-50

Contact Us Mail us 24/7 info@jayanalytix.ca Our Locations 3542 Walker Rd, Windsor, ON N8W 3S4 50875 Sevilla Cir, Novi, MI 48375 Call us 24/7 Phone: +1 (437) 291-3057 Working Days Mon – Fri : 9am – 5pm Sat – Sun : 9am – 12pm Contact Us Questions? We Have The Answers! We aim to answer your queries at the earliest! Get in touch with us

View full company profile

QA Lead — AI Systems & Models Testing at Jay Analytix

Description

Required Skills

Additional Information

About Jay Analytix

Trending job searches

Get More from Metaintro

Ready to find a role that actually fits?

Required Skills

Additional Information

About Jay Analytix