Posted on 2026/02/19
AI QA ENGINEER (AGENTIC & GENERATIVE)
BLYK Engineering Services Inc.
Dallas, TX, United States
Job description Must have:
Agentic QA Engineer Generative AI & Agentic Systems (Agent, MultiAgent Testing)
Summary
We are seeking a hands-on AI Engineer to design and execute end-to-end testing strategies for agentic AI solutions, including multi-agent systems in production-grade environments. This role partners with the Agentic Operations Team to ensure resiliency, reliability, accuracy, latency, orchestratio...n correctness, and scale. You will establish QA frameworks, build reusable test artifacts, drive macro-level validations across complex workflows, and lead the QA function for Agentic AI from Dev to Prod.
• Quality Strategy & Leadership
• Agentic & MultiAgent Testing
• Reliability, Resiliency, and Latency
• Accuracy & Macro-Level Validations
• Scale & Orchestration
Required Qualifications
• 10+ years in Software QA/Testing, with 2+ years in AI/ML or LLM-based systems; hands-on experience testing agentic/multi-agent architectures.
• Strong programming skills in Python or TypeScript/JavaScript; experience building test harnesses, simulators, and fixtures.
• Experience with LLM evaluation (exact/soft match, BLEU/ROUGE, BERTScore, semantic similarity via embeddings), guardrails, and prompt testing.
• Expertise in distributed systems testing latency profiling, resiliency patterns (circuit breakers, retries), chaos engineering, and message queues.
• Familiarity with orchestration frameworks (LangChain, LangGraph, LlamaIndex, DSPy, OpenAI Assistants/Actions, Azure OpenAI orchestration, or similar).
• Proficiency with CI/CD (GitHub Actions/Azure DevOps), observability (OpenTelemetry, PrometheGrafana, Datadog), and feature flags/canaries.
• Solid understanding of privacy/security/compliance in AI systems (PII handling, content policies, model safety).
• Excellent communication and leadership skills; proven ability to work cross-functionally with Ops, Data, and Engineering. Show full description

Zero to AI Engineer
Skip the degree. Learn real-world AI skills used by AI researchers and engineers. Get certified in 8 weeks or less. No experience required.
Find AI, ML, Data Science Jobs By Location
Find Jobs By Position