QA Engineer with AI (Copilot, Chat Analytics)

Qualifications

• We are seeking a skilled QA Engineer with a strong background in AI-based tools like Copilot, chat analytics, and intelligent automation to join our quality engineering team
• The ideal candidate will bring deep expertise in designing and executing test strategies for systems that integrate LLMs (Large Language Models), AI Copilots, and conversational analytics platforms
• 4+ years of experience as a QA Engineer or Software Tester in agile environments
• Experience testing AI/ML-enabled applications, Copilot features, or LLM-based assistants
• Familiarity with chat analytics platforms (e.g., Google CCAI, AWS Contact Lens, LivePerson, Genesys, or custom NLP engines)
• Solid understanding of AI testing techniques: prompt validation, hallucination detection, confidence scoring, etc
• Proficient in test automation tools: Selenium, Cypress, Playwright, Postman, REST-assured
• Basic understanding of NLP, embeddings, chatbots, sentiment analysis, and intent classification
• Strong experience with REST APIs, JSON, data payloads, and integration test strategies
• Knowledge of CI/CD tools like Jenkins, GitHub Actions, GitLab CI
• Excellent problem-solving skills and the ability to debug complex AI output behaviors
• Strong documentation and communication skills

Responsibilities

• This role focuses on ensuring product quality, performance, and intelligent behavior alignment with business and user expectations
• Design and implement test plans for AI-enabled applications, Copilot features, and chat analytics systems
• Perform functional, regression, exploratory, and system-level testing on intelligent automation workflows and conversational AI models
• Collaborate with data scientists, AI engineers, and developers to validate AI output quality, relevance, and compliance
• Test Copilot integrations across IDEs, enterprise tools, and custom workflows to ensure expected assistance behaviors
• Validate chat analytics pipelines, including NLP tagging, sentiment detection, and intent classification
• Ensure AI models adhere to fairness, bias, explainability, and accuracy criteria
• Automate test cases using Python, Selenium, Playwright, or Cypress, with custom frameworks for LLM-based validation
• Perform data validation and end-to-end testing on chat transcripts, embeddings, metadata pipelines, and insight dashboards
• Document detailed test cases, QA reports, and defect logs using JIRA, TestRail, or similar tools
• Collaborate with DevOps to integrate AI-specific test suites into CI/CD pipelines
• Contribute to defining KPIs for AI system performance and user interaction accuracy

Full Description

Job Title : QA Engineer with AI (Copilot, Chat Analytics)

Job Location : Atlanta, GA Hybrid (Wed & Thurs)

Job Type: Contract

Job Summary:

We are seeking a skilled QA Engineer with a strong background in AI-based tools like Copilot, chat analytics, and intelligent automation to join our quality engineering team. The ideal candidate will bring deep expertise in designing and executing test strategies for systems that integrate LLMs (Large Language Models), AI Copilots, and conversational analytics platforms. This role focuses on ensuring product quality, performance, and intelligent behavior alignment with business and user expectations.

Key Responsibilities:

• Design and implement test plans for AI-enabled applications, Copilot features, and chat analytics systems.

• Perform functional, regression, exploratory, and system-level testing on intelligent automation workflows and conversational AI models.

• Collaborate with data scientists, AI engineers, and developers to validate AI output quality, relevance, and compliance.

• Test Copilot integrations across IDEs, enterprise tools, and custom workflows to ensure expected assistance behaviors.

• Validate chat analytics pipelines, including NLP tagging, sentiment detection, and intent classification.

• Ensure AI models adhere to fairness, bias, explainability, and accuracy criteria.

• Automate test cases using Python, Selenium, Playwright, or Cypress, with custom frameworks for LLM-based validation.

• Perform data validation and end-to-end testing on chat transcripts, embeddings, metadata pipelines, and insight dashboards.

• Document detailed test cases, QA reports, and defect logs using JIRA, TestRail, or similar tools.

• Collaborate with DevOps to integrate AI-specific test suites into CI/CD pipelines.

• Contribute to defining KPIs for AI system performance and user interaction accuracy.

Required Skills & Qualifications:

• 4+ years of experience as a QA Engineer or Software Tester in agile environments.

• Experience testing AI/ML-enabled applications, Copilot features, or LLM-based assistants.

• Familiarity with chat analytics platforms (e.g., Google CCAI, AWS Contact Lens, LivePerson, Genesys, or custom NLP engines).

• Solid understanding of AI testing techniques: prompt validation, hallucination detection, confidence scoring, etc.

• Proficient in test automation tools: Selenium, Cypress, Playwright, Postman, REST-assured.

• Basic understanding of NLP, embeddings, chatbots, sentiment analysis, and intent classification.

• Strong experience with REST APIs, JSON, data payloads, and integration test strategies.

• Knowledge of CI/CD tools like Jenkins, GitHub Actions, GitLab CI.

• Excellent problem-solving skills and the ability to debug complex AI output behaviors.

• Strong documentation and communication skills.