UK Defence AI Evaluation & Planning SME

About the Role

Cyberfortis Consulting Limited is seeking an experienced UK Defence AI Evaluation & Planning Subject Matter Expert (SME) to support the independent evaluation, benchmarking and validation of an advanced AI-enabled capability within a UK Defence technology programme.

This position plays an important role in ensuring that emerging AI technologies meet demanding standards of technical performance, reliability, security and operational effectiveness.

The successful candidate will combine strong analytical expertise with practical experience evaluating complex AI systems, establishing robust testing methodologies and assessing their performance against defined requirements.

Working alongside specialists in artificial intelligence, cybersecurity and systems engineering, the SME will contribute to technical assurance, performance improvement and evidence-based decisions throughout the development lifecycle.

We are particularly interested in professionals who can look beyond conventional AI performance metrics to assess how effectively intelligent systems perform in complex, real-world environments.

Key Responsibilities

• Design, develop and implement comprehensive AI evaluation and benchmarking frameworks.
• Establish measurable performance indicators, evaluation criteria and acceptance thresholds.
• Assess AI model and system performance, including accuracy, reliability, robustness, consistency and scalability.
• Develop representative test scenarios, datasets and validation methodologies.
• Evaluate AI-generated outputs for factual accuracy, relevance, explainability, traceability and potential limitations.
• Assess the effectiveness of AI-enabled workflows and their ability to deliver consistent, reproducible results.
• Identify performance gaps, technical weaknesses, potential risks and opportunities for improvement.
• Conduct independent technical reviews and structured validation exercises.
• Evaluate the effectiveness of human oversight, system transparency and appropriate control mechanisms.
• Collaborate with AI engineers, technical architects and cybersecurity specialists to review findings and recommend improvements.
• Support iterative testing and reassessment following system enhancements.
• Produce clear benchmarking reports, technical assessments and assurance evidence.
• Contribute to technical reviews and evaluation activities at key project milestones.

Essential Requirements

• Demonstrable professional experience in AI/ML evaluation, benchmarking, model validation or technical AI assurance.
• Strong understanding of contemporary AI technologies, including large language models (LLMs), generative AI and AI-enabled analytical systems.
• Experience designing and executing structured evaluation methodologies.
• Proven ability to establish quantitative and qualitative performance measures.
• Experience assessing AI systems for accuracy, reliability, robustness and consistency.
• Understanding of AI-related risks, including hallucinations, model limitations, data quality issues and security vulnerabilities.
• Ability to interpret technical findings and translate evaluation results into practical recommendations.
• Strong analytical thinking, independent judgement and attention to detail.
• Excellent technical documentation, reporting and stakeholder communication skills.
• Ability to work collaboratively with multidisciplinary engineering and security teams.
• Active UK Security Clearance (minimum SC).
• Based in the United Kingdom and able to meet applicable project security requirements.

Desirable Experience

• Previous experience supporting UK Defence, government, national security or other security-sensitive technology programmes.
• Experience evaluating AI-enabled decision-support or analytical systems.
• Familiarity with retrieval-augmented generation (RAG), AI agents and multi-component AI architectures.
• Practical experience with AI testing tools, evaluation frameworks and automated benchmarking techniques.
• Knowledge of adversarial testing, AI red teaming, model robustness and security evaluation.
• Understanding of responsible AI, model governance and risk management.
• Familiarity with relevant standards and frameworks, including NIST AI RMF and ISO/IEC 42001.
• Experience contributing to technical assurance, independent verification and validation, or formal acceptance testing.
• Background in operational research, complex systems evaluation or high-assurance technology environments.