Research & Landscape Analysis

• Conduct structured literature reviews of AI safety and security research, including reports from AI safety institutes, evaluation labs, and research organisations.
• Track frontier and open-weight model releases and assess their safety and security implications.
• Monitor emerging research on evaluation, red-teaming, robustness, interpretability, alignment, AI security, RAG, and agentic systems.
• Assess relevant evaluation frameworks, benchmarks, attack methodologies, and testing tools.
• Contribute to the Global & Regional AI Safety Landscape briefing and internal knowledge base.

Technical Testing & Evaluation

• Design and implement evaluation pipelines using open-source frameworks and custom Python tooling to test publicly accessible models and AI-enabled products.
• Build and maintain test scripts, model/API integrations, structured test cases, scoring rubrics, automated grading, and reporting pipelines.
• Test key risk areas including prompt injection, indirect injection, jailbreaks, data leakage/PII exposure, hallucination, adversarial robustness, bias/fairness, over- and under-refusal, and agent/tool-use safety.
• Evaluate application-level risks across RAG, system prompts, tool/function calling, external APIs, memory/state, and agentic workflows.
• Conduct structured adversarial testing and red-teaming, including multi-turn and adaptive attacks where appropriate.
• Develop reusable attack libraries, evaluation datasets, edge cases, and regression test suites.
• Run comparative benchmarking across open-weight and commercial models and products, combining quantitative scorecards with qualitative failure analysis.

Evaluation Methodology & Failure Analysis

• Design rigorous and reproducible evaluations covering objectives, test-set construction, sampling, baselines, scoring criteria, assumptions, and limitations.
• Apply appropriate quantitative and statistical methods, including repeated trials, uncertainty analysis, and error analysis.
• Assess the reliability and limitations of automated evaluation, including LLM-as-judge approaches and human-evaluation agreement where relevant.
• Investigate, reproduce, classify, and document failures and identify likely model-, prompt-, data-, retrieval-, tool-, or application-level causes.
• Assess severity and prioritise findings based on likelihood, exploitability, impact, and potential downstream consequences.

Automation, Regression & Mitigation Testing

• Develop automated safety regression testing to detect behavioural changes across model versions, prompts, fine-tuning, RAG configurations, tools, and application releases.
• Maintain evaluation pipelines and results repositories for longitudinal tracking and benchmarking.
• Evaluate proposed safeguards through pre- and post-mitigation testing and verify that fixes do not introduce new failure modes.
• Apply appropriate version control, logging, configuration management, and reproducibility practices.

Advisory, Training & Knowledge Sharing

• Support AI safety/security readiness assessments, gap analyses, model risk registers, and incident-response playbooks.
• Translate testing results into practical recommendations for AI risk identification, evaluation, monitoring, and management.
• Contribute technical content to workshops and hands-on AI safety testing modules.
• Present methodologies and findings at relevant conferences, working groups, and technical dialogues.