Author: Erkuden Rios, TECNALIA
Artificial Intelligence (AI) systems are increasingly targeted by adversarial threats such as data poisoning and model manipulation. The European Union’s AI Act and ENISA’s threat landscape emphasize the need for AI systems to be safe, transparent, and ethically sound. The AI44CYBER solution for AI trustworthiness, TRUST4AI, supports compliance with these regulations and promotes human oversight.
In this post we summarize how TRUST4AI can support AI risk assessment, with a focus on TRUST4AI.Security, which goal is to prevent adversarial attacks on AI systems.
In the context of the research on how to ensure AI trustworthiness, AI4CYBER developed a security risk assessment methodology for AI systems available at: https://ieeexplore.ieee.org/document/10475824 , and at: https://zenodo.org/records/11209106
This is a comprehensive methodology for assessing security risks in AI systems, aiming at ensuring their technical robustness and trustworthiness. As AI technologies become integral to modern digital systems, they also expand the attack surface, necessitating specialized cybersecurity approaches.
The AI4CYBER risk assessment methodology for AI systems builds on top of the KYKLOS 4.01 project’s risk assessment methodology for security and privacy of industrial solutions, which has been specified and extended to address specific AI systems related threats, adversarial testing techniques, and mitigations.
The methodology builds on existing frameworks like:
- NIST’s AI Risk Management Framework (AI RMF)2: Focuses on governance, mapping, measuring, and managing AI risks.
- MITRE ATT&CK3 and MITRE ATLASTM 4: Provide taxonomies and threat intelligence for adversarial AI attacks.
- ENISA’s multi-layer cybersecurity framework5: Offers guidance for AI-specific and sector-specific security practices.
- Cloud Security Alliance’s CSA IoT Security Framework6 which should be used when AI systems include Internet of Thinks (IoT) components.
In summary, AI4CYBER’s methodology consists of two iterative phases:
- Procedural Risk Assessment: In the initial phase, based on external and internal triggers, the potential threats and necessary security controls that should be incorporated into the AI system are identified.
- Security Testing-Based Risk Assessment: In the second phase, a more dynamic approach is taken, utilizing cybersecurity testing techniques, including adversarial testing techniques, to evaluate potential threats and vulnerabilities of the AI system.
The methodology proposes to integrate tools such as TRUST4AI.Security to support the evaluation of AI system risks.
Based on DECIDER tool by CISA7, TRUST4AI.Security supports AML threat discovery and mitigation selection in procedural risk assessment, as well as execution of adversarial tests to validate AI system defences (and implemented mitigations) in security testing-based risks assessment phase.
TRUST4AI.Security integrates with AI4SIM component of AI4CYBER framework which includes two adversarial machine learning simulators: the AAG (Adversarial Attack Generator) for evasion attacks, and the MAIP tool, currently focused on poisoning threats. TRUST4AI.Security integrates also with the Adversarial Robustness Toolbox (ART)8, an open source solution which includes multiple types of AML techniques.
TRUST4AI.Security leverages the AI4CYBER Threat Knowledge Base (KB) that extends SPARTA project’s KB9 with updated taxonomies and mappings to known threats and mitigations. It incorporates findings from recent literature, modernised taxonomies to align with standards like NIST AI 100-2 E202310 on AML and mappings to Tactics, techniques and procedures (TTPs) and mitigations in MITRE ATLASTM.
Conclusion
The AI4CYBER promotes the holistic, agile, and automated approach to AI security risk assessment. The evaluation should be continuous, and it can benefit from automation tools such as TRUST4AI.Security which addresses initial identification of potential AML threats and recommended mitigations, as well as automatic testing of adversarial threats. TRUST4AI.Security integrates with AI4SIM for the simulation of adversarial attacks to test system resilience. By enhancing AI robustness, TRUST4AI.Security contributes to building trustworthy AI systems together with the other two components in AI4CYBER framework addressing AI trustworthiness: TRUST4AI.XAI and TRUST4AI.Fairness.