Breaking AI systems like an attacker: Practical threat modelling for LLMs

The rapid adoption of AI systems, particularly those incorporating large language models (LLMs), has introduced a new class of security challenges that are not well addressed by traditional threat modelling approaches. Unlike conventional applications, AI systems operate through probabilistic outputs, dynamic context handling, and complex interaction patterns, creating novel and difficult-to-formalise attack surfaces.

Current security practices often attempt to retrofit existing models onto AI systems, resulting in incomplete or ineffective threat assessments. For example, standard data flow analysis may fail to capture how prompt manipulation can alter system behaviour, or how indirect interactions can lead to unintended data exposure.

This presentation addresses these gaps by introducing a practical, adversary-led approach to threat modelling AI systems, grounded in established methodologies such as PASTA, but adapted to the unique characteristics of AI-driven architectures.

The session begins by redefining key elements of threat modelling in the context of AI, including:

• What constitutes an "asset" in an AI system,
• How trust boundaries shift when models generate outputs dynamically, and
• How attacker objectives evolve in AI-enabled environments.

The presentation then demonstrates how to:

• Identify AI-specific attack surfaces, including prompt injection and model manipulation,
• Model attacker interaction pathways across system components, and
• Evaluate the impact of these attacks on confidentiality, integrity, and availability.

A detailed, worked example of an AI-enabled system will illustrate how these threats manifest in practice. This includes mapping attack scenarios to concrete mitigation strategies, such as input validation approaches, architectural controls, and monitoring mechanisms.

The approach is designed to be directly applicable, enabling practitioners to extend existing threat modelling practices to AI systems without requiring entirely new frameworks.

By focusing on adversary behaviour and real-world attack patterns, this session provides a structured and practical pathway for securing AI systems in operational environments.