Ship fast, regret later: A year of breaking AI applications

Organisations across every sector have moved rapidly to embed AI capabilities into their applications and workflows, often faster than their security practices have adapted. This session presents findings from a year of penetration testing engagements targeting AI-enabled applications, drawing out the patterns, and recurring failure modes that have emerged as teams rush to ship AI features without fully understanding the technology and the new attack surfaces they are introducing.

Specifically, this session will examine how:

• Traditional application security assumptions break down when model behaviour, prompt handling, and inference enter the threat model
• Common implementation shortcuts, taken under delivery pressure, create exploitable conditions that developers and security teams are not yet trained to recognise
• The boundary between application logic and model behaviour is poorly understood, inconsistently enforced, and frequently exploitable
• Existing security testing methodologies are lagging behind the reality of what AI-enabled applications expose, and how organisations are currently assessing risk they cannot yet accurately characterise
• The gap between "we used an AI security checklist" and "we tested what the model actually does under adversarial conditions" remains wide and consequential

The session will connect these findings to practical remediation guidance and the emerging discipline of AI-specific security testing, grounding recommendations in real-world engagement experience rather than theoretical frameworks.