When generative AI goes to war: Why ethics theatre, human-in-the-loop myths, and weak governance are failing cyber security today at scale

Cyber security is entering a phase where generative AI is no longer a chatbot on a website. It is becoming embedded inside products, workflows, analyst tooling, communications pipelines, and autonomous decision loops. Yet much of today’s GRC thinking still sounds as though we are discussing a university ethics seminar, not an operational contest against adversaries. We are told to apply ethics, keep humans in the loop, and comply with law. Those slogans sound elevated, but in practice they often collapse under pressure.

This talk argues that the root problem is a category error. Many current cyber security guidelines and GRC frameworks treat generative AI as if it were mainly a content generator that must be morally reminded to behave. That is already outdated. Once generative AI gains tools, memory, delegation, and runtime permissions, the problem is no longer prompt quality or value statements. The problem becomes operational control under conflict. Ethics is not a control mechanism. Human review is not a control mechanism. Law is not a runtime control mechanism. At best, these are external constraints. At worst, they become decorative theatre that weakens defenders while hostile groups and states impose no comparable self-restraint.

I will challenge three weak assumptions directly. First, the fantasy that broad ethical language can govern systems in environments where humans do not even share stable values. Second, the lazy belief that human-in-the-loop is a serious safeguard, when in reality human complacency, fatigue, delay, rubber-stamping, and poor situational awareness make that claim fragile in many high-speed settings. Third, the comforting assumption that regulation alone will solve the problem, even though generative AI capability can be exported as services, tokens, workflows, and cross-border infrastructure from regulatory safe havens. In practice, much of current GRC is not designed for adversarial, high-tempo systems.

The talk then turns from criticism to a harder alternative: harness engineering as the real control surface. I will present a model for building powerful generative AI agents that are not neutered into uselessness, but are still constrained where it matters: tool access, action classes, side effects, escalation paths, replayability, budget limits, identity separation, and irreversible operations. The principle is simple: free reasoning, tight runtime cage.

I will ground this in our own practical experience building a generative AI scammer-baiting agent used against fraudulent shopping and investment operations. In that work, the system must sometimes operate near legal and operational boundaries to obtain payment identifiers, contact points, and mule infrastructure from scammers. That requires realism, persistence, and at times controlled deception. But it also requires hard limits so the agent does not drift into harmful conduct such as facilitating money laundering, coaching criminal tradecraft, or taking unauthorised actions. The real engineering challenge is not making the agent polite. It is maximising operational utility without dragging your own organisation into legal, reputational, or safety failure.

This session offers a blunt thesis: weak GRC language is comforting but ineffective. Serious generative AI security needs a doctrine built for adversarial reality, not moral vanity.