Petrov's loop: What good decisions look like when the machine is in the room (v2)

In 1983, the Soviet early-warning system reported five inbound missiles. Lieutenant Colonel Stanislav Petrov was on duty. The system was confident. The protocol was clear. Petrov did not follow it. He reasoned iteratively — a genuine first strike would involve hundreds of missiles; the satellite had known software bugs; the data felt wrong. He returned to the question. He did not commit. His hesitation prevented a nuclear exchange.

We are now building AI systems for security-critical decisions and actively debating whether to remove Petrov's loop — whether to trust the single-shot output, skip the return to the question. The engineering arguments for speed and efficiency are real. So is the risk.

This talk is about what good decisions look like when AI is in the room — grounded in the specific contexts security and product leaders face today.

Three failure modes existing governance frameworks were not designed to detect:

• Rational compliance blindness:​ AI systems trained on rational-agent models will fail during the phase of a crisis when dangerous behaviour looks identical to optimal behaviour. The person who boards the train without knowing they were exposed five days ago is, by the numbers, behaving optimally. The AI flags nothing. The absence of a red flag is the red flag. Governance built on anomaly detection will miss the most dangerous actors — the ones doing everything right within the information available to them.
• Result-laundering:​ When AI produces a correct output through an incorrect process, governance cannot distinguish success from deferred failure. A model that shortcuts to the right answer is not a success — it is an undetected future failure. Governance must audit process fidelity, not just outcomes.
• Complexity as camouflage:​Security architectures elaborate enough that no team fully understands them have not become more secure — they have become unfalsifiable. The Maginot Line was never breached. France went around it in eight days. Complex systems fail in ways that defeat the post-mortem.

Against each failure mode, the talk proposes a concrete operational intervention: how to build Petrov's loop into AI-assisted decisions (the return to the question, the second pass, the named human who signed the call); how to distinguish knowing-that from knowing-how before shipping to production; and the audit test — if three senior engineers cannot produce a complete decision map in one week, the system is not production-ready for security-sensitive contexts, regardless of benchmark performance.

Australia's regulatory context makes this urgent: APRA CPS 234 scales with AI adoption; the reformed Privacy Act's automated decision-making obligations take effect December 2026; ISO 42001 is the new compliance surface security teams must audit intelligently. None of these can be governed from a distance.

The talk closes with a decision framework leaders can implement immediately: what to ask before approving an AI-assisted security system, what accountability looks like when the decision is shared between a model and a human, and what the organisation must be able to say when something goes wrong.