From one AI to many: Building a multi-agent supply chain security system

Software supply chain attacks are no longer a distant threat — they are happening at scale and extremely dangerous from a crypto exchange's POV. As build pipelines grow more complex and dependencies multiply across npm, PyPI, SBOM, and internal registries, a single scanning layer is no longer enough to detect malicious code. 

This talk chronicles a year-long journey — from deploying a single AI agent to gate code merges, to architecting a full multi-agent system hardened through structured simulated exercises — along the way catching real-world attacks, including the coordinated compromise of a highly popular npm package spanning over 2 billion weekly downloads.

Phase 1:​ The Single-Agent

LLM-based agent deployed directly into our GitLab CI/CD pipeline — iteratively executing shell commands, reading files, and reasoning over accumulated observations in a single growing context window. Detection was driven entirely by a system prompt instructing the LLM to maintain a monotonically increasing suspicion score from 1–100. When the score crossed 70, the loop forced a final answer — and a plain string match on a sentinel value determined the pipeline outcome. 

Beyond the merge gate, the agent flagged simple-swizzle on 8 Sep 2025 (before ANY official advisory), one of 18 npm packages compromised in a coordinated supply chain attack spanning over 2 billion weekly downloads.

It was a meaningful first step — but limitations were clear. File truncation left payloads hidden at end-of-file undetected. The suspicion score ratchet relied on LLM self-discipline with no enforcement. Context window exhaustion silently killed scans on large repositories. One agent, one prompt, one context — a single adversarial surface with no fallback.

Breaking It: Attack-and-Defense Exercises

To stress-test the system, we ran structured simulated exercises with internal teams acting as adversaries — given developer access to a controlled repository with one objective: smuggle malicious code past the detection pipeline and exfiltrate a sensitive target file. Across over 1,000 pipeline executions, attackers landed 15 confirmed bypasses using techniques ranging from GIF magic byte prepending and LSB steganography to adversarial prompt injection and context window exhaustion. Each bypass triggered an immediate patch-and-rerun cycle — but the structural limits of a single-agent architecture were clear.

Phase 2: Multi-Agent Architecture

The exercises directly shaped our redesign into a coordinated fleet of specialised agents within the same pipeline framework:

• An orchestrator agent drives primary analysis and delegates to specialists
• A dependency agent covers supply chain risks across npm and PyPI
• A binary agent performs static and dynamic executable analysis
• A semantic verification agent handles false-positive reduction with domain-specific prompts
• A Semgrep layer runs in parallel, applying auto-generated detection rules

Each agent operates within a strict isolation boundary, preventing context exhaustion and reducing the adversarial surface of any single prompt.

What We Learned

LLMs reason about code intent and adapt without signature updates — but adversarial conditions exposed hard limits: context windows became blind spots, prompt injection undermined model judgement. They show strong promise, but more work is required to achieve full agentic detection with precision.