Simon Willison’s Weblog

How to stop AI’s “lethal trifecta” (via) This is the second mention of the lethal trifecta in the Economist in just the last week! Their earlier coverage was Why AI systems may never be secure on September 22nd - I wrote about that here, where I called it "the clearest explanation yet I've seen of these problems in a mainstream publication".

I like this new article a lot less.

It makes an argument that I mostly agree with: building software on top of LLMs is more like traditional physical engineering - since LLMs are non-deterministic we need to think in terms of tolerances and redundancy:

The great works of Victorian England were erected by engineers who could not be sure of the properties of the materials they were using. In particular, whether by incompetence or malfeasance, the iron of the period was often not up to snuff. As a consequence, engineers erred on the side of caution, overbuilding to incorporate redundancy into their creations. The result was a series of centuries-spanning masterpieces.

AI-security providers do not think like this. Conventional coding is a deterministic practice. Security vulnerabilities are seen as errors to be fixed, and when fixed, they go away. AI engineers, inculcated in this way of thinking from their schooldays, therefore often act as if problems can be solved just with more training data and more astute system prompts.

My problem with the article is that I don't think this approach is appropriate when it comes to security!

As I've said several times before, In application security, 99% is a failing grade. If there's a 1% chance of an attack getting through, an adversarial attacker will find that attack.

The whole point of the lethal trifecta framing is that the only way to reliably prevent that class of attacks is to cut off one of the three legs!

Generally the easiest leg to remove is the exfiltration vectors - the ability for the LLM agent to transmit stolen data back to the attacker.