Simon Willison’s Weblog

He notes that one simulated test saw an AI-enabled drone tasked with a SEAD mission to identify and destroy SAM sites, with the final go/no go given by the human. However, having been ‘reinforced’ in training that destruction of the SAM was the preferred option, the AI then decided that ‘no-go’ decisions from the human were interfering with its higher mission – killing SAMs – and then attacked the operator in the simulation.

[UPDATE: This turned out to be a "thought experiment" intentionally designed to illustrate how these things could go wrong.]

— Highlights from the RAeS Future Combat Air & Space Capabilities Summit