Items tagged generativeai, security in Dec, 2023
Filters: Year: 2023 × Month: Dec × generativeai × security × Sorted by date
OpenAI Begins Tackling ChatGPT Data Leak Vulnerability (via) ChatGPT has long suffered from a frustrating data exfiltration vector that can be triggered by prompt injection attacks: it can be instructed to construct a Markdown image reference to an image hosted anywhere, which means a successful prompt injection can request the model encode data (e.g. as base64) and then render an image which passes that data to an external server as part of the query string.
Good news: they’ve finally put measures in place to mitigate this vulnerability!
The fix is a bit weird though: rather than block all attempts to load images from external domains, they have instead added an additional API call which the frontend uses to check if an image is “safe” to embed before rendering it on the page.
This feels like a half-baked solution to me. It isn’t available in the iOS app yet, so that app is still vulnerable to these exfiltration attacks. It also seems likely that a suitable creative attack could still exfiltrate data in a way that outwits the safety filters, using clever combinations of data hidden in subdomains or filenames for example. # 21st December 2023, 4:10 am
Recommendations to help mitigate prompt injection: limit the blast radius
I’m in the latest episode of RedMonk’s Conversation series, talking with Kate Holterhoff about the prompt injection class of security vulnerabilities: what it is, why it’s so dangerous and why the industry response to it so far has been pretty disappointing.
[... 539 words]Announcing Purple Llama: Towards open trust and safety in the new world of generative AI (via) New from Meta AI, Purple Llama is “an umbrella project featuring open trust and safety tools and evaluations meant to level the playing field for developers to responsibly deploy generative AI models and experiences”.
There are three components: a 27 page “Responsible Use Guide”, a new open model called Llama Guard and CyberSec Eval, “a set of cybersecurity safety evaluations benchmarks for LLMs”.
Disappointingly, despite this being an initiative around trustworthy LLM development,prompt injection is mentioned exactly once, in the Responsible Use Guide, with an incorrect description describing it as involving “attempts to circumvent content restrictions”!
The Llama Guard model is interesting: it’s a fine-tune of Llama 2 7B designed to help spot “toxic” content in input or output from a model, effectively an openly released alternative to OpenAI’s moderation API endpoint.
The CyberSec Eval benchmarks focus on two concepts: generation of insecure code, and preventing models from assisting attackers from generating new attacks. I don’t think either of those are anywhere near as important as prompt injection mitigation.
My hunch is that the reason prompt injection didn’t get much coverage in this is that, like the rest of us, Meta’s AI research teams have no idea how to fix it yet! # 8th December 2023, 6:36 am