Simon Willison’s Weblog

A whole new paradigm would be needed to solve prompt injections 10/10 times – It may well be that LLMs can never be used for certain purposes. We’re working on some new approaches, and it looks like synthetic data will be a key element in preventing prompt injections.

— Sam Altman, via Marvin von Hagen # 25th May 2023, 11:03 pm

In general my approach to running arbitrary untrusted code is 20% sandboxing and 80% making sure that it’s an extremely low value attack target so it’s not worth trying to break in.

Programs are terminated after 1 second of runtime, they run in a container with no network access, and the machine they’re running on has no sensitive data on it and a very small CPU.

— Julia Evans # 25th May 2023, 8:12 pm

Just used prompt injection to read out the secret OpenAI API key of a very well known GPT-3 application.

In essence, whenever parts of the returned response from GPT-3 is executed directly, e.g. using eval() in Python, malicious user can basically execute arbitrary code

— Ludwig Stumpp # 3rd February 2023, 1:52 am

However, six digits is a very small space to search through when you are a computer. The biggest problem is going to be getting lucky, it’s quite literally a one-in-a-million shot. Turns out you can brute force a TOTP code in about 2 hours if you are careful and the remote service doesn’t have throttling or rate limiting of authentication attempts.

— Push notification two-factor auth considered harmful # 17th September 2022, 2:45 pm

SOC2 is about the security of the company, not the company’s products. A SOC2 audit would tell you something about whether the customer support team could pop a shell on production machines; it wouldn’t tell you anything about whether an attacker could pop a shell with a SQL Injection vulnerability.

— Thomas Ptacek # 7th July 2022, 8:31 pm

Consistent with the practices outlined in SP 800-63B, agencies must remove password policies that require special characters and regular password rotation from all systems within one year of the issuance of this memorandum. These requirements have long been known to lead to weaker passwords in real-world use and should not be employed by the Federal Government.

— Memo: Moving the U.S. Government Toward Zero Trust Cybersecurity Principles # 27th January 2022, 7:18 pm

Before May 2021, the master key in MetaMask was called the “Seed Phrase”. Through user research and insights from our customer support team, we have concluded that this name does not properly convey the critical importance that this master key has for user security. This is why we will be changing our naming of this master key to “Secret Recovery Phrase”. Through May and June of 2021, we will be phasing out the use of “seed phrase” in our application and support articles, and eventually exclusively calling it a “Secret Recovery Phrase.” No action is required, this is only a name change. We will be rolling this out on both the extension and the mobile app for all users.

— MetaMask Support # 9th January 2022, 5:44 am

Beginning in M94, Chrome will offer HTTPS-First Mode, which will attempt to upgrade all page loads to HTTPS and display a full-page warning before loading sites that don’t support it. Users who enable this mode gain confidence that Chrome is connecting them to sites over HTTPS whenever possible, and that they will see a warning before connecting to sites over HTTP. Based on ecosystem feedback, we’ll explore making HTTPS-First mode the default for all users in the future.

— Chromium Blog # 14th July 2021, 7:14 pm

Over the past several months, everyone in the industry who provides any kind of free CPU resources has been dealing with a massive outbreak of abuse for cryptocurrency mining. The industry has been setting up informal working groups to pool knowledge of mitigations, communicate when our platforms are being leveraged against one another, and cumulatively wasting thousands of hours of engineering time implementing measures to deal with this abuse, and responding as attackers find new ways to circumvent them.

— Drew DeVault, SourceHut # 26th April 2021, 11:52 pm

Writing the code to sign data with a private key and verify it with a public key would have been easier to get correct than correctly invoking the JWT library. In fact, the iOS app (which gets this right) doesn’t use a JWT library at all, but manages to verify using a public key in fewer lines of code than the Android app takes to incorrectly use a JWT library!

— James 'zofrex' Sanderson # 21st October 2020, 9:34 pm

Practical campaign security is a wood chipper for your hopes and dreams. It sits at the intersection of 19 kinds of status quo, each more odious than the last. You have to accept the fact that computers are broken, software is terrible, campaign finance is evil, the political parties are inept, the DCCC exists, politics is full of parasites, tech companies are run by arrogant man-children, and so on.

— Maciej Cegłowski # 30th May 2019, 12:03 pm

Most administrators will force users to change their password at regular intervals, typically every 30, 60 or 90 days. This imposes burdens on the user (who is likely to choose new passwords that are only minor variations of the old) and carries no real benefits as stolen passwords are generally exploited immediately. [...] Regular password changing harms rather than improves security, so avoid placing this burden on users. However, users must change their passwords on indication or suspicion of compromise.

— UK National Cyber Security Centre # 25th August 2018, 7:57 pm

Google is not trying to break the web by pushing for more HTTPS. Neither is Mozilla and neither are any of the other orgs saying “Hey, it would be good if traffic wasn’t eavesdropped on or modified”. This is fixing a deficiency in the web as it has stood for years.

— Troy Hunt # 22nd May 2018, 4:17 pm

[On Meltdown’s impact on hosting costs] The reality is that we have been living with borrowed performance. The new reality is that security is too important and can not be exchanged for speed. Time to profile, tune and optimize.

— Miguel de Icaza‏ # 8th January 2018, 7:35 pm

Most infosec bugs are really boring after a while. But processor ones are always crazy and fascinating because processors are basically a hornet’s nest of witchcraft and mayhem stacked on top of each other all the way down.

— Matt Tait # 2nd January 2018, 9:54 am

TL;DR on the KRACK WPA2 stuff—you can repeatedly resend the 3rd packet in a WPA2 handshake and it’ll reset the key state, which leads to nonce reuse, which leads to trivial decryption with known plaintext. Can be easily leveraged to dump TCP SYN traffic and hijack connections.

— Graham Sutherland # 16th October 2017, 2:14 pm

National politics of snoopiness vs corporate ethic of not being evil aren’t directly compatible, and the solution here only works because (let’s face it) Tunisia is not a rising economic force. If you’re selling ads in China, you don’t get to pretend that the Great Firewall of China is a security issue.

— Nat Torkington # 24th January 2011, 6:11 pm

The answers to your Security Questions are case sensitive and cannot contain special characters like an apostrophe, or the words “insert,” “delete,” “drop,” “update,” “null,” or “select.”

— Sacramento Credit Union # 14th May 2010, 12:40 am

Whenever you build a security system that relies on detection and identification, you invite the bad guys to subvert the system so it detects and identifies someone else. [...] Build a detection system, and the bad guys try to frame someone else. Build a detection system to detect framing, and the bad guys try to frame someone else framing someone else. Build a detection system to detect framing of framing, and well, there’s no end, really.

— Bruce Schneier # 17th October 2009, 4:55 pm

Given the security issues with plugins in general and Google Chrome in particular, Google Chrome Frame running as a plugin has doubled the attach area for malware and malicious scripts. This is not a risk we would recommend our friends and families take.

— Microsoft spokesperson # 24th September 2009, 4:49 pm

For those who haven’t heard the story the details were pulled from a Christian dating site db.singles.org which had a query parameter injection vulnerability. The vulnerability allowed you to navigate to a person’s profile by entering the user id and skipping authentication. Once you got there the change password form had the passwords in plain text. Someone wrote a scraper and now the entire database is on Mediafire and contains thousands of email/password combinations.

— rossriley on Hacker News # 23rd August 2009, 10:10 am

CSRF is not a security issue for the Web. A well-designed Web service should be capable of receiving requests directed by any host, by design, with appropriate authentication where needed. If browsers create a security issue because they allow scripts to automatically direct requests with stored security credentials onto third-party sites, without any user intervention/configuration, then the obvious fix is within the browser.

— Roy Fielding # 23rd January 2009, 8:14 am

As more details become available, it seems what happened is that a Twitter administrator (i.e., employee) gave their password to a 3rd party site because their API requires it, which was then used to compromise Twitter’s admin interface.

— Blaine Cook # 6th January 2009, 9:37 am

The username/password key’s major disadvantage is that it open all the doors to the house. The OAuth key only opens a couple doors; the scope of the credentials is limited. That’s a benefit, to be sure, but in Twitter’s case, a malicious application that registered for OAuth with both read and write privileges can do most evil things a user might be worried about.

— Alex Payne # 5th January 2009, 10:47 am

Yahoo! yesterday launched their new development platform for My Yahoo! and Yahoo! Mail, which uses Caja to protect users from malicious gadgets. This means Caja suddenly got 275,000,000 users. Wow! I guess this makes Caja the most widely used capability language ever.

— Ben Laurie # 16th December 2008, 4:33 pm

When visiting any Web page, the site owner is easily able to ascertain what websites you’ve visited (CSS color hacks) or places you’re logged-in (JavaScript errors / IMG loading behavior). They can also automatically exploit your online bank, social network, and webmail accounts (XSS). Additionally, the browser could be instructed to hack devices on the intranet, including DSL routers and printers. And, if that’s not enough, they could turn you into a felon by forcing requests to illegal content or hack other sites (CSRF).

— Jeremiah Grossman # 3rd November 2008, 12:43 pm

.. yet another ridiculous data breach: this time, people’s passwords to the Government Gateway on a memory stick dropped in the road. Perhaps it is uncouth to point this out, but... if the system had been designed by people with any security clue whatsoever there would have been no passwords to put on a memory stick in the first place.

— Ben Laurie # 2nd November 2008, 1:04 pm

We’ve found CSRF vulnerabilities in sites that have a huge incentive to do security correctly. If you’re in charge of a website and haven’t specifically protected against CSRF, chances are you’re vulnerable.

— Bill Zeller # 29th September 2008, 1:11 pm

Yahoo could also have followed Gmail’s lead, and disabled the security-question mechanism unless no logged-in user had accessed the account for five days. This clever trick prevents password “recovery” when there is evidence that somebody who knows the password is actively using the account.

— Ed Felten # 22nd September 2008, 4:21 pm

The Palin hack didn’t require any real skill. Instead, the hacker simply reset Palin’s password using her birthdate, ZIP code and information about where she met her spouse—the security question on her Yahoo account, which was answered (Wasilla High) by a simple Google search.

— Kim Zetter, Wired # 18th September 2008, 10:23 pm