Feed Sign in with OpenID OpenID

Simon Willison’s Weblog

Fighting RFCs with RFCs

Google’s recently released Web Accelerator apparently has some scary side-effects. It’s been spotted pre-loading links in password-protected applications, which can amount to clicking on every “delete this” link — bypassing even the JavaScript prompt you carefully added to give people the chance to think twice.

"Aah," I hear you cry, "but RFC 2616 clearly states that you shouldn’t perform state changing operations with a GET or HEAD method!"

In particular, the convention has been established that the GET and HEAD methods SHOULD NOT have the significance of taking an action other than retrieval.

I’ll see your RFC 2616 and raise you an RFC 2119:

SHOULD NOT This phrase, or the phrase “NOT RECOMMENDED” mean that there may exist valid reasons in particular circumstances when the particular behavior is acceptable or even useful, but the full implications should be understood and the case carefully weighed before implementing any behavior described with this label.

Hiding your dangerous delete links behind an authentication scheme is a perfectly acceptable compromise. Web Accelerator is B.A.D.

Update: Be sure to read the excellent discussion brewing in the comments. Hiding behind authentication may not be as acceptable a compromise as I had first thought.

Update 2: If you haven’t been following the comments, I’ve had a change of heart. Even in the absence of Web Accelerator, hiding behind authentication leaves your application open to some very nasty security vulnerabilities (malicious pages can piggy-back your session and cause havoc making dangerous GET calls). I still think the RFC language covers people who thought long and hard before implementing a dangerous GET, but if you haven’t thought about security and accelerating caching proxies such as Web Accelerator you haven’t been thinking hard enough.

Update 3: So, it turns out using POST is no defence at all against CSRF attacks. I’ve been learning a whole bunch of interesting stuff this evening.

Posted 6th May 2005 at 8:39 pm

This is Fighting RFCs with RFCs by Simon Willison, posted on 6th May 2005.

Tagged , , , , ,

View blog reactions

Next: Eurovision scores

Previous: Giving away the index

28 comments

  1. Agreed. Nice play...

    Adam Michela - 6th May 2005 20:46 - #

  2. "Hiding your dangerous delete links behind an authentication scheme is a perfectly acceptable compromise." That sounds awfully dangerous to me. What happens when an already-authenticated user visits a malicious web page that causes their browser to request one of those dangerous delete URLs? I don't think the above statement takes "the full implications" into account.

    Matt Brubeck - 6th May 2005 20:47 - #

  3. It's really not a "perfectly acceptable compromise," Simon. Just because something is sitting behind an authentication scheme does not mean that the semantics of "SHOULD NOT" change. At the time the decision was made to use GET in a non-safe/non-idemponent manner, the "full implications" were different. Now we have services that are using HTTP for what it's worth and we need to reconsider the implications. Accelerated caching proxies are not limited to google. I can't even use the damn thing and I probably wouldn't if I could but the concept is legitimate given some basic adherence to best practices. Attempting GET/HEAD requests on links pulled from a page is something that could be extremely useful in many scenarios.

    What I've been trying to figure out is whether there is any guarantee that a link found in an a/href attribute is guaranteed to be describing a GETable link. I can't think of any case where this wouldn't be the case but it occurs to me that blindly assuming that any link you come across should be resolved with GET is a bad assumption.

    I think both Google and the rampant misuse of GET in non-safe/non-idempotent situations are serious issues. Everyone's wrong here. Please don't encourage this blatant misuse of a very fundamental aspect of web interoperability.

    Ryan Tomayko - 6th May 2005 21:00 - #

  4. Matt: that's a very good point, and one I hadn't really considered. The malicious page would have to be quite specifically targetted, but you could definitely do a lot of damage that way. You could defend against that attack by encoding randomly generated strings in to the links, created in such a way as to make "guessing" them unrealistic.

    Simon Willison - 6th May 2005 21:05 - #

  5. the full implications should be understood and the case carefully weighed before implementing any behavior described with this label.

    The full implications, in this case, are that you can't assume GET is a deliberate link activation by a user, so even if you break the SHOULD, it would be insane to do so for destructive actions like deletes.

    RFC 2616 also says:

    The important distinction here is that the user did not request the side-effects, so therefore cannot be held accountable for them.

    It's difficult to think of a scenario where it's acceptable to change state even when the user hasn't requested it, but that doesn't mean you can simply disregard the SHOULD safely.

    People are talking about this as if it's a new problem. I remember similar applications in the mid-90s that pre-fetched linked pages, and they uncovered similar problems.

    If a web developer ignores the SHOULD in the face of historical precedents (and, IMO, common sense), then they have forfeited any right to complain when it breaks, and should take responsibility instead of pointing the finger at the application exposing the error.

    Jim - 6th May 2005 21:47 - #

  6. Jim and Ryan: you both make excellent arguments. I've used delete links in the past knowing full well that they're "bad", but I'd always assumed that authenticated web applications could get away with that kind of abuse without any harmful side effects. I'm obviously not alone - support for this kind of thing is built in to Rails (the framework has a :confirm attribute for easily adding JavaScript confirmation to dangerous links) and is evident in a huge number of web applications.

    The thing is, we're not doing this out of ignorance. GET links have a number of genuine advantages over POST buttons from a usability point of view - a classic example is that I can bulk delete items with GET links by opening each link in turn in a background tab. I'm convinced now that password protected web applications are not the special case I had thought they were, but it's a tough pill to swallow for the open-in-new-tab reason alone. Maybe Firefox needs a "POST this form in a new tab in the background" ability.

    Simon Willison - 6th May 2005 21:56 - #

  7. Amusingly, Google itself is guilty of this bad practice.

    Matt Brubeck - 6th May 2005 22:16 - #

  8. a classic example is that I can bulk delete items with GET links by opening each link in turn in a background tab.

    That sounds like a shortcoming of the web application that is covered up by a feature of the UA, to be honest. Perhaps understandable when things like GWA were niche products, but not now a big name like Google is doing it.

    Wouldn't that bulk deletion be more useful if, say, it was implemented as buttons to the side of each item, that activated XMLHttpRequest to delete while staying in the same place on the same page, simultaneously removing the item from the DOM?

    Obviously it's more work than a set of links, but that seems to me to be a corner that people felt safe cutting until somebody highlighted the problem by doing something perfectly reasonable and according to spec, but nevertheless reacts badly with this deviation from spec.

    Maybe Firefox needs a "POST this form in a new tab in the background" ability.

    I've actually wanted this for a long time, for various reasons.

    Jim - 6th May 2005 22:17 - #

  9. "malicious pages can piggy-back your session and cause havoc making dangerous GET calls"

    Can't you do this as easily for POSTs: with hidden frames that do posts automtically (onload)?

    Julien Couvreur - 6th May 2005 22:26 - #

  10. Julien: indeed you can. I just posted about this to the Rails mailing list. Since their web archive isn't working at the moment, here's my email in full:

    On 6 May 2005, at 22:42, Rick Olson wrote:

> How's this safer.  Sure, you have to be a bit craftier, but I'd think
> you could easily simulate POSTs from some javascript.  Does Rails (or
> web servers in general) protect against this at all?  You can probably
> simulate referral links also.

Rats, you can. I just built a quick proof of concept which adds an  
iframe, then writes the HTML for a form to it and submits it. You  
could include hundreds of those on the page in the exact same way as  
the image trick.

This is the exact problem that the XMLHttpRequest same-domain  
security policy is meant to fix, but it seems all-too-easy to work  
around.

It looks like the only way to build this kind of functionality  
securely is to add a randomly generated hidden form field variable to  
the form, which confirms that the POSTed data came from a form that  
was first retrieved from that site. Some kind of mechanism for doing  
this easily would be an excellent addition to Rails, or indeed any  
other framework.

It's interesting to note that the same trick can be used to secure  
against the GET attack I described previously, which actually weakens  
my earlier argument.
    By "ealier argument", I refer to a message in which I suggested that POST forms had a security advantage over GET in guarding against the malicious third party web page attack.

    Simon Willison - 6th May 2005 23:03 - #

  11. Some of this doesn't jibe well. Opening mass amounts of tabs is not a "shortcoming" of a web application. Web applications are designed to be accessed by a browser. Browsers add (or remove) functionality from web applications. Some web apps, and certainly good designers, take this into account.

    And session hijacking...This is not the only security threat in web applications (cross site scripting, SQL injections, etc) and it is not applicable in every situation. Can you use GWA on your company's internal Project Manager? Session hijacking is not an issue in this situation, yet the same mayhem is unleashed upon your internal web app as is unleashed on a public, authentication based web app like Backpack.

    I think the web is changing and GWA didn't take that into account. Sure, these same types of scenarios (delete links) have been available since the dawn of the web, but how available have they been to Joe Sixpack? Google may have followed the RFC but where does that leave the end users of these new, popular apps? A bit out of luck, it seems.

    Chris - 6th May 2005 23:03 - #

  12. Just read your newest comment, Simon. Despite my above comment, I completely agree that Rails should have this type of "formkey" built in. The more security, the better.

    Also, I've seen this question floating around a bit and am unsure if everyone knows the answer. Checking the referring site is NOT safe! The referring site URL is sent to the sever in the HTTP header. This means it can easily be spoofed by someone who knows what they're doing. Chris Shiflett has some words on the subject.

    Chris - 6th May 2005 23:09 - #

  13. Referrer spoofing for CSRF attacks (that's apparently what this kind of thing is called) is actually a lot harder than you would think - CSRF attacks only work because they trick a user's browser which is already logged in in to making a malicious GET or POST request, and there's no way for a malicious page to modify the referrer (at least not without using XMLHttpRequest, which wouldn't work due to the same-domain security policy).

    That said, referrer checking is still a bad idea because it's relatively common for security-sensitive users to disable referrers in their browser - or for companies to run proxies that strip referrers from ALL outgoing HTTP traffic.

    Simon Willison - 6th May 2005 23:18 - #

  14. That's what I think too. I didn't know that these hidden form fields where called "formkeys" though. Good to have a name to refer to a common pattern (although it applies beyond forms).

    I hope that as frameworks start including formkey support as discussed above, not all pages and forms will get protected that way ("The more security, the better"). The W3C guideline should be enough for most of the GET vs. POST issues (so that crawlers and GWA wouldn't screw stuff by accident). But formkeys should really only be used on forms that require some security.

    Reset password pages and delete email forms should be protected with formkeys, but I hope not all forms will get locked. Otherwise, un-official cross-site integration functionalities (bookmarklet, Greasemonkey user scripts, ...) and other user-driven innovations are going to suffer big time.

    Julien Couvreur - 6th May 2005 23:33 - #

  15. I don't think Greasemonkey scripts will suffer that much, because they will have access to the formkey (they can read it through the DOM). The same goes for screen scrapers etc - they'll just have to take the extra step of loading up the "form" page and scraping it for the necessary key.

    Simon Willison - 6th May 2005 23:35 - #

  16. Opening mass amounts of tabs is not a "shortcoming" of a web application.

    My point was that if mass deletion is a common use-case of a particular web application, then the web application should make it easy to do so without falling back on a workaround, particularly one that is only available in some browsers.

    Browsers add (or remove) functionality from web applications. Some web apps, and certainly good designers, take this into account.

    Having to open a bunch of extraneous tabs just to delete multiple items efficiently doesn't sound like good design to me. It sounds like a workaround that only expert users would think of.

    Sure, these same types of scenarios (delete links) have been available since the dawn of the web, but how available have they been to Joe Sixpack?

    I remember these types of web accelerators were quite popular in demo form on magazine cover disks back when everybody was on dialup. They certainly weren't uncommon.

    Jim - 6th May 2005 23:39 - #

  17. WordPress has had referrer checking for CSRF protection for a while now and it's been working great. There have been some bumps, but along the way we've developed some pretty comprehensive documentation for enabling referrers: http://codex.wordpress.org/Enable_Sending_Referrer s

    Matt - 7th May 2005 00:14 - #

  18. I don't think Greasemonkey scripts will suffer that much, because they will have access to the formkey (they can read it through the DOM).

    Actually, I have a couple of Greasemonkey scripts that cause POSTs to other domains. If these other domains were to reject the POST because the formkey is missing, these script are dead. And there is no formkey in the current page that can be scraped in the DOM...

    Julien Couvreur - 7th May 2005 00:19 - #

  19. Julien: is there any reason you can't make an extra GM_xmlhttpRequest call to pull in the previous page from that domain and scrape the formkey from it before making your POST?

    Simon Willison - 7th May 2005 00:34 - #

  20. Yes, most people don't immediately realize all teh consequences of GET vs. POST. I wrote about it a year ago:
    http://www.sencer.de/article/92/get-considered-har mful-sometimes

    And it was again Jesse, that made me aware there is more to it:
    http://www.sencer.de/article/122/securing-forms-wi th-post-is-not-enough

    Lots of sites/applications still get this wrong.

    Sencer - 7th May 2005 00:41 - #

  21. I didn't think of using GM_xmlhttpRequest. It's a good workaround if needed.

    There is still a problem for bookmarklets though (I have a bookmarklet for bookmarking pages, using a hidden popup and a post), but GM is kind of replacing bookmarklets anyways...

    Julien Couvreur - 7th May 2005 05:37 - #

  22. I consider sending cookie information as part of a pre-fetched request a security vulnerability in some cases. More information here: Poor web applications and pre-fetch security issues. It is an implementation issue, hiding behind logins does not help at all, Google is just pointing this out the hard way now. Also - when using POST developers should double-check that the same variables can not be populated using GET. You should exclusivly pull the POST variables and filter them yourself. Checking the Referrer does not help much at all either.

    Nik Cubrilovic - 7th May 2005 07:36 - #

  23. Simon, I don't understand why you need to have get links for deletion and open them in multiple tabs.

    If you have a single page with multiple items that can be deleted on it, why aren't there checkboxes for each item with a delete submission that deletes all checked (pending confirmation, presumably)?

    Lach - 9th May 2005 03:07 - #

  24. I believe the "SHOULD NOT" meaning "NOT RECOMMENDED" means that is the rule not to perform any other actions than retrieval. Obviously there are cases, like tracking what a visitor is viewing, which is doing more than just retrieval, hence the absolute "MUST NOT" not being used in RFC 2616. [GWA doesnt break tracking either, I believe, it identifies itself so filtering data should be trivial] Deleting data is definately not suitable for just a GET link, imo, and any application which does this is broken.

    Ren - 11th May 2005 15:20 - #

  25. I think that alot of people here are getting stuck on the details and are missing the big picture.

    The Google Web Accelerator is a lousy idea for a litany of other reasons. Not only can it screw up websites, but it is one of the most blatant privacy violations on the web and makes it difficult for webmasters to know who is visting what parts of their website.

    Brian Duffy - 12th May 2005 14:30 - #

  26. How does it invade my privacy if I choose not to run it? And you never really know who or what is visiting parts of your website anyways, so that can't really be used as a reason. If you've written an web application thats broken, quit passing the buck and fix it.

    Ren - 12th May 2005 18:38 - #

  27. Maybe Firefox needs a "POST this form in a new tab in the background" ability.

    Safari can do this (splat-click the submit button). I never realised Firefox can't until I read your post. It's a bit of a contrived case though, surely?

    Rupert Jabelman - 14th May 2005 01:34 - #

  28. Brian: How much worse is Google Web Accelerator than (to name two examples) Durham University forcing all internal users to surf via a Squid caching proxy, or a school using Microsoft's ISA Server to ensure that all users are using a caching, prefetching proxy for their web access?

    My guess is that the only complaint here is one of scale (more people use GWA than are stuck behind a proxy), to which I'd respond that unlike people stuck behind a proxy, GWA users have chosen to use GWA; most proxy users are forced to use the caching proxy.

    Simon Farnsworth - 31st May 2005 17:18 - #

Comments are closed.

Previously hosted at http://simon.incutio.com/archive/2005/05/06/bad

A django site