Simon Willison’s Weblog

33 items tagged “s3”

For a Django application, deployed on Heroku, what are my options for storing user-uploaded media files?

S3 is really a no-brainer for this, it’s extremely inexpensive, very easy to integrate with and unbelievably reliable. It’s so cheap that it will be practically free for testing purposes (expect to spend pennies a month on it).

[... 88 words]

The excess capacity story is a myth. It was never a matter of selling excess capacity, actually within 2 months after launch AWS would have already burned through the excess Amazon.com capacity.  Amazon Web Services was always considered a business by itself, with the expectation that it could even grow as big as the Amazon.com retail operation.

Werner Vogels # 5th January 2011, 3:13 pm

Amazon S3: Versioning Proposal. The us-west-1 S3 bucket region now optionally supports versioning—once enabled on a bucket, all previous versions of keys will be preserved. # 24th January 2010, 1:38 pm

OpenStreetMap Rendering Database. Amazon have added an OpenStreetMap snapshot as a public data set, thanks to some smart prompting by Jeremy Dunck. # 10th October 2009, 1:05 pm

App Engine outage postmortem. Interesting peek behind the scenes. The primary cause of the error was a bug in a GFS (Google File System) Master server caused by a MapReduce process sending a malformed filehandle, reminiscent of the error which took down S3 last year. # 9th July 2009, 12:49 pm

AWS Import/Export: Ship Us That Disk! Andrew Tanenbaum said “Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway”, and now you can ship your storage device direct to Amazon and have them load the data in to an S3 bucket for you. # 21st May 2009, 11:22 am

aws—simple access to Amazon EC2 and S3. The best command line client I’ve found for EC2 and S3. “aws put --progress my-bucket-name/large-file.tar.gz large-file.tar.gz” is particularly useful for uploading large files to S3. Written in Perl (with no dependencies), shelling out to curl to do the heavy lifting. # 19th May 2009, 11:38 am

Amazon Elastic MapReduce (via) Hadoop as a service. Basically a web based GUI around Hadoop—you could roll this yourself on EC2 but for a small markup on regular EC2 prices you get to avoid the extra work setting everything up. Data processing scripts can be written in Java, Ruby, Perl, Python, PHP, R, or C++ and are loaded in to S3 before firing off the job. # 2nd April 2009, 10:25 am

Oscars 2009: the interactive results | guardian.co.uk. My latest project for the Guardian, put together on very short notice. Updates live as the results are announced, and allows Twitter users to vote on their favourite for each category by sending a specially formatted message to @guardianfilm—jQuery and Ajax polling against S3 under the hood. # 23rd February 2009, 2:19 am

How Tarsnap uses Amazon Web Services (via) Useful case study, including some thoughts on SimpleDB. # 14th December 2008, 7:35 pm

Amazon SimpleDB a complete flop? Terry asks if anyone is actually using SimpleDB (related Google searches indicate not, and I’ve personally not heard of anyone using it despite plenty of usage of S3 and EC2). One factor might be that lock-in to EC2 and S3 is pretty small, but if you rely on SimpleDB you’ll need to rewrite your entire application to escape. # 2nd December 2008, 10:17 am

Amazon CloudFront. The Amazon CDN front end for S3 has launched. Traffic is 2 cents per GB more than S3. I’d like to see a price comparison with existing CDNs; I have a hunch it’s an order of magnitude less expensive. # 18th November 2008, 2:37 pm

backup_to_s3.py. I wrote Yet Another S3 backup script today. It’s a thin wrapper about boto that doesn’t do anything particularly impressive, but it fits my brain. # 21st September 2008, 6:51 pm

We’re Never Content. Amazon will be releasing a proper edge caching CDN on top of S3 “before the end of the year”. # 18th September 2008, 12:30 pm

Amazon S3 Availability Event: July 20, 2008. Don’t let the newspeak put you off; this is an honest and informative description of the bug that took down S3 last Sunday, although it does include the world’s longest way of saying “we turned it off and on again”. # 27th July 2008, 5:42 pm

Django File Uploads (via) Nearly two years in the making, Django’s file upload capacity has received a major (and backwards incompatible) upgrade. Previously, files were uploaded by default in to RAM—now, files larger than 2.5MB are streamed to a temporary file and extensive hooks are provided to customise where they end up—streaming to S3, for example. # 1st July 2008, 5 pm

Browser Uploads to S3 using HTML POST Forms. I didn’t know you could do this: create a regular HTML form that gives people permission to upload direct to your own S3 bucket, using a signed JSON policy statement in a hidden form field to prevent third parties from abusing your S3 account. # 27th June 2008, 12:11 pm

so-you-wanna-see-an-image (via) WordPress.com use Amazon S3 to store images (presumably to save having to create a massive scalable redundant filesystem themselves) but the images are served via a load balanced memcached / varnishd caching system that they control. # 1st May 2008, 10:13 am

Amazon takes EC2 to the next level with persistent storage volumes. You can store a snapshot of a storage volume to S3 with a single API call, making backups trivial. # 14th April 2008, 8:04 am

Amazon SimpleDB overview. Attribute values are limited to 1,024 bytes; Amazon suggest that you store larger fields in S3 and use SimpleDB to query metadata about those objects. # 14th December 2007, 11:39 am

LastGraph. Now Available. Andrew Godwin has relaunched his LastGraph Last.fm graphing application. The new version is built on Django and S3 and uses Andrew’s Graphication graphing library based on Cairo. # 15th October 2007, 10:02 pm

Amazon Gets an SLA (But I Still Can’t Use It). “Ontario’s Freedom of Information and Protection of Privacy Acts (FIPPA) don’t allow me to store sensitive information (e.g., students’ work) in jurisdictions that permit secret warrants, like those mandated by the USA PATRIOT Act.” # 9th October 2007, 3 pm

Amazon S3 Service Level Agreement (via) Went in to effect on the 1st of October. Promises 99.9% uptime over a monthly billing cycle or you get “service credits” towards future S3 payments. # 9th October 2007, 12:52 am

Interview with Leah Culver: The Making of Pownce. Django + Perlbal + S3 + AIR. # 7th July 2007, 10:05 pm

Processing Web Documents using Alexa Web Search, Amazon S3 and Amazon EC2. I’m not sure when it happened, but Alexa Web Search can be hooked in to EC2 now—presumably with free bandwidth between the two. # 1st July 2007, 7:19 pm

Mass Video Conversion Using AWS. How to use S3, SQS, EC2, ffmpeg and some Python to bulk convert videos with Amazon Web Services. # 3rd April 2007, 11:44 pm

boto. Python library for accessing Amazon’s S3, SQS and EC2 Web Services, with excellent documentation. # 11th February 2007, 12:17 am

An S3 AJAX Wiki. Les continues to innovate against S3. # 22nd April 2006, 7:09 pm

How I’m using Amazon S3 to serve media files. Adrian’s saving server overhead on ChicagoCrime by serving media from S3. # 7th April 2006, 6:51 pm

JavaScript apps with read/write access to S3. JS apps hosted on S3 could read and write to the store. # 4th April 2006, 9:33 am