Simon Willison’s Weblog

Blogmarks tagged amazon

Filters: Type: blogmark × amazon ×


Amazon’s Away Teams laid bare: How AWS’s hivemind of engineers develop and maintain their internal tech (via) Some interesting insights into how Amazon structure their engineering organization to maximize team productivity in a service-oriented environment. Two things that stood out to me: each service is owned by a “home team”, but sometimes features that are needed by other teams can be built by forming an “away team” to build out that functionality. Secondly, Amazon has a concept of “bar raisers” who are engineers across the organization who help approve key design and architectural decisions. It’s possible to go against the recommendation of a bar raiser but “such a move is noted and made visible to higher levels of management”. # 14th May 2019, 6:32 pm

Amazon S3: Versioning Proposal. The us-west-1 S3 bucket region now optionally supports versioning—once enabled on a bucket, all previous versions of keys will be preserved. # 24th January 2010, 1:38 pm

OpenStreetMap Rendering Database. Amazon have added an OpenStreetMap snapshot as a public data set, thanks to some smart prompting by Jeremy Dunck. # 10th October 2009, 1:05 pm

Tile Drawer (via) The most inspired use of EC2 I’ve seen yet: center a map on an area, pick a Cascadenik stylesheet URL (or write and link to your own) and Tile Drawer gives you an Amazon EC2 AMI and a short JSON snippet. Launch the AMI with the JSON as the “user data” parameter and you get your own OpenStreetMap tile rendering server, which self-configures on startup and starts rendering and serving tiles using your custom design. # 26th August 2009, 9:32 am

Introducing Amazon Virtual Private Cloud (VPC). Amazon now let you create a network of private EC2 instances completely isolated from the internet and the rest of the EC2 cloud, then link them back to your home network via a VPN. # 26th August 2009, 8:42 am

Evidence of OpenID at Amazon. It looks like Amazon are using OpenID for SSO between their different properties—I clicked a link to sign in to AWS and the URL had OpenID query string parameters. # 6th July 2009, 1:25 am

AWS Import/Export: Ship Us That Disk! Andrew Tanenbaum said “Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway”, and now you can ship your storage device direct to Amazon and have them load the data in to an S3 bucket for you. # 21st May 2009, 11:22 am

EC2: Creating an Image. Here’s the easier way of creating your own AMI: start with a running instance in EC2, then customise it to fit your purposes and create a new bundle (and then AMI) using the ec2-bundle-vol command. # 19th May 2009, 7:50 pm

HOWTO Building a self-bundling Debian AMI. Not as terrifying as you would have thought. Also contains some neat hints as to how some of the more magical parts of EC2 work (like the way your SSH public key automatically ends up in /root/.ssh/authorized_keys). # 19th May 2009, 7:49 pm

New Features for EC2: Elastic Load Balancing, Auto Scaling, and Amazon CloudWatch. EC2 now fulfils the promise of “magic scaling in the cloud” out of the box—CloudWatch monitors performance of your EC2 instances without needing to install any monitoring software, Auto Scaling allows you to configure “scaling triggers” which start up new instances based on information from CloudWatch, and Elastic Load Balancing balances requests across all available instances. # 18th May 2009, 10:07 am

Amazon Says Listing Problem Was an Error, Not a Hack (via) “A friend within the company told him that someone working on Amazon’s French site mistagged a number of keyword categories, including the ’Gay and Lesbian’ category, as pornographic, using what’s known internally as the Browse Nodes tool. Soon the mistake affected Amazon sites worldwide.” # 14th April 2009, 8:32 am

How to cause moral outrage from the entire Internet in ten lines of code. Looks legit—the author claims to have sparked this weekend’s #amazonfail moral outrage (where Amazon where accused of removing Gay and Lesbian books from their best seller rankings) by exploiting a CSRF hole in Amazon’s “report as inappropriate” feature to trigger automatic takedowns. EDIT: His claim is disputed elsewhere (see comments) # 13th April 2009, 7:48 pm

Experiences deploying a large-scale infrastructure in Amazon EC2. “At OpenX we recently completed a large-scale deployment of one of our server farms to Amazon EC2. Here are some lessons learned from that experience.” # 10th April 2009, 9:43 am

Finding similar items with Amazon Elastic MapReduce, Python, and Hadoop streaming. Tutorial for running Hadoop jobs on Elastic MapReduce using Python and the 2005 Audioscrobbler dataset. # 7th April 2009, 9:19 am

Amazon Elastic MapReduce (via) Hadoop as a service. Basically a web based GUI around Hadoop—you could roll this yourself on EC2 but for a small markup on regular EC2 prices you get to avoid the extra work setting everything up. Data processing scripts can be written in Java, Ruby, Perl, Python, PHP, R, or C++ and are loaded in to S3 before firing off the job. # 2nd April 2009, 10:25 am

Manage Amazon EC2 With New Web-Based AWS Management Console. Finally! I’m amazed it took Amazon so long to do this. Managing EC2 instances from a custom Firefox extension was pretty bizarre. It’s a very nice interface, built on top of YUI. Unfortunately you still have to manage your entire virtual server farm using a single shared Amazon account. # 9th January 2009, 9:34 am

Amazon SimpleDB—Now With Select. So now all three of Yahoo!, Amazon and Google have invented their own SQL-like languages (YQL, SimpleDB and GQL)—though it looks like Yahoo!’s is the only one that attempts to provide joins. # 18th December 2008, 8:59 am

Amazon CloudFront. The Amazon CDN front end for S3 has launched. Traffic is 2 cents per GB more than S3. I’d like to see a price comparison with existing CDNs; I have a hunch it’s an order of magnitude less expensive. # 18th November 2008, 2:37 pm

Coming Soon: Amazon EC2 With Windows. It’s not instantly clear if you need to source your own Windows licenses or if the license comes as part of the hourly VM charge. If it’s the latter, I can see this being fantastically useful for both automated and manual cross-browser testing—throw up a Windows VM for just as long as you need to run your tests, running them through rdesktop. # 1st October 2008, 9:16 am

Cheap, Easy Audio Transcription with Mechanical Turk. Andy Baio’s in-depth tutorial on submitting HITs to Mechanical Turk. I hadn’t realised how straight forward and powerful the interface has become. # 25th September 2008, 6:37 pm

Google’s Usability Research on Federated Login. Fascinating—suggests an approach to federated auth based on the Amazon.com “Yes, I have a password” login flow. Feels convoluted to me but apparently it tests really well against a mainstream audience. The more research shared around this stuff the better. # 22nd September 2008, 8:56 pm

We’re Never Content. Amazon will be releasing a proper edge caching CDN on top of S3 “before the end of the year”. # 18th September 2008, 12:30 pm

Persistent Django on Amazon EC2 and EBS—the easy way. Useful tutorial on getting Django up and running on EC2 with EBS for a persistent PostgreSQL database. # 21st August 2008, 9:32 pm

Amazon Elastic Block Store (EBS). EC2 just got a whole lot more useful—you can now create “block level storage volumes” (think virtual hard drives) and mount them to an EC2 instance for real persistent storage—but because they’re virtual you can clone them, snapshot them and benefit from automatic replication. # 21st August 2008, 10:15 am

Amazon S3 Availability Event: July 20, 2008. Don’t let the newspeak put you off; this is an honest and informative description of the bug that took down S3 last Sunday, although it does include the world’s longest way of saying “we turned it off and on again”. # 27th July 2008, 5:42 pm

Browser Uploads to S3 using HTML POST Forms. I didn’t know you could do this: create a regular HTML form that gives people permission to upload direct to your own S3 bucket, using a signed JSON policy statement in a hidden form field to prevent third parties from abusing your S3 account. # 27th June 2008, 12:11 pm

EC2: Introducing Elastic IP Addresses and Availability Zones. Big news from Amazon: EC2 can now provide static IP addresses which you can dynamically map to one of your instances, along with “availability zones” so you can specify that instances run in different data centres. Hosting an entire application on EC2 just got a whole lot more practical. # 27th March 2008, 10:33 am

Amazon.com: amazon oddities. Warning: reading the user reviews on these items has the potential to soak up hours. # 21st March 2008, 2:54 am

Eventually Consistent. Werner Vogels explains the trade-offs involved in building scalable, highly-available data stores such as Amazon’s SimpleDB. # 20th December 2007, 5:59 pm

Amazon SimpleDB overview. Attribute values are limited to 1,024 bytes; Amazon suggest that you store larger fields in S3 and use SimpleDB to query metadata about those objects. # 14th December 2007, 11:39 am