59 items tagged “aws”
2019
athena-sqlite (via) Amazon Athena is the AWS tool for querying data stored in S3—as CSV, JSON or Apache Parquet files—using SQL. It’s an interesting way of buliding a very cheap data warehouse on top of S3 without having to run any additional services. Athena recently added a query federation SDK which lets you define additional custom data sources using Lambda functions. Damon Cortesi used this to write a custom connector for SQLite, which lets you run queries against data stored in SQLite files that you have uploaded to S3. You can then run joins between that data and other Athena sources.
Serverless Microservice Patterns for AWS (via) A handy collection of 19 architectural patterns for AWS Lambda collected by Jeremy Daly.
Using 6 Page and 2 Page Documents To Make Organizational Decisions (via) I’ve been thinking a lot recently about the challenges of efficiently getting to consensus within a larger organization spread across multiple locations and time zones. This model described by Ian Nowland based on his experience at AWS seems very promising. The goal is to achieve a decision or “disagree and commit” consensus using a max 6 page document and a one hour meeting. The first fifteen minutes of the meeting are dedicated to silently reading the document—if you’ve read it already you are given the option of arriving fifteen minutes late.
Colm MacCárthaigh tells the inside story of how AWS responded to Heartbleed. The Heartbleed SSL vulnerability came out five years ago. In this Twitter thread Colm, who was Amazon’s principal engineer for Elastic Load Balancer at the time, describes how the AWS team responded to something that “was scarier than any bug I’d ever seen”. It’s a cracking story.
The Cloud and Open Source Powder Keg (via) Stephen O’Grady’s analysis of the Elastic v.s. AWS situation, where Elastic started mixing their open source and non-open source code together and Amazon responded by releasing their own forked “open distribution for Elasticsearch”. World War One analogies included!
2018
AWS Ground Station – Ingest and Process Data from Orbiting Satellites. OK this is cool. “Instead of building your own ground station or entering in to a long-term contract, you can make use of AWS Ground Station on an as-needed, pay-as-you-go basis. [...] You don’t need to build or maintain antennas, and can focus on your work or research. We’re starting out with a pair of ground stations today, and will have 12 in operation by mid-2019. Each ground station is associated with a particular AWS Region; the raw analog data from the satellite is processed by our modem digitizer into a data stream (in what is formally known as VITA 49 baseband or VITA 49 RF over IP data streams) and routed to an EC2 instance that is responsible for doing the signal processing to turn it into a byte stream.”
The Free Stack—Running your application for free on AWS (via) Parikshit Agnihotry provides a useful rundown of quite how much you can get done using the first 12 month free tier of AWS API Gateway, Lambda, DynamoDB, S3 and CloudFront.
2017
Landsat on AWS (via) TIL Amazon make data from the Landsat 8 satellite available for free on S3 (though they are no doubt hoping you’ll pay for EC2 instances to process the data). “All new Landsat 8 scenes are made available each day, often within hours of production. The satellite images the entire Earth every 16 days at a roughly 30 meter resolution”.
2013
Using AWS, as my cloud, what is left for me to work on? Is it enough for me to just write the html+css code and programming language code (python)? Or do I stil have to work with mysql and backend stuff? I am pretty new at programming, so I hope it i...
Using a cloud server platform like Amazon EC2 unfortunately will not protect you from needing to understand basic server adminstration—it’s not that different from running your own physical server, except that if you screw up the configuration it’s much easier to throw everything away and start from scratch.
[... 134 words]What kind of website can be run on AWS for 10, 100, 1 thousand, 10 thousand, 100 thousand, 1 million dollars per month?
“But is there a simple way to say that for 10$ per month you can run website on AWS, that has X unique users and Y data transfer...”
[... 166 words]For a Django application, deployed on Heroku, what are my options for storing user-uploaded media files?
S3 is really a no-brainer for this, it’s extremely inexpensive, very easy to integrate with and unbelievably reliable. It’s so cheap that it will be practically free for testing purposes (expect to spend pennies a month on it).
[... 88 words]2009
Evidence of OpenID at Amazon. It looks like Amazon are using OpenID for SSO between their different properties—I clicked a link to sign in to AWS and the URL had OpenID query string parameters.
AWS Import/Export: Ship Us That Disk! Andrew Tanenbaum said “Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway”, and now you can ship your storage device direct to Amazon and have them load the data in to an S3 bucket for you.
aws—simple access to Amazon EC2 and S3. The best command line client I’ve found for EC2 and S3. “aws put --progress my-bucket-name/large-file.tar.gz large-file.tar.gz” is particularly useful for uploading large files to S3. Written in Perl (with no dependencies), shelling out to curl to do the heavy lifting.
Manage Amazon EC2 With New Web-Based AWS Management Console. Finally! I’m amazed it took Amazon so long to do this. Managing EC2 instances from a custom Firefox extension was pretty bizarre. It’s a very nice interface, built on top of YUI. Unfortunately you still have to manage your entire virtual server farm using a single shared Amazon account.
2008
How Tarsnap uses Amazon Web Services (via) Useful case study, including some thoughts on SimpleDB.
Persistent Django on Amazon EC2 and EBS—the easy way. Useful tutorial on getting Django up and running on EC2 with EBS for a persistent PostgreSQL database.
Browser Uploads to S3 using HTML POST Forms. I didn’t know you could do this: create a regular HTML form that gives people permission to upload direct to your own S3 bucket, using a signed JSON policy statement in a hidden form field to prevent third parties from abusing your S3 account.
2007
Amazon Gets an SLA (But I Still Can’t Use It). “Ontario’s Freedom of Information and Protection of Privacy Acts (FIPPA) don’t allow me to store sensitive information (e.g., students’ work) in jurisdictions that permit secret warrants, like those mandated by the USA PATRIOT Act.”
Amazon S3 Service Level Agreement (via) Went in to effect on the 1st of October. Promises 99.9% uptime over a monthly billing cycle or you get “service credits” towards future S3 payments.
Processing Web Documents using Alexa Web Search, Amazon S3 and Amazon EC2. I’m not sure when it happened, but Alexa Web Search can be hooked in to EC2 now—presumably with free bandwidth between the two.
Mass Video Conversion Using AWS. How to use S3, SQS, EC2, ffmpeg and some Python to bulk convert videos with Amazon Web Services.
boto. Python library for accessing Amazon’s S3, SQS and EC2 Web Services, with excellent documentation.
2006
An S3 AJAX Wiki. Les continues to innovate against S3.
How I’m using Amazon S3 to serve media files. Adrian’s saving server overhead on ChicagoCrime by serving media from S3.
JavaScript apps with read/write access to S3. JS apps hosted on S3 could read and write to the store.
Backing Up Flickr Photos with Amazon S3. 25 lines of Python.
BitBucket—Experimenting with Amazon S3 Service in Python (via) Nice pythonic API.
Amazon S3. Game changing.