Simon Willison’s Weblog

We Need to Stop Google’s Exploitation of Open Communities. Mikel Maron from OpenStreetMap is justifiably angry about Google MapMaker, which copies OpenStreetMap’s model of crowdsourcing geographic data (even copying the OSM idea of Mapping Parties) but keeps the data under a much more restrictive license, and uses the Google brand to market itself to African governments. 22nd April 2011, 10 am

Why Facebook open-sourced its datacenters. Jon Stokes speculates that Facebook plan to use open source hardware to compete with Google at datacenter efficiency . This isn’t a new pattern. Years ago when I worked at Yahoo! I was furiously jealous of the secret sauce technologies that allowed Google to build big applications faster than anyone else, such as BigTable and map/reduce. Today, the open source world has created better, free alternatives—sponsored in part by Facebook, Yahoo! and other Google competitors. 9th April 2011, 7:54 am

Google APIs & Developer Products. Presented as a sort-of-periodic table. There’s quite a bit of stuff on here I didn’t know about. 28th January 2011, 11:25 am

Getting Started—Google URL Shortener API. The API for the goo.gl URL shortener is really nice—no API key required, easy to create a short URL and you can retrieve detailed stats breakdowns (similar to bit.ly) as JSON for any URL. 13th January 2011, 3:49 am

Google and Microsoft Cheat on Slow-Start. Should You? Fascinating optimisation tricks by some of the big websites, which violate the RFC governing the TCP slow-start algorithm in order to perform better in the common case. 3rd December 2010, 7:03 pm

“Last I heard Google’s search index was sharded by document rather than by term.”. Fascinating comment by jasonwatkinspdx on Hacker News. 8th October 2010, 8:03 pm

Closure Compiler Service (via) A hosted version of the Google Closure Compiler (JavaScript minifier) running on App Engine. It has both a user interface and a REST API, which means you can use it as part of an automated build process without needing to set up a local copy of the software. 9th August 2010, 1:17 pm

App Engine at Google I/O 2010. OpenID and OAuth are now baked in to the AppEngine users API. They’re also demoing two very exciting new features—a mapper API for doing map/reduce style queries against the data store, and a Channel API for building comet applications. 20th May 2010, 3:30 pm

Google Font Directory: Font Preview. Handy tool for trying out the 18 open source fonts Google have released, along with server-side browser sniffing technology that serves up the correct version (including for IE6). The browser sniffing makes me a bit uncomfortable—will it play well with intermediate caches? What happens if I save a local copy of a page and then open it up in a different browser? 20th May 2010, 3:20 pm

Stack Overflow Blog: OpenID, One Year Later. Google’s support is a huge deal—61% of Stack Overflow accounts use Google. Google’s implementation of directed identity has caused problems though, since Google provide a different OpenID for each domain making it hard for Stack Overflow, Server Fault and Super User to correlate accounts. Their solution is to require a (verified) e-mail address from Google OpenID users using sreg and use that as a key for the accounts. 14th April 2010, 8:46 pm

Why Google MapMaker is not Open. Non-commercial use only, strict attribution requirements and you aren’t allowed to use the data for services that might compete with Google. This is why I’m disappointed every time I see Google encouraging people to contribute to Map Make, especially in the developing world—if those people contributed to OpenStreetMap instead they would be building something far more valuable for their community. 16th March 2010, 10:41 am

RE2: a principled approach to regular expression matching. Google have open sourced RE2, the C++ regular expression library they developed for Google Code Search, Sawzall, Bigtable and other internal projects. Unlike PCRE it avoids the potential for exponential run time and unbounded stack usage and guarantees that searches complete in linear time, mainly by dropping support for back references. 12th March 2010, 9:28 am

Google Image Charts: Mathematical (TeX) Formulas (via) I’m not sure when they added this, but you can now use the Google Charts Image API to render mathematical formulas, specified using TeX syntax. Wordpress.com and Wikipedia have both offered this feature for quite a while, but now you can use it anywhere on the Web. 12th February 2010, 9:42 am

WARNING: Google Buzz Has A Huge Privacy Flaw. Interesting one this: by default, Buzz creates a public profile for you that lists the people you follow—but your default set of followers is derived from the people you contact most frequently using Gmail. This means users of Buzz may inadvertently reveal their most frequent contacts, which is an issue for people like journalists with anonymous sources, unhappy employees seeking new work or even people having e-mail based affairs. 11th February 2010, 11:30 am

Fixing the Google Account problem. 3,000+ words explaining how to open a Google Doc invitation sent to an e-mail address that isn’t associated with your Google account. Worth reading just to get an idea for the enormous complexity involved in running a large scale identity system and designing an interface for managing aliases and multiple profiles. Google haven’t got it right yet—has anyone else? 25th January 2010, 11:21 am

HTTP + Politics = ? Mark Nottingham ponders the technical implications of Australia’s decision to apply a filter to all internet traffic. Australia is large enough (and far enough away from the northern hemisphere) that the speed of light is a performance issue, but filtering technologies play extremely poorly with optimisation technologies such as HTTP pipelining and Google’s SPDY proposal. 15th December 2009, 3:36 pm

A piece with a lot of screenshots about the close tab behaviour in Google Chrome. If you click “close” with your mouse, Chrome doesn’t resize the remaining tabs until you mouse away from the area. This means you can click “close” multiple times without having to chase the close button. I hadn’t noticed this, partly because Chrome doesn’t do it if you hit Command-W. They even switch the position of the close button in RTL languages such as Arabic. 11th December 2009, 9:19 am

EtherPad is Back Online Until Open Sourced. Fantastic news. EtherPad just got acquired by Google and announced the team would be joining the Google Wave effort and the existing service would be shut down. Lots of people complained, so they’re going to keep it alive until they’ve open sourced the code! 6th December 2009, 9:08 am

Google Analytics goes async. This is excellent news—the latest version of the Google Analytics JavaScript is designed to allow for asynchronous loading, so it won’t hold up the rendering of your page. Analytics and banner ads are the two worst offenders when it comes to slowing down page loads. Now if only a banner ad vendor would follow suit... 2nd December 2009, 6:30 pm

Negative Cashback from Bing Cashback (via) Some online stores show you a higher price if you click through from Bing—and set a cookie that continues to show you the higher price for the next three months. It’s unclear if this is Bing’s fault—comments on Hacker News report that Google Shopping sometimes suffers from the same problem (POST UPDATED: I originally blamed Bing for this). 23rd November 2009, 9:24 pm

Chromium OS User Experience. The 2 minute UI concept video is probably the best way to understand the ideas behind Google’s Chrome OS. 19th November 2009, 10:12 pm

SPDY: The Web, Only Faster. Alex Russell explains the benefits of Google’s SPDF proposal (a protocol that upgrades HTTP)—including header compression, multiplexing, the ability to send additional resources such as images and stylesheets down without needing the data:uri hack and Comet support built in to the core assumptions of the protocol. 13th November 2009, 1 pm

Awkward Suggestions (via) The Google search box “suggest” feature returns very different results depending on the quality of your grammar—“how 2” v.s. “how might one” is particularly illuminating. 12th November 2009, 10:31 am

The Go Programming Language. A brand new systems programming language, designed by Robert Griesemer and Unix/Plan 9 veterans Rob Pike and Ken Thompson and funded by Google. Concurrency is supported by lightweight communicating processes called goroutines. “It feels like a dynamic language but has the speed and safety of a static language.” 11th November 2009, 7 am

Introducing Closure Tools. Google have released the pure-JavaScript library, apparently used for Gmail, Google Docs and Google Maps. It comes with a powerful JavaScript optimiser tool with linting built in and an accompanying Firebug extension to ensure the obfuscated code it produces can still be debugged. There’s also a template system which precompiles down to JavaScript and can also be called from Java. 6th November 2009, 7:33 am

Google Dashboard. New Google product which shows exactly how much information Google have stored against your account, all on one page. This is a really useful tool, and hopefully will help set a powerful precedent for other sites to follow. 5th November 2009, 2:03 pm

Cartographer.js. “Thematic mapping for Google Maps”—which means an easy way of adding heat maps (aka chloropleths), pie charts and point clusters as a layer over a Google map. 1st November 2009, 1:20 pm

Official Google Webmaster Blog: A proposal for making AJAX crawlable. It’s horrible! The Google crawler would map url#!state to url?_escaped_fragment_=state, then expect your site to provide rendered HTML that reflects that state (they even go as far as to suggest running a headless browser within your web server to do this). Just stick to progressive enhancement instead, it’s far less hideous. It looks like the proposal may have originated with the GWT team. 8th October 2009, 5:52 pm

Google Docs OCR. Whoa, the Google Docs API just got really interesting—you can upload an image to it (POST /feeds/default/private/full?ocr=true) and it will OCR the text and turn it in to a document. Since this is Google, I imagine they’ll also be using the processed documents to further improve their OCR technology. 29th September 2009, 9:57 pm

Google’s first press release. From 1999, announcing $25 million in equity funding. I’m impressed to see that the mission statement already stated “Google’s mission is to organize the world’s information, making it universally accessible and useful.” 27th September 2009, 8:35 pm

OpenID: Now more powerful and easier to use! The OpenID+OAuth hybrid protocol (where a user can sign in with OpenID and grant an application access to their OAuth protected resources such as a contact list at the same time) is now supported by Google, Yahoo! and MySpace—this feels like OpenID finally coming of age. 25th September 2009, 9:08 pm

Gmail for Mobile: Reducing Startup Latency. Cheeky iPhone optimisation trick—parsing 200 KB of JavaScript takes an iPhone 2.2 device 2.6 seconds, so Gmail embeds code components in /* comments */ in a script tag and evals them on demand later on when the features are needed. 23rd September 2009, 10:29 pm

More technical details about Google Chrome Frame. It’s implemented as a Browser Helper Object, uses IE’s cookies, history and password-remembering, includes the WebKit developer tools and appends “chromeframe” to the regular IE user agent string—though not apparently the Chrome Frame version itself. 23rd September 2009, 10:20 pm

PubSubHubbub for Google Alerts. “Think of it as a search API that tells *you* when it finds new results.” 23rd September 2009, 9:30 pm

Introducing Google Chrome Frame. Here’s what Alex Russell has been up to at Google: An IE plugin (for 6, 7 and 8 on all Windows versions) which embeds the Google Chrome rendering engine—sites can then opt-in to using it by including a X-UA-Compatible meta tag. Seems to be aimed at corporate networks which mandate IE for badly written intranet applications—they can roll this out without retraining users to use another browser or breaking their existing in house apps. 23rd September 2009, 9:57 am

Static Maps API v2. The new version of the Google Static Maps API (static images generated using arguments in a URL, no JavaScript required) adds support for paths, areas and automatically geocoding addresses to specify locations of markers and the centre of the map. 26th August 2009, 9:01 am

AdSense for Feeds: What’s all the hubbub about PubSubHubbub? “Today we’re happy to announce initial support in FeedBurner for the PubSubHubbub protocol.” 24th July 2009, 6:45 pm

Farewell to Mashup Editor. It’s not just Microsoft Popfly that’s shutting down—Google Mashup Editor will be gone in four weeks time (this was announced in January). You get to keep your code, but I don’t know enough about Mashup Editor to know if the code is usable once the system has shut down. 17th July 2009, 1:05 pm

Google’s Chiller-less Data Center. Google are operating an outside data center in Belgium with no chillers (refrigeration units used to cool water, but at a high cost in energy) making “local weather forecasting a larger factor in its data center management”. On the 10 or so days of the year when Belgium is too warm, they can simply shut down the data center and shift the workload elsewhere. 16th July 2009, 9:50 am

Google Will Eat Itself. “We generate money by serving Google text advertisments on a network of hidden Websites. With this money we automatically buy Google shares. We buy Google via their own advertisment!” 10th July 2009, 12:15 pm

App Engine outage postmortem. Interesting peek behind the scenes. The primary cause of the error was a bug in a GFS (Google File System) Master server caused by a MapReduce process sending a malformed filehandle, reminiscent of the error which took down S3 last year. 9th July 2009, 12:49 pm

Codecs for <audio> and <video>. HTML 5 will not be requiring support for specific audio and video codecs—Ian Hickson explains why, in great detail. Short version: Apple won’t implement Theora due to lack of hardware support and an “uncertain patent landscape”, while open source browsers (Chromium and Mozilla) can’t support H.264 due to the cost of the licenses. 2nd July 2009, 10:16 am

Google asked people in Times Square:“What is a browser?”. Stuff like this makes me despair for creating a secure web—what chance do people have of surfing safely if they don’t understand browsers, web sites, operating systems, DNS, URLs, SSL, certificates... 20th June 2009, 1:25 am

Dealing with election results data. Alf Eaton loaded the Guardian’s European election results spreadsheet in to Google’s new Fusion Tables tool. 12th June 2009, 6:06 pm

Announcing Google Maps API v3. Sounds like a complete rewrite, with performance as the key goal. Only a developer preview at the moment, but my favourite feature is that API keys are no longer required. 28th May 2009, 1:22 am

geocoders. A fifteen minute project extracted from something else I’m working on—an ultra simple Python API for geocoding a single string against Google, Yahoo! Placemaker, GeoNames and (thanks to Jacob) Yahoo! Geo’s web services. 27th May 2009, 10:02 am

Offline Processing on App Engine: a Look Ahead. A session at IO next week: “App Engine was designed to run request-driven web applications, although this will change in the coming year with the release of a number of offline computing components. In this session, we’ll explore the task queue/executor model of computation and some of the more interesting applications.” 20th May 2009, 12:40 pm

Google container data center tour (on YouTube). 45,000 servers in 45 shipping containers, along with some serious looking plumbing. 26th April 2009, 10:14 pm

And Now For Something Entire... Oooh! Shiny! Alex Russell on O3D, the new 3D browser plugin from Google that makes OpenGL accessible to JavaScript (and embeds V8 so performance won’t suck even on slower browsers). 22nd April 2009, 12:19 pm

London’s abandoned Underground Stations on Google Street View. “The network is littered with buildings that belonged to stations that closed their doors to the public because routes were changed and diverted, or because there was just too little traffic to make them viable. Here are some of the remnants of disused Underground stations that you can see on Google’s Street View of London.” 14th April 2009, 2:51 pm

Reducing XSS by way of Automatic Context-Aware Escaping in Template Systems (via) The Google Online Security Blog reminds us that simply HTML-escaping everything isn’t enough—the type of escaping needed depends on the current markup context, for example variables inside JavaScript blocks should be escaped differently. Google’s open source Ctemplate library uses an HTML parser to keep track of the current context and apply the correct escaping function automatically. 14th April 2009, 9:26 am

Running Rhino and Helma NG on Google App Engine. Helma NG is a JavaScript web app framework, which now works on App Engine out of the box. 12th April 2009, 12:52 pm

Using Scala with Google App Engine. Scala works, but I haven’t seen confirmation on actors yet (which are likely to break due to their dependency on threads). 11th April 2009, 3:28 pm

Dynamic languages on Google App Engine—an overview. Ola Bini’s notes on exploring the new Java support for App Engine with the aim of getting JVM dynamic languages such as JRuby running. Restrictions include a complete lack of threads (which will make it hard to get Scala up and running), but JRuby trunk now works without modification. 8th April 2009, 2:08 pm

App Engine: Scheduled Tasks With Cron. Cron tasks simply hit a URL on your application, and can be run as frequently as once a minute. They made up their own syntax, which much nicer than traditional unix cron. 8th April 2009, 2:04 pm

Google uncloaks once-secret server. Instead of a data centre wide UPS and redundant power supplies, each Google server has its own 12V battery. They live in standard shipping containers, each holding 1,160 servers. 2nd April 2009, 10:47 am

ProjectPlan—unladen-swallow. A branch of Python 2.6 aiming to radically improve performance (the target is a 5x improvement), by compiling Python to machine code using LLVM’s JIT engine. I think this is a Google 20% time project (or maybe not, see the comments). An early version without LLVM is already available for download. 30th March 2009, 10:09 am

django-gae2django. An implementation of the Google App Engine API (datastore, memcache, urlfetch, users and mail) that runs on Django, allowing you to take an existing application written for App Engine and deploy it on your own server on top of Django. 9th March 2009, 3:37 pm

Map Maker for Developers. Tiles from Google’s Map Maker crowdsourcing effort are now available in the JS and static maps APIs on an opt-in basis. Maybe I’m misunderstanding something here, but Google Map Maker seems like a big step backwards for open geographic data. People donate their mapping efforts to Google, who keep them—unlike OpenStreetMap, where the donated efforts are made available under a Creative Commons license. 21st February 2009, 9:05 am

Write to a Google Spreadsheet from a Python script. I didn’t know Google Spreadsheets could directly serve dynamic images that automatically update when the underlying data changes. 16th February 2009, 9:02 pm

Google App Engine 1.1.9 boosts capacity and compatibility. Niall summarises the recent changes to App Engine. urllib and urllib2 support plus massively increased upload limits and request duration quotas will make it a whole lot easier to deploy serious projects on the platform. 16th February 2009, 8:35 pm

Specify your canonical. You can now use a link rel=“canonical” to tell Google that a page has a canonical URL elsewhere. I’ve run in to this problem a bunch of times—in some sites it really does make sense to have the same content shown in two different places—and this seems like a neat solution that could apply to much more than just metadata for external search engines. 14th February 2009, 11:28 am

Plaxo sees 92% success rate with OpenID/OAuth hybrid method. Really wish I could have been at the OpenID UX Summit hosted by Facebook yesterday—sounds like an awful lot of important problems are being solved. 11th February 2009, 5:20 pm

Yahoo! Query Language thoughts. An engineer on Google’s App Engine provides an expert review of Yahoo!’s YQL. I found this more useful than the official documentation. 9th February 2009, 10:29 pm

Google App Engine: A roadmap update! Receiving e-mail, background tasks and XMPP. I predict a bunch of sites will start building small parts of their overall functionality on App Engine when some of these features land (much easier than hosting your own custom XMPP server). 9th February 2009, 7 pm

Recreating the button. Fascinating article from Doug Bowman on the work that went in to creating custom CSS buttons for use across Google’s different applications, avoiding images to improve performance ensure they could be easily styled using just CSS. I’d love to see the Google Code team turn this in to a full open source release—the more sites using these buttons the more familiar they will become to users at large. 5th February 2009, 9:50 pm

Post-Commit Web Hooks for Google Code Project Hosting (via) I really, really like web hooks (which I’ve been calling “callback APIs”, but it looks like “web hooks” is the term that’s sticking). I’m interested in their scaling challenges—I’ve heard XMPP advocates argue that a web hook style model simply won’t scale for really large sites. 4th February 2009, 10:22 am

Sharding Counters on Google App Engine. “While the datastore for App Engine scales to support a huge number of entities it is important to note that you can only expect to update any single entity, or entity-group, about five times a second”. This article explains a technique for sharding writes across multiple counters in detail, including a way to keep a memcache counter updated at the same time for faster reads. 27th January 2009, 8:27 pm

google-mobwrite. Neal Fraser’s terrifyingly clever differential synchronization algorithm (for SubEthaEdit-style collaboration over the web) is now available as an open source Python and JavaScript library. 24th January 2009, 11:55 pm

AJAX APIs Playground. Ferociously useful collection of executable and editable example code for all(?) of Google’s JavaScript APIs, including Google Maps and the increasingly interesting Visualization API. 22nd January 2009, 6:38 pm

Leo Hickman on the carbon cost of Googling. Alex Wissner-Gross (who published the 7g/search figures) appears to be including Google’s extra capacity, so total CO2 output divided by number of searches. Google’s 0.2g/search estimate includes just the energy used by the servers processing your query. 12th January 2009, 3:31 pm

Powering a Google search. I thought the recent estimate of each Google search producing 7g of CO2 was a little high—Google have responded with a claim that the amount is 0.2g instead. 12th January 2009, 11:15 am

Amazon SimpleDB—Now With Select. So now all three of Yahoo!, Amazon and Google have invented their own SQL-like languages (YQL, SimpleDB and GQL)—though it looks like Yahoo!’s is the only one that attempts to provide joins. 18th December 2008, 8:59 am

Now You Can Sign Into Friend Connect Sites With Your Twitter ID. Great. Now even Google is asking me for my Twitter password. Slow clap. How’s that Twitter OAuth beta coming along? 15th December 2008, 5:20 pm

OurDelta Builds for MySQL (via) A community supported “alternative distro” of MySQL, incorporating new features from Google and other sources by maintaining a clean set of patches against the MySQL source tree (which I guess is why it’s not considered a fork). I recognise some of the patches from the excellent “High Performance MySQL, 2nd Edition”. 8th December 2008, 4:20 pm

Clearing up inaccuracies about the Google OpenID IDP launch. Google took some undeserved flack when they launched their OpenID provider. For the record, whitelisting providers fits my definition of the “Open” in OpenID perfectly (providers and consumers are free to impose whatever policies they like). 8th November 2008, 11:11 pm

Code your own election mashup with Google’s JSON data. The data that powered Google’s US election results map is available to download as a bunch of JSON files. 6th November 2008, 8:24 pm

New OpenID Implementations Abound. I’ve missed linking to a bunch of OpenID news recently—in particular, Google Accounts are becoming OpenID identifiers and LiveJournal has quietly ugraded its consumer support to OpenID 2.0. 30th October 2008, 5:11 pm

Why Google App Engine is broken and what Google must do to fix it. Aral Balkan describes a number of critical issues with App Engine. If you’re considering building something serious on it you need to read this article; I’ve run in to several of these problems myself just running toy projects on the platform. Here’s hoping they get addressed in the near future. 3rd October 2008, 10 pm

Logout/Login CSRF. Alf Eaton built an example page (this link goes to his description, not the page itself) that uses a login CSRF attack to log you in to Google using an account he has created. Scary. 24th September 2008, 10:18 pm

Google’s Usability Research on Federated Login. Fascinating—suggests an approach to federated auth based on the Amazon.com “Yes, I have a password” login flow. Feels convoluted to me but apparently it tests really well against a mainstream audience. The more research shared around this stuff the better. 22nd September 2008, 8:56 pm

OAuth Playground (via) Neat OAuth API explorer from the Google Data APIs team. 20th September 2008, 4:40 pm

YouTube: djangocon tag. Google have started posting videos of presentations at DjangoCon on YouTube. 16th September 2008, 2:43 am

Google wants your Hotmail, Yahoo and AOL contacts. And they’re using the password anti-pattern to get them! Despite both Yahoo! and Hotmail (and Google themselves; not sure about AOL) offering a safe, OAuth-style API for retrieving contacts without asking for a password. This HAS to be a communications failure somewhere within Google. Big internet companies stand to lose the most from widespread abuse of the anti-pattern, because they’re the ones most likely to be targetted by phishers. Shameful. 15th September 2008, 10:39 am

The story behind Google Chrome. Superbly researched by Niall Kennedy—a detailed overview of the staff and acquisitions that went in to Google Chrome. 4th September 2008, 1:50 am

We haven’t changed the name of the conference to “Over Quota”. Aral is having intermittent App Engine quota problems, which are proving impossible to debug. I had a similar problem with an App Engine app a while ago—the quota / debugging story really needs fixing. 3rd September 2008, 1:37 pm

V8 Design Elements. High level design details of Google’s V8 JavaScript engine, including how it uses “hidden classes” to optimise object property lookups and a bit of information on the machine code generation and garbage collection. 2nd September 2008, 11:58 pm

Chromium. Google Chrome is out! Here’s the open source project, including the code for the new V8 JavaScript virtual machine. 2nd September 2008, 9:06 pm

Google Chrome, the comic book (via) Google have finally announced a browser project, though it’s currently vapourware (or rather comicware), existing only as a Scott McCloud comic. Still, it looks fascinating—entirely open source, WebKit with a brand new JavaScript VM, every tab running in a separate process for smarter memory usage and some new UI concepts and anti-pishing measures thrown in as well. 1st September 2008, 7:45 pm

Google’s undocumented favicon to png convertor (via) Showing the favicon of a domain next to a link is a really nice trick, but it’s slightly tricky to achieve as IE won’t display a .ico file if you link to it from an img element, so you need to convert the images server-side. This undocumented Google API does that for you, meaning it’s much easier to add favicons as a feature to your site. 30th August 2008, 8:40 pm

Gears for Safari Beta. “Chances are it will break your browser. Please proceed with caution.” 26th August 2008, 4:27 pm

Keyczar (via) New open source cryptography toolkit from Google, designed to get algorithm selection, key rotation and versioning right so you don’t have to. Java and Python versions are available; the Python version depends on PyCrypto. 13th August 2008, 1:20 pm

Underscores are now word separators, proclaims Google. I missed this story last year—the change was announced by Matt Cutts at WordCamp 2007. 13th August 2008, 1:06 pm

knol: content w/out context, collaboration, capital, or coruscation. danah boyd: “A system that is driven by individualism quickly becomes a tool for self-promoters” 3rd August 2008, 3:13 pm

Protocol Buffers: Google’s Data Interchange Format. Open sourced today. Highly efficient binary protocol for storing and transmitting structured data between C++, Java and Python. Uses a .proto file describing the data structure which is compiled to classes in those languages for serializing and deserializing. 3-10 times smaller and 20-100 times faster than XML. 8th July 2008, 8:20 am

ratproxy. “A semi-automated, largely passive web application security audit tool”—watches you browse and highlights potential XSS, CSRF and other vulnerabilities in your application. Created by Michal Zalewski at Google. 3rd July 2008, 2:35 pm

OAuth for Google Data APIs (via) Awesome. Now, how’s OAuth support shaping up over at Twitter (who are serious offenders when it comes to encouraging the password anti-pattern, despite Twitter engineers being key to the creation of the original OAuth spec)? 27th June 2008, 7:49 am

Google Trends for Websites: myspace.com,facebook.com. New fun tool from Google Trends. 20th June 2008, 8:50 pm

The X-Robots-Tag HTTP header. News to me, but both Google and Yahoo! have supported it since last year. You can add per-page robots exclusion rules in HTTP headers instead of using meta tags, and Google’s version supports unavailable_after which is handy for content with a known limited shelf-life. 9th June 2008, 9:21 am

Google Finance Comet. Google Finance now shows live stock quotes, updated by Comet. 4th June 2008, 8:36 am

Google Gears renamed “Gears”. “We want to make it clear that Gears isn’t just a Google thing. We see Gears as a way for everyone to get involved with upgrading the web platform.” Support for Firefox 3 and Safari is being added and Opera are integrating Gears with both their desktop and mobile browsers. 29th May 2008, 12:38 am

Google Earth in a browser (sort of), Scriptable, a quick peek and poke. Dan Catt on Google’s new browser plugin version of Google Earth... which conveniently exposes a JavaScript API to the browser in the form of the “ge” object, which can then be poked at interactively using Firebug. 28th May 2008, 11:13 pm

Using Memcache with Google App Engine. Brad Fitzpatrick’s 20% time project. 28th May 2008, 11:11 pm

Google AJAX Libraries API (via) Google are hosting copies of jQuery, Prototype, mooTools and Dojo on their CDN, with a promise to permanently host different versions and an optional JavaScript API to dynamically load the most recent version of a library. I wish they’d stop capitalising Ajax though. 27th May 2008, 5:56 pm

Tracking Christmas Cheer with Google Charts. Brian Suda’s Google Charts tutorial on 24 ways has proved invaluable for figuring out how to handle grid lines and axis labels, both of which are pretty unintuitive (and not hugely helped by the official documentation). 26th May 2008, 9:43 pm

Search Engine Optimization Through Hoax News. Devious new black-hat SEO technique: invent a news story that’s pure link-bait. The recent “13 year old steals dad’s credit card to buy hookers” story was a hoax: it was a pure play for PageRank. 22nd May 2008, 6:09 pm

Doctype: /trunk/goog. Google’s newly released JavaScript library (pure JavaScript, so more along the lines of YUI and jQuery than GWT). I haven’t found the documentation for it yet, but the code is extremely well commented. UPDATE: The documentation is spread throughout Doctype. 14th May 2008, 9:12 pm

Doctype on Google Code. Alternative way of browsing Google Doctype—if you link to articles here instead of using the permalinks in the official version non-JavaScript user agents will be able to access the content you’ve linked to. 14th May 2008, 8:34 pm

Google Doctype. So now we know what Mark Pilgrim’s been doing at Google... heading up a project to create an encyclopaedia of web development. The JavaScript UI for browsing it is a bit weird (though you do at least get real pages if you disable JavaScript in your browser). 14th May 2008, 8:30 pm

Google AJAX Search API: Flash and Server Side Access. Over a year after Google shot down their SOAP Search API, they’ve quietly released a JSON based one under the guise of supporting “Flash and other non JavaScript environments”. Comes with the strange requirement that an HTTP referer be sent with every request; the API key is optional. 22nd April 2008, 7:16 pm

Quotation search in Google News (via) Extremely impressive application of (I suppose) natural language processing in Google News—it now extracts quotations from news stories, even handling things like “he said” and “she said” and resolving them back to the speaker. 19th April 2008, 7:22 am

KML: A new standard for sharing maps. Google’s KML format, which is already supported by both Microsoft and Yahoo!’s map software, has been accepted under the wing of the Open Geospatial Consortium and is now an international standard. 14th April 2008, 6:36 pm

OpenID for Google Accounts. Google App Engine integrates with Google’s user accounts, so Ryan Barrett (of Google) used it to build an idproxy.net style OpenID provider. 9th April 2008, 1:09 am

Running Django on Google App Engine. Django 0.96 is included, but you need to disable the ORM related parts and use the Google App Engine Bigtable interface instead. 8th April 2008, 1:15 pm

Google App Engine. Write applications in Python using a WSGI compatible application framework, then host them on Google’s highly scalable infrastructure. The most exciting part is probably the Datastore API, which provides external developers with access to Bigtable for the first time. 8th April 2008, 7:25 am

An OpenSocial Foundation. “Today we are pleased to announce that Google is joining together with Yahoo! and MySpace in the creation of a non-profit foundation for the open and transparent governance of the OpenSocial specifications and intellectual property.” Good move; I’d personally love to see this happen with Google Gears. 25th March 2008, 2:51 pm

Introducing the Google Contacts Data API. Brilliant! (and about time)—now there’s no excuse for asking your users for their Gmail username and password so you can import contacts from their address book. Yahoo! and Microsoft need to catch up on this one fast. 6th March 2008, 11:29 pm

The real reason Google’s clicks are flat. Rich Skrenta explains that Google’s recent reduction of the clicable area in Adsense ads, while reducing click-throughs by 60%, will eventually balance out due to non-accidental click-throughs being worth more to advertisers. 4th March 2008, 4:34 am

Social Graph API. This is freaking awesome. Input one or more URLs to your profile pages and it returns a huge dump of crawled relationship data, based on XFN, FOAF and OpenID links. No API key required and it supports JSON callbacks so you can incorporate it in to a site without even needing to write any extra server-side code. 3rd February 2008, 10:34 pm

New feature: Blogger as OpenID provider (via) You can now enable your Blogger blog as an OpenID. 18th January 2008, 1:38 pm

Poorly Macbook, ineffective error message design. Nat’s MacBook died the other day, throwing out some impressively meaningless error symbols. How exactly are you meant to Google for a circle with a line through it? 13th January 2008, 11:31 pm

Google apps for your newsroom. How the LJ World team use online tools like Google Spreadsheet, Swivel, ManyEyes and Google MyMaps to collaborate with the newsroom and build data-heavy applications even faster. 7th January 2008, 9:24 pm

OpenID and Google’s Blogger. Blogger gets it wrong by displaying a nickname derived from the OpenID URL (in Malcolm’s case, “blog”) instead of the user entered nickname. 30th December 2007, 10:35 am

EU: Microsoft’s Last Stand Against Google’s Acquisition of DoubleClick. Notable for some truly incomprehensible chartjunk from Microsoft. 27th December 2007, 12:26 pm

David Airey: Google’s Gmail security failure leaves my business sabotaged (via) Gmail had a CSRF hole a while ago that allowed attackers to add forwarding filter rules to your account. David Airey’s domain name was hijacked by an extortionist who forwarded the transfer confirmation e-mail on to themselves. 26th December 2007, 12:16 pm

Google Reader ruins Christmas (via) New sharing feature automatically reveals shared items to Gmail contacts, causing political rows. 25th December 2007, 2:59 pm

ExtInfoWindow 1.0: Ajax powered, CSS customization. Finally, a semi-official way of creating customised info windows for the Google Maps API. You lose the default shadow but gain the ability to style the entire info window using CSS. 15th December 2007, 12:22 pm

Negative numbers in the Google Chart API. Stuart has some ingenious tricks for showing negative values on Google Charts, based on transforming the data to positive values and then relabeling the axes. 8th December 2007, 9:03 am

Google Chart API (via) Really neat charting API from Google—simply encode your chart data and configuration options in to a URL and Google will serve up a nicely rendered PNG. No API key required. It’s like a documented version of the Google Groups rounded corners API. 6th December 2007, 5:37 pm

Blogger: OpenID commenting (via) I may be wrong, but I think this is the first Google property to support OpenID in any way. 30th November 2007, 7:10 pm

google-axsjax (via) “The AxsJAX framework can inject accessibility enhancements into existing Web 2.0 applications using any of several standard Web techniques”—including bookmarklets and Greasemonkey. The enhancements conform to W3C ARIA, supported by Firefox 2.0 and later. 14th November 2007, 5:18 pm

Gmail Greasemonkey API (via) The new version of Gmail includes API hooks for Greasemonkey script authors. The documentation is by Mark Pilgrim, author of Greasemonkey Hacks. 7th November 2007, 10:38 am

Marc Andreesen on Open Social. Marc describes it as an open standard for implementing Facebook style “containers” that other applications can live in. My initial assumption that it was an implementation of the Social Graph paper ideas was incorrect. 31st October 2007, 4:58 pm

Google Announces the OpenSocial API. I doubt the similarity between this and Brad Fitzpatrick’s social graph paper are a coincidence—what IS impressive is that he only joined Google a couple of months ago. 31st October 2007, 4:34 pm

Mailplane (via) A commercial OS X Gmail client built around a site-specific browser. 25th October 2007, 7:57 am

The password anti-pattern. What I don’t understand is why Google / Yahoo! / other webmail providers haven’t just deployed a simple OAuth-style API for accessing the address book. Sites have been scraping them for years anyway; surely it’s better to offer an official API than continue to see users hand out their passwords? 12th October 2007, 9:25 am

BarCampLondon3. 24th-25th of November in Google’s London offices (by Victoria train station). The last BarCamp London was a blast—I’m really looking forward to this. 10th October 2007, 5:20 pm

Google Maps, HTML version. Google’s mostly undocumented accessible version of Google Maps. Robin Christopherson demonstrated this yesterday at FOWA. 4th October 2007, 9:31 am

Google GMail E-mail Hijack Technique. Apparently Gmail has a CSRF vulnerability that lets malicious sites add new filters to your filter list—meaning an attacker could add a rule that forwards all messages to them without your knowledge. 27th September 2007, 10:29 am

Firefox 3 Antiphishing Sends Your URLs To Google. Stories like this crop up every now and then, but no one ever seems to mention that the Google Toolbar has been doing this since it was released (more than five years ago) provided you have PageRank display turned on. 25th September 2007, 11:04 pm

Google To “Out Open” Facebook On November 5. “Google will announce a new set of APIs on November 5 that will allow developers to leverage Google’s social graph data. They’ll start with Orkut and iGoogle (Google’s personalized home page), and expand from there to include Gmail, Google Talk and other Google services over time.” 21st September 2007, 11:23 pm

Google Maps API gets clickable polylines and polygons. Interesting explanation of how they optimised calculating the distance to the nearest point on a polyline. 8th September 2007, 12:58 pm

XFML (via) Throwing the new home for the XFML specification some Google juice; the domain name got nabbed by a squatter. 30th August 2007, 10:27 pm

Ganeti (via) New from Google (developed in the Zurich office): virtual server management tool designed to “facilitate cluster management”, built on top of Xen. 30th August 2007, 9:51 pm

Google Web Toolkit: Towards a better web. Good overview of why GWT exists, but I take exception to the title: requiring JavaScript to even display something does not make the web “better”. 29th August 2007, 8:21 pm

An update on Google Video feedback. Google are now offering a real refund to everyone who bought a video, and are letting people keep the Google Checkout credit as well. Purchased videos will keep working for six months. 21st August 2007, 7:54 am

Brighton geek venues. Nat’s latest project: a neat Google Maps mashup listing venues for geek events in Brighton, managed using Google MyMaps to edit a KML file. 16th August 2007, 1:38 am

Microformats in Google Maps (via) No doubt thanks to the influence of Kevin Marks. 31st July 2007, 11:36 pm

tesseract-ocr. Open source OCR, sponsored by Google. I just sat in on a talk on this at OSCON and the complexity of the problem is pretty incredible. 26th July 2007, 8:23 pm

YouTube Scalability Talk. Kyle Cordes’ notes on a Google Tech Talk on scaling YouTube by Cuong Do. 14th July 2007, 10:26 pm

Google Translate (beta). Google’s beta translator based on statistical analysis of things like the United Nations corpus. I have no idea how long this has been available; it isn’t linked from their homepage. 3rd July 2007, 4:43 pm

Google Health Advertising Blog: My opinion and Google’s (via) A follow up to the post I linked to earlier. 1st July 2007, 9:22 pm

Does negative press make you Sicko? (via) Google’s Health Advertising Blog encourages the healthcare industry to buy ads against Sicko as part of an “issue management campaign” to help “educate” the public. Creepy. 30th June 2007, 6 pm

My Google Tech Talk on OpenID. I gave this extended and improved version of my “Implications of OpenID” talk at Google on Monday. Fast turnaround on the video! 28th June 2007, 8 am

google-diff-match-patch (via) Robust algorithms to perform the operations required for synchronizing plain text, in Java, JavaScript and Python. 9th June 2007, 6:15 pm

Semi-synchronous replication for MySQL (via) Google’s patch for MySQL which enables more reliable master-slave replication (a transaction isn’t committed until at least one slave has replicated the data). 5th June 2007, 10:07 pm

Google Gears DB Abstractions. Here come the ORMs. 4th June 2007, 8:54 am

Cross Domain Frame Communication with Fragment Identifiers. Google are using this crazy iframe/fragment trick for their new Mapplets API. 31st May 2007, 2:15 pm

Apollo will include Google Gears technology. Looks like Google really worked on the partnerships for this one. 31st May 2007, 8:30 am

Dojo Offline on Google Gears. “The great news is that the Dojo crew were in the loop wrt this project, and Brad has ported Dojo Offline to use Google Gears as the base platform.” 31st May 2007, 8:28 am

RSS Bling goes Offline with Google Gears. Google Gears is Google’s new offline JavaScript framework. Dion Almaer (a Google employee) has a nice example of code using Google Gears on Ajaxian. 31st May 2007, 8:27 am

The Google Maps Street View team? They’re posing outside the Googleplex so I’m guessing this is the team that built it. 30th May 2007, 12:29 pm

Howto: Google Maps Street View outside US. Add “&gl=us” at the end of the URL to avoid the evil geo IP restriction and play with Google’s latest toy. 30th May 2007, 7:40 am

Top XSS exploits by PageRank. Yahoo!, MSN, Google, YouTube, MySpace, FaceBook all feature. 29th May 2007, 10:07 pm

Google Image Search does faces (via) The undocumented imgtype=face parameter. Kind of creepy. 28th May 2007, 8:08 pm

The Oxford Guide free WiFi plotted on Google Maps. The guide offers a geocoded Atom feed which can be directly plotted on a Google Map. 26th May 2007, 9:55 am

A whole new experience for Google Analytics. I absolutely cannot wait to get my hands on the new interface. Maybe I’ll finally be able to find the search referrals page! 8th May 2007, 9:09 pm

Introduction and Yahoo! Pipes. The official Google Maps API blog describes how to plot KML output from Yahoo! Pipes. 3rd May 2007, 10 pm

Seven JavaScript Techniques You Should Be Using Today (via) Sound advice from Dustin Diaz, who is now a Googler. 24th April 2007, 8:20 am

Google AJAX Feed API (via) Simple cross-domain proxy to allow JavaScript to access any publically addressable syndication feed, with the same logic as Google Reader providing normalisation. 18th April 2007, 5:29 pm

Google Reader Theme. Jon Hicks’ beautiful alternative skin for Google Reader, installable as a user stylesheet for various browsers. 16th April 2007, 10:03 pm

Google My Maps: Bodeans. I’ve been talking about how useful a simple tool for creating custom maps would be for ages... looks like Google beat me to it. Here’s one I created showing the location of Bodeans, an excellent Kansas-style BBQ joint in Soho, London. It’s a shame the URLs suck. 5th April 2007, 5:40 pm

How to beat Google, part 1. Rick Skrenta with 12 steps to taking on Google in the search engine space, including some great insights in to smart UI design. 27th March 2007, 12:02 am

New Open Source Utility Library for the Google Maps API (via) Google are taking a hybrid approach to development on their Maps API—an open source utility library layered on top of their closed source, obfuscated core code. 23rd March 2007, 9:09 pm

My photos tagged “cheese” on a Google Map. You can paste a Flickr GeoRSS feed directly in to the Google Maps query box. 23rd March 2007, 1:55 am

KML and GeoRSS support added to the Google Maps API. Since Flickr can output GeoRSS, this means you can now plot your Flickr photos on a Google Map (if you’re so inclined). 23rd March 2007, 1:03 am

Two visions. It looks like Mark Pilgrim is going to be joining Hixie at Google. 20th March 2007, 8:32 am

Google Video: How do I enter transcripts? Neat feature of Google Video I hadn’t seen before: you can upload timestamped transcripts of your videos. Anyone seen a video that uses these? 12th March 2007, 10:44 pm

Google Seattle conference on scalability. Google are hosting a conference on scalability in Seattle on June 23rd. They’ve just put out the CfP. 10th March 2007, 4:37 pm

Add OpenSearch to your site in five minutes. OpenSearch is easy. DeWitt demonstrates how you don’t even need a site search engine to implement it if you take advantage of Google’s site: operator. 9th February 2007, 12:52 am

This site may harm your computer. Tom Dyson’s personal weblog was flagged by Google as hosting malicious software, without any clue as to what the problem was. Sure looks like a false positive to me. 5th February 2007, 9:26 am

On Space Art in Sebastopol... Awesome. Our giant mosaic space invaders are going to show up on Google Earth! 22nd January 2007, 10:44 pm

Designing Google Reader’s trends. “But beyond the visualization, this serves as a good example of collecting and understanding the ambient information that flows through our digital lives.” 15th January 2007, 12:53 am

Details of Google’s Latest Security Hole. For a brief while you could use Blogger Custom Domains to point a Google subdomain at your own content, letting you hijack Google cookies and steal accounts for any Google services. 14th January 2007, 1:36 pm

MacFUSE: FUSE for Mac OS X. Mac support for user-space custom file systems, API compatible with those already written for Linux. Amit Singh runs kernelthread.com; I hadn’t realised that he had moved to Google. 12th January 2007, 9:47 am

Beginning of the end for open web data APIs? Google just ditched their SOAP API in favour of a crippled Ajax widget. What are the implications for other free-as-in-beer APIs? 20th December 2006, 12:44 am

Google Code gets wikis and file downloads. Someone finally wrote a project wiki that stores its pages inside the Subversion repository. 16th December 2006, 12:35 pm

Google’s own cornershop. Google groups has an undocumented API for generating rounded corners. 14th December 2006, 7:34 pm

Making GWT Better. Explains the philosophy behind GWT. It’s all about the tools! 12th December 2006, 5:53 pm

GWT 1.3 Release Candidate is 100% Open Source. At least you can see how the code generator works now. 12th December 2006, 5:50 pm

Google Mondrian. Internal Google application, powered in part by Django! 1st December 2006, 11:27 am

Good Agile, Bad Agile. Includes interesting insight in to Google development processes. 27th September 2006, 3:10 pm

Chris Shiflett: Google XSS Example (via) UTF-7 is a nasty vector for XSS. 24th December 2005, 5:21 pm

Maps released. Google Maps Safari support is being worked on. 8th February 2005, 12:03 pm

Google Search: spong monkeys. I’m second. Rock! 25th February 2004, 3:32 pm