Simon Willison's Weblog: yahoo-pipes

Twitter, reformatted

2010-03-18T01:10:36+00:00

I wrote a Yahoo! Pipe to clean up Twitter’s RSS feeds—removing the username prefix and filtering out items that begin with “@” or “RT”..

Tags: pipes, rss, twitter, yahoo-pipes

Today's News and Yahoo!'s Developer Program

2009-07-30T12:20:30+00:00

Today's News and Yahoo!'s Developer Program

“For SearchMonkey and BOSS, we currently do not have anything concrete to tell you” ... “We wanted to let you know that today’s news does not affect these products [YUI, YQL, Pipes]”.

Tags: boss, pipes, searchmonkey, yahoo, yahoo-pipes, ydn, yql, yui

Hack Day tools for non-developers

2009-07-28T14:23:53+00:00

We're about to run our second internal hack day at the Guardian. The first was an enormous amount of fun and the second one looks set to be even more productive.

There's only one rule at hack day: build something you can demonstrate at the end of the event (Powerpoint slides don't count). Importantly though, our hack days are not restricted to just our development team: anyone from the technology department can get involved, and we extend the invitation to other parts of the organisation as well. At the Guardian, this includes journalists.

For our first hack day, I put together a list of "tools for non-developers" - sites, services and software that could be used for hacking without programming knowledge as a pre-requisite. I'm now updating that list with recommendations from elsewhere. Here's the list so far:

Freebase

Originally a kind of structured version of Wikipedia, Freebase changed its focus last year towards being a "social database about things you know and love". In other words, it's the most powerful OCD-enabler in the history of the world. Create your own "Base" on any subject you like, set up your own types and start gathering together topics from the millions already available in Freebase - or add your own. Examples include the Battlestar Galactica base, the Tall Ships base and the fabulous Database base. If you are a developer the tools in the Make Things with Freebase section are top notch.

Dabble DB

Dabble is a weird combination of a spreadsheet, an online database and a set of visualisation tools. Watch the 8 minute demo to get an idea of how powerful this is - you can start off by loading in an existing spreadsheet and take it from there. You'll need to sign up for the free 30 day trial.

Google Docs

You can always build a hack in Excel, but Google Spreadsheets is surprisingly powerful and means that you can collaborate with others on your hack (including developers, who can use the Google Docs API to get at the data in your spreadsheet). Check out the following tutorials, which describe ways of using Google Spreadsheets to scrape in data from other webpages and output it in interesting formats:

There's also a simple way to create a form that submits data in to a Google Spreadsheet.

Yahoo! Pipes

Visual tools for combining, filtering and modifying RSS feeds. Combine with the large number of full-content feeds on guardian.co.uk for all sorts of interesting possibilities. Here's a tutorial that incorporates Google Docs as well.

Google My Maps

Google provide a really neat interface for adding your own points, lines and areas to a Google Map. Outputs KML, a handy file format for carting geographic data around between different tools.

If you already have a KML or GeoRSS feed URL from somewhere (e.g. the output of a Yahoo! Pipe), you can paste it directly in to the Google Maps search box to see the points rendered on a map.

Google SketchUp

A simple to use 3D drawing package that lets you create 3D models of real-world buildings and then import them in to Google Earth.

OpenStreetMap

Try your hand at some open source cartography on OpenStreetMap, the geographic world's answer to Wikipedia. If you have the equipment you can contribute GPS traces, otherwise there's a clever online editor that will let you trace out roads from satellite photos - or you could just make sure your favourite pub is included on the map. The export tools can provide vector or static maps, and if you export as SVG you can further edit your map in Illustrator or Inkscape.

CloudMade Maps

Commercial tools built on top of OpenStreetMap, the most exciting of which allows you to create your own map theme by setting your preferred colours and line widths for various types of map feature.

Many Eyes

IBM Research's suite of data visualisation tools, with a wiki-style collaboration platform for publishing data and creating visualisations.

Dapper

Dapper provides a powerful tool for screen scraping websites, without needing to write any code. Output formats include RSS, iCalendar and Google Maps.

TiddlyWiki

TiddlyWiki is a complete wiki in a single HTML file, which you can save locally and use as a notebook, collaboration tool and much more. There's a large ecosystem of plugins and macros which can be used to extend it with new features - see TiddlyVault for an index.

WolframAlpha

The "computational knowledge engine" with the hubristic search-based interface, potentially useful as a source of data and a tool for processing and visualising that data.

Tumblr

Useful as both an input and an output for feeds processed using other tools, and with a smart bookmarklet for collecting bits and pieces from around the web.

The UCSB Toy Chest

An outstanding list of tools that people "without programming skills (but with basic computer and Internet literacy) can use to create interesting projects", compiled by the English department at UC Santa Barbara.

Your help needed

There must be dozens, if not hundreds of useful tools missing from the above. Tell me in the comments and I'll add them to the list.

Tags: freebase, google, google-maps, guardian, hackday, mapping, nondevelopers, openstreetmap, pipes, sketchup, tools, yahoo-pipes, computer-literacy

My Guardian OpenPlatform API'n'Data Hacks'n'Mashups Roundup

2009-03-31T22:04:20+00:00

My Guardian OpenPlatform API'n'Data Hacks'n'Mashups Roundup

Superb collection of Guardian Open Platform mashups from Tony Hirst, all of which use free online tools such as Yahoo! Pipes and Many Eyes. We invited Tony in to give a tech talk at the Guardian last week.

Tags: guardian, manyeyes, mashups, openplatform, tony-hirst, yahoo-pipes

Data Scraping Wikipedia with Google Spreadsheets

2008-10-16T14:37:33+00:00

Data Scraping Wikipedia with Google Spreadsheets

I hadn’t played with =importHTML in Google spreadsheets, which lets you suck in data from an HTML table or list somewhere on the web. This tutorial takes it further, bringing Wikipedia, Yahoo! Pipes and KML in to the mix.

Tags: google-docs, googlespreadsheet, importhtml, kml, mashups, scraping, wikipedia, yahoo-pipes

Questioning Steve Ballmer

2007-10-01T23:57:53+00:00

This morning I attended a half day briefing at Microsoft UK entitled "The Online Opportunity - What Makes a Successful Web 2.0 Start-Up?". Despite the buzzword laden title the event was well worth the trip up from Brighton, mainly due to the Q&A with Steve Ballmer (a pretty rare opportunity).

Of the other speakers my favourite was Brent Hoberman of lastminute.com and more recently mydeco.com. He presented without slides, choosing instead to simply blasting through dozens of lessons he learnt working on lastminute.com, both when it was a plucky startup and once it had morphed in to a large public company. Thankfully Jeremy took copious notes.

I was lucky enough to get the opportunity to throw a question at Steve. I considered asking how he planned to lure open source developers (used to controlling their entire stack) back to Microsoft tools, but another question had touched on patents so instead I asked the following (paraphrased):

This event is all about encouraging startups - but one of the biggest problems a startup faces is that it's almost impossible to invent anything without violating someone's patent. Big companies can use their patent portfolios to defend themselves, but small companies have no way to fight back. It's kind of like the Cold War.

Here are the points I can remember from Steve's answer:

The patent system (in the US) was designed for the industrial revolution and altered once to deal with the pharmaceutical industry. It hasn't yet been updated for software, but some kind of change is obviously needed. That said, we shouldn't throw the baby out with the bath water - patents are still needed to encourage innovation in both large and small companies.
At the moment, it's hard to say if small or big companies benefit most. Steve thinks it's actually the smaller companies - it's rare for a big company to crush a small company with a patent, but you often hear about small companies with a patent and nothing to lose going after the big guys.
Microsoft are lobbying for patent reform both in the US and the European Union.

About half way through Steve's talk the current favourite Microsoft demos got an airing: Popfly (a mashup editor written in Silverlight, reminiscent of Yahoo! Pipes), Seadragon and the awesome Photosynth. I hadn't realised Photosynth was actually available for regular people to play with, although my attempts at getting it working with Parallels on my Mac have sadly failed.

Tags: brent-hoberman, lastminute, microsoft, mydeco, patents, photosynth, popfly, seadragon, silverlight, startups, steve-ballmer, yahoo-pipes

Introduction and Yahoo! Pipes

2007-05-03T22:00:22+00:00

Introduction and Yahoo! Pipes

The official Google Maps API blog describes how to plot KML output from Yahoo! Pipes.

Tags: google, google-maps, maps, xml, yahoo, yahoo-pipes