OpenIDHack Day tools for non-developers
We’re about to run our second internal hack day at the Guardian. The first was an enormous amount of fun and the second one looks set to be even more productive.
There’s only one rule at hack day: build something you can demonstrate at the end of the event (Powerpoint slides don’t count). Importantly though, our hack days are not restricted to just our development team: anyone from the technology department can get involved, and we extend the invitation to other parts of the organisation as well. At the Guardian, this includes journalists.
For our first hack day, I put together a list of “tools for non-developers”—sites, services and software that could be used for hacking without programming knowledge as a pre-requisite. I’m now updating that list with recommendations from elsewhere. Here’s the list so far:
Freebase
Originally a kind of structured version of Wikipedia, Freebase changed its focus last year towards being a “social database about things you know and love”. In other words, it’s the most powerful OCD-enabler in the history of the world. Create your own “Base” on any subject you like, set up your own types and start gathering together topics from the millions already available in Freebase—or add your own. Examples include the Battlestar Galactica base, the Tall Ships base and the fabulous Database base. If you are a developer the tools in the Make Things with Freebase section are top notch.
Dabble DB
Dabble is a weird combination of a spreadsheet, an online database and a set of visualisation tools. Watch the 8 minute demo to get an idea of how powerful this is—you can start off by loading in an existing spreadsheet and take it from there. You’ll need to sign up for the free 30 day trial.
Google Docs
You can always build a hack in Excel, but Google Spreadsheets is surprisingly powerful and means that you can collaborate with others on your hack (including developers, who can use the Google Docs API to get at the data in your spreadsheet). Check out the following tutorials, which describe ways of using Google Spreadsheets to scrape in data from other webpages and output it in interesting formats:
- Data Scraping Wikipedia with Google Spreadsheets
- Calling Amazon Associates/Ecommerce Web Services from a Google Spreadsheet
There’s also a simple way to create a form that submits data in to a Google Spreadsheet.
Yahoo! Pipes
Visual tools for combining, filtering and modifying RSS feeds. Combine with the large number of full-content feeds on guardian.co.uk for all sorts of interesting possibilities. Here’s a tutorial that incorporates Google Docs as well.
Google My Maps
Google provide a really neat interface for adding your own points, lines and areas to a Google Map. Outputs KML, a handy file format for carting geographic data around between different tools.
If you already have a KML or GeoRSS feed URL from somewhere (e.g. the output of a Yahoo! Pipe), you can paste it directly in to the Google Maps search box to see the points rendered on a map.
Google SketchUp
A simple to use 3D drawing package that lets you create 3D models of real-world buildings and then import them in to Google Earth.
OpenStreetMap
Try your hand at some open source cartography on OpenStreetMap, the geographic world’s answer to Wikipedia. If you have the equipment you can contribute GPS traces, otherwise there’s a clever online editor that will let you trace out roads from satellite photos—or you could just make sure your favourite pub is included on the map. The export tools can provide vector or static maps, and if you export as SVG you can further edit your map in Illustrator or Inkscape.
CloudMade Maps
Commercial tools built on top of OpenStreetMap, the most exciting of which allows you to create your own map theme by setting your preferred colours and line widths for various types of map feature.
Many Eyes
IBM Research’s suite of data visualisation tools, with a wiki-style collaboration platform for publishing data and creating visualisations.
Dapper
Dapper provides a powerful tool for screen scraping websites, without needing to write any code. Output formats include RSS, iCalendar and Google Maps.
TiddlyWiki
TiddlyWiki is a complete wiki in a single HTML file, which you can save locally and use as a notebook, collaboration tool and much more. There’s a large ecosystem of plugins and macros which can be used to extend it with new features—see TiddlyVault for an index.
WolframAlpha
The “computational knowledge engine” with the hubristic search-based interface, potentially useful as a source of data and a tool for processing and visualising that data.
Tumblr
Useful as both an input and an output for feeds processed using other tools, and with a smart bookmarklet for collecting bits and pieces from around the web.
The UCSB Toy Chest
An outstanding list of tools that people “without programming skills (but with basic computer and Internet literacy) can use to create interesting projects”, compiled by the English department at UC Santa Barbara.
Your help needed
There must be dozens, if not hundreds of useful tools missing from the above. Tell me in the comments and I’ll add them to the list.
Dapper lets you screen-scrape a Website into RSS/Atom, XML, and even Web hook POST: http://www.dapper.net/open/
Putting GeoRSS or KML into the Google Maps search box is probably even easier than MyMaps.
Quite a lot of non-developers find TiddlyWiki useful for editing and repurposing data, especially from wikis (there are adaptors for MediaWiki, Confluence, etc)
Thanks Paul, I'll add all three.
Tumblr and twitterfeed work well with yahoo pipes for quick code free hacks
Dan W - 28th July 2009 16:27 - #
The obvious missing thing here's Excel, isn't it? I've seen some amazing things done with it alone.
The next one is is a little shameless of me, but I've got to mention Timetric (http://timetric.com/), haven't I? The formula language (http://timetric.com/help/formulae/) lets you build a lot of the things you'd want to do if Excel were real-time...
For people who don't program but want to, the best ever programming gateway drugs I've seen are Nodebox (basically Processing done in Python) and Processing itself: http://nodebox.net/code/index.php/Home, http://processing.org/. It might be worth having a copy of Ben Fry's "Visualizing Data" kicking around – it's an amazing book.
Google Fusion Tables, maybe? Wolfram Alpha too, arguably.
For publishing, there's tumblr and soup.io, which can reblog RSS feeds – an easy way of displaying the output of your Pipe in a cool way, I reckon..
Andrew Walkingshaw - 28th July 2009 16:35 - #
YQL just crosses over into the developer side, but given the console with its mmany examples, you could include it. Google Charts API creates graphics with URLs. And the Google API sandbox is helpful for dipping toes into their APIs.
Will you tag these in Delicious (which I've also used for Y!pipes hacks)? Would be nice to have a collaborative tag there with these descriptions...
For any mapping need Mapspread (http://mapspread.com/) offers accurate geo-coding without coding: Just upload a spreadsheet with your adresses, let the magic happen and publish the data as KML, Geo-RSS, CSV or XLS. Also Mapspread let's you build complete map applications.
The Google Gadget from FusionCharts (http://www.fusioncharts.com/gg/) provides also the possibility to display data using various visualization methods.
Benjamin Wiederkehr - 28th July 2009 17:04 - #
Google AJAX API Playground
(http://code.google.com/apis/ajax/playground/)
This is definitely on the techie end of the spectrum, but could be useful for power users.
I know you have Dabble listed, but the Magic/Replace tool they built for cleaning data can be really handy for less technical folks:
http://cleanupdata.com/
Here are a few more suggestions for your list:
For creating interactive timelines:
Dipity: http://www.dipity.com/
Timetoast: http://www.timetoast.com/
Simile widgets: http://www.simile-widgets.org/
For converting PDF, JPG, GIF, TIFF or BMP files to text (i.e., OCR): http://www.free-ocr.com/
For creating charts and maps in Flash: http://www.amcharts.com/
Chrys Wu - 28th July 2009 18:07 - #
A visual programming interface like vvvv? http://vvvv.org/tiki-index.php
Batch Geocode is good for running through a table of place names and plotting them on a map (KML output too): http://www.batchgeocode.com/
GeoCommons Finder and Maker can also be handy for making maps: http://www.geocommons.com/
It's good, but there are lots of alternatives to Google My Maps. http://www.umapper.com/ is a good one.
Google's Visualisation API is a good complement to their Spreadsheets http://code.google.com/apis/visualization/
I still use http://delicious.com for collections of links too. Can't beat it if the data's already arranged online and you want to gather and categorise a subset of it.
Tom Carden - 28th July 2009 18:27 - #
I bet the Twitter search Atom feed would integrate well with Yahoo! pipes.
I see you've got Freebase in the list. It works well with semantic analysis.
An example of what it can do: http://www.zemanta.com/blog/api-showcase-zemanta-f or-freebase-acre/
Zemanta has up to 10,000 API calls per day for free. Open Calais also has a free API.
http://www.opencalais.com/
http://www.zemanta.com/api/
On that note, Programmable Web has a big list of APIs, a bunch of them free and with libraries: http://www.programmableweb.com/apis/
I work at Yahoo! and do every quarterly hackday as well as travel with the hacku program.
For internal hack days, we are all coders already, so no tools are directly provided. A linux box with a webserver is all I use.
For the hacku program (http://developer.yahoo.com/hacku/) we are dealing with university students who are in computer science but not yet 100% proficient in web programming. For them, we usually teach rudimentary HTML, PHP, CSS, and Javascript. They then copy and paste all our examples together and can end up with something neat. The examples are here : http://paulisageek.com/hacku/
So, if you have 1 hour before they start hacking away, I would set up an optional session that the interested people can go to, to learn basic web programming. They'll appreciate learning it and will then have the tools to glue together all the output from your other tools.
As for tools, I have to plug our yahoo stuff. Big upvote for YQL, its my favorite. Placemaker is neat for parsing places from text : http://developer.yahoo.com/geo/placemaker/. Fireeagle is good if you need user's location information : http://developer.yahoo.com/fireeagle/.
Good luck, and if you need help with that, please ask, but the slidedeck on the hacku page should be a good start.
Animoto - pimped up soundslides... (http://animoto.com/)
And Soundslides: http://www.soundslides.com
Ben Parker - 28th July 2009 21:20 - #
well in addition to timetric and many-eyes I'd add verifiable.com and swivel.com for good balance.
apart from that, in addition to google docs there's the google chart API (http://code.google.com/apis/chart/) which is really nice for simple, static charts, and google fusion tables (http://tables.googlelabs.com/) which is nifty too.
Zotero. Wonderful FF extension for collecting and indexing text from the web, from libraries, and from journals. The current version only works on FF3 which means it's not yet installable at the grauniad. But it's an indispensable tool for some kinds of journalism. It will soon be possible to share collections of data.
www.zotero.org
Very small hack useful a hundred times a day when you're editing on eof th epaper's sites -- urlbarext. http://bit.ly/2Ljqhn
Andrew Brown - 29th July 2009 09:04 - #
yubnub.org
is the command line utility to access your bookmarks in far less clicks!
Shantanu Oak - 29th July 2009 09:59 - #
A while ago I worked on a list like this for English students, and one of the items there is Scratch, for making little animations and games.
Britta G - 29th July 2009 10:14 - #
Britta: that list is amazing. Adding it now.
Don't know whether this counts, but although http://ccn.jottit.com might need a geek to install the server, it simply uses a private friendfeed room as "admin interface", and embedding the widget itself in a blog or such should be easy enough for non-geeks.
It's currently being used in the campaign against the biometric database [brrr] in Israel and could be handy for other activist orgs.
Our ZXTM is available at no cost for developers. This license is restricted in performance but has the full functionality.
A great tool for making stuff happen at the application/network junction...
http://www.zeus.com/downloads/zxtm.html
Nick Bond - 29th July 2009 14:00 - #
Awesome list. I'm going to add a bunch of this stuff to http://toolkit.snd.org. There's already a bunch posted there under Data Scraping, Data Visualization and just plain Databases.
Chris Amico - 29th July 2009 23:17 - #
Seems like you should have Pachube there in that list: http://www.pachube.com/ a whole world of realtime sensor and environment data, lots of mashups with twitter, pipes, second life, sketchup, augmented reality
Turtle - 30th July 2009 11:58 - #
I'll second Britta's special mention of Scratch. I think it should have its own place on this list.
I've used it to successfully teach a group of 10 year olds how to program for the first time. It is an amazingly friendly environment for people who are interested in programming but would otherwise be paralyzed by a traceback.
http://scratch.mit.edu/
Keyvan - 9th August 2009 03:36 - #
This is an awesome list! I think my next day off will be kinda intriguing.
I did not know some of these ones, good list. Thanks !
Jeroen Weiqi - 13th August 2009 16:05 - #
Sounds like fun! You should include Tornado in this list.
At our next hack day I'll be doing something useful with Open ID. Monster just released a new version of his Silverlight widget which facilitates seriously quick & easy implimentation of Open ID
http://mark.mymonster.nl/2009/09/11/openid-user-co ntrol-in-silverlight-part-2-openid-integration/
Jean - 12th September 2009 21:37 - #
Can't believe you couldn't use PowerPoint... :-)
Frank - 18th September 2009 17:20 - #
wow````this is a good article,a good list.i will keep it.add to my favorites
eric - 19th September 2009 09:43 - #
Thanks for the info!
Much appreciated =)
richard williams - 21st September 2009 01:31 - #
Is the person doing the voice for Google my maps 7 years old?
Veritas - 22nd September 2009 07:18 - #