Recent entries
Letterboxing on Lundy four days ago
Last week Natalie and I spent a delightful two days with our friends Hannah and Adam on the beautiful island of Lundy in the Bristol Channel, 12 miles off the coast of North Devon.
I’ve been wanting to visit Lundy for years. The island is managed by the Landmark Trust, a UK charity who look after historic buildings and make them available as holiday rentals.
Our first experience with the Landmark Trust was the original /dev/fort back in 2008 when we rented a Napoleonic Sea Fortress on Alderney in the Channel Islands. Ever since then I’ve been keeping an eye out for opportunities to try out more of their properties: just two weeks ago we stayed in Wortham Manor and used it as a staging ground to help prepare a family wedding.
I cannot recommend the Landmark Trust experience strongly enough: each property is unique and fascinating, they are kept in great condition and if you split the cost of a larger rental among a group of friends the price can be comparable to a youth hostel.
Lundy is their Crown Jewels: they’ve been looking after the island since the 1960s and now offer 23 self-catering properties there.
We took the ferry out on Tuesday morning (a truly horrific two hour voyage) and back again on Thursday evening (thankfully much calmer). Once on Lundy we stayed in Castle Keep South, a two bedroom house in the keep of a castle built in the 13th century by Henry III, after he retook the island from the apparently traitorous William de Marisco (who was then hanged, drawed and quartered for good measure—apparently one of the first ever uses of that punishment). Lundy has some very interesting history attached to it.
The island itself is utterly spectacular. Three miles long, half a mile wide, surrounded by craggy cliffs and mostly topped with ferns and bracken. Not a lot of trees except for the more sheltered eastern side. A charming population of sheep, goats, Lundy Ponies and some highland cattle with extremely intimidating horns.
(“They’re complete softies. We call that one Boris because he looks like Boris Johnson”—a lady who works in the Tavern)
Lundy has three light houses (two operational, one retired), the aforementioned castle, a charming little village, a church and numerous fascinating ruins and isolated buildings, many of which you can stay in. It has the remains of two crashed WWII German Heinkel He 111 bombers (which we eventually tracked down).
It also hosts what is quite possibly the world’s best Letterboxing trail.
Letterboxing?
Letterboxing is an outdoor activity that is primarily pursued in the UK. It consists of weatherproof boxes hidden in remote locations, usually under a pile of rocks, containing a notebook and a custom stamp. The location of the boxes is provided by a set of clues. Given the clues, your challenge is to find all of the boxes and collect their stamps in your notebook.
On Lundy the clues can be purchased from the village shop.
I had dabbled with Letterboxing a tiny bit in the past but it hadn’t really clicked with me until Natalie (a keen letterboxer) encouraged us to give it a go on Lundy.
It ended up occupying almost every waking moment of our time there, and taking us to every far-flung corner of the island.
There are 28 letterboxes on Lundy. We managed to get 27 of them—and we would have got them all, if the last one hadn’t been located on a beach that was shut off from the public due to grey seals using it to raise their newly born pups! The pups were cute enough that we forgave them.
To give you an idea for how it works, here’s the clue for letterbox 27, “The Ugly”:
There were letterboxes in lighthouses, letterboxes in ruins, letterboxes perilously close to cliff-faces, letterboxes in church pews, letterboxes in quarries, letterboxes in caves. If you thought that letterboxing was for kids, after scrabbling down more perilous cliff paths than I can count I can assure you it isn’t!
On Thursday I clocked up 24,000 steps walking 11 miles and burned 1,643 calories. For comparison, when I ran the half marathon last year I only burned 1,222. These GPS tracks from my Apple Watch give a good impression of how far we ended up walking on our second day of searching.
When we checked the letterboxing log book in the Tavern on Wednesday evening we found most people who attempt to hit all 28 letterboxes spread it out over a much more sensible timeframe. I’m not sure that I would recommend trying to fit it in to just two days, but it’s hard to imagine a better way of adding extra purpose to an exploration of the island.
Should you attempt letterboxing on Lundy (and if you can get out there you really should consider it), a few tips:
- If in doubt, look for the paths. Most of the harder to find letterboxes were at least located near an obvious worn path.
- “Earthquake” is a nightmare. The clue really didn’t help us—we ended up performing a vigorous search of most of the area next to (not inside) the earthquake fault.
- The iPhone compass app is really useful for finding bearings. We didn’t use a regular compass at all.
- If you get stuck, check for extra clues in the letterboxing log book in the tavern. This helped us crack Earthquake.
- There’s more than one pond. The quarry pond is very obvious once you find it.
- Take as many different maps as you can find—many of the clues reference named landmarks that may not appear on the letterboxing clue map. We forgot to grab an offline copy of Lundy in the Google Maps app and regretted it.
If you find yourself in Ilfracombe on the way to or from Lundy, the Ilfracombe Museum is well worth your time. It’s a classic example in the genre of “eccentric collects a wide variety of things, builds a museum for them”. Highlights include a cupboard full of pickled bats and a drawer full of 100-year-old wedding cake samples.
The subset of reStructuredText worth committing to memory 28 days ago
reStructuredText is the standard for documentation in the Python world.
It’s a bit weird. It’s like Markdown but older, more feature-filled and in my experience signifcantly harder to remember.
There are plenty of guides and cheatsheets out there, but when writing simple documentation for software projects I think there’s a subset that is worth committing to memory. I’ll describe that subset here.
First though: when writing reStructuredText having a live preview render is extremely useful. I use rst.ninjs.org for this. If you don’t trust that hosted version (it round-trips your documentation through the server in order to render it) you can run a local copy instead using the underlying source code.
Paragraphs
Paragraphs work the same way as Markdown and plain text. They are nice and easy.
This is the first paragraph. No need to wrap the text (though you can wrap at e.g. 80 characters without affecting rendering).
This is the second paragraph.
Headings
reStructuredText section headings are a little surprising.
Markdown has multiple levels of heading, each with a different number of prefix hashes:
# Markdown heading level 1
## Markdown heading level 2
..
###### Markdown heading fevel 6
In reStructuredText there is no single format for these different levels. Instead, the format you use first will be treated as an H1, the next format as an H2 and so on. Here’s the description from the official documentation:
Sections are identified through their titles, which are marked up with adornment: “underlines” below the title text, or underlines and matching “overlines” above the title. An underline/overline is a single repeated punctuation character that begins in column 1 and forms a line extending at least as far as the right edge of the title text. Specifically, an underline/overline character may be any non-alphanumeric printable 7-bit ASCII character. […] There may be any number of levels of section titles, although some output formats may have limits (HTML has 6 levels).
This is deeply confusing. I suggest instead standardizing on the following:
=====================
This is a heading 1
=====================
This heading has = signs both above and below, and they extend past the text by a single character in each direction.
This is a heading 2
===================
This is a heading 3
-------------------
This is a heading 4
~~~~~~~~~~~~~~~~~~~
If you need more levels, you can invent them using whatever character you like—but try to stay consistent within your project.
Bulleted lists
As with headings, you can use a variety of characters for these. I suggest sticking with asterisks.
A blank line is required before starting a bulleted list.
* A bullet point
* Another bullet point
If you decide to wrap your text (I tend not to) you must maintain the indentation on the wrapped lines:
* A bulleted list item. Since the text is wrapped each subsequent
line of text must be indented by two spaces.
* Second list item.
Nested lists are supported, but you MUST leave a blank line above the first inner list bullet point or they won’t work:
* This is the first bullet list item. Here comes a sub-list:
* Hello sublist
* Sublist two
* Back to the parent list.
Inline markup
I only use three inline markup features: bold, italic and code.
**Bold text** is surrounded by two asterisks.
*Italic text* is one asterisk.
``inline code`` uses two backticks at either side of the code.
Links
Links are my least favorite feature of reStructuredText. There are several different ways of including them, but the one I use most often (and hence have committed to memory) is this one:
`a link, note the trailing underscores <http://example.com>`__
So that’s a backtick at the start, then the link text, then the URL contained in greater than / less than symbols, then another backtick and then TWO underscores to finish it off.
Why two underscores? Because if you only use one, the text part of the link is remembered and can be used to duplicate your link later on—see example below. In my experience this is more trouble than it’s worth.
A more complex link syntax example (documented here) looks like this:
See the `Python home page`_ for info.
This link_ is an alias to the link above.
.. _Python home page: http://www.python.org
.. _link: `Python home page`_
I can’t remember this at all, so I stick with the anonymous hyperlink syntax instead.
Code blocks
The easiest way to embed a block of code is like this:
::
# This is a code example
print("It needs to be indented")
The :: indicates that a code block is coming up. The blank line after the :: before the indentation starts is required.
Most renderers have the ability to apply syntax highlighting. To specify that a block should have syntax highlighting for a specific language, replace the :: in the above example with one of the following:
.. code-block:: sql
.. code-block:: javascript
.. code-block:: python
Images
There are plenty of options for embedding images, but the most basic syntax (worth remembering) looks like this:
.. image:: full_text_search.png
This will embed an image of that filename that sits in the same directory as the document itself.
Internal references
In my opinion this is the key feature that makes reStructuredText more powerful than Markdown for larger documentation projects.
Again, there in a vast and complex array of options around this, but the key thing to remember is how to add a reference name to a specific section and how to link to that section later on.
Names are applied to section headings, by adding some magic text before the heading itself. For example:
.. _full_text_search:
Full-text search
================
Note the format: two periods, then a space, then an underscore, then the label, then a colon at the end.
The label full_text_search is now associated with that heading. I can link to it from any page in my documentation project like so:
:ref:`full_text_search`
Note that the leading underscore isn’t included in this reference.
The link text displayed will be the text of the heading, in this case “Full-text search”. If I want to replace that link text with something custom, I can do so like this:
Learn about the :ref:`search feature <full_text_search>`.
This syntax is similar to the inline hyperlink syntax described above.
Learning more
I extracted the patterns I describe in this post from the Datasette documentation—I encourage you to dig around in the source code to see how it all works.
The definitive guide to reStructuredText is the reStructuredText Markup Specification. My favourite of the various quick references is the Restructured Text (reST) and Sphinx CheatSheet by Thomas Cokelaer.
I’m a huge fan of Read the Docs for hosting documentation—it’s the key reason I use reStructuredText in my projects. Unsurprisingly, they offer extensive documentation to help you make the most of their platform.
Analyzing US Election Russian Facebook Ads one month ago
Two interesting data sources have emerged in the past few weeks concerning the Russian impact on the 2016 US elections.
FiveThirtyEight published nearly 3 million tweets from accounts associated with the Russian “Internet Research Agency”—see my article and searchable tweet archive here.
Separately, the House Intelligence Committee Minority released 3,517 Facebook ads that were reported to have been bought by the Russian Internet Research Agency as a set of redacted PDF files.
Exploring the Russian Facebook Ad spend
The initial data was released as zip files full of PDFs, one of the least friendly formats you can use to publish data.
Ed Summers took on the intimidating task of cleaning that up. His results are incredible: he used the pytesseract OCR library and PyPDF2 to extract both the images and the associated metadata and convert the whole lot into a single 3.9MB JSON file.
I wrote some code to convert his JSON file to SQLite (more on the details later) and the result can be found here:
https://russian-ira-facebook-ads.datasettes.com/
Here’s an example search for “cops” ordered by the USD equivalent spent on the ad (some of the spends are in rubles, so I convert those to USD using today’s exchange rate of 0.016).
One of the most interesting things about this data is that it includes the Facebook ad targetting options that were used to promote the ads. I’ve built a separate interface for browsing those—you can see the most frequently applied targets:
And by browsing through the different facets you can construct e.g. a search for all ads that targeted people interested in both interests:Martin Luther King and interests:Police Brutality is a Crime: https://russian-ira-facebook-ads.datasettes.com/russian-ads-919cbfd/display_ads?_targets_json=[“d6ade”%2C“40c27”]
New tooling under the hood
I ended up spinning up several new projects to help process and explore this data.
sqlite-utils
The first is a new library called sqlite-utils. If data is already in CSV I tend to convert it using csvs-to-sqlite, but if data is in a less tabular format (JSON or XML for example) I have to hand-write code. Here’s a script I wrote to process the XML version of the UK Register of Members Interests for example.
My goal with sqlite-utils is to take some of the common patterns from those scripts and make them as easy to use as possible, in particular when running inside a Jupyter notebook. It’s still very early, but the script I wrote to process the Russian ads JSON is a good example of the kind of thing I want to do with it.
datasette-json-html
The second new tool is a new Datasette plugin (and corresponding plugin hook) called datasette-json-html. I used this to solve the need to display both rendered images and customized links as part of the regular Datasette instance.
It’s a pretty crazy solution (hence why it’s implemented as a plugin and not part of Datasette core) but it works surprisingly well. The basic idea is to support a mini JSON language which can be detected and rendered as HTML. A couple of examples:
{
"img_src": "https://raw.githubusercontent.com/edsu/irads/03fb4b/site/images/0771.png",
"width": 200
}
Is rendered as an HTML <img src=""> element.
[
{
"label": "location:United States",
"href": "/russian-ads/display_ads?_target=ec3ac"
},
{
"label": "interests:Martin Luther King",
"href": "/russian-ads/display_ads?_target=d6ade"
},
{
"label": "interests:Jr.",
"href": "/russian-ads/display_ads?_target=8e7b3"
}
]
Is rendered as a comma-separated list of HTML links.
Why use JSON for this? Because SQLite has some incredibly powerful JSON features, making it trivial to output JSON as part of the result of a SQL query. Most interestingly of all it has json_group_array() which can work as an aggregation function to combine a set of related rows into a single JSON array.
The display_ads page shown above is powered by a SQL view. Here’s the relevant subset of that view:
select ads.id,
case when image is not null then
json_object("img_src", "https://raw.githubusercontent.com/edsu/irads/03fb4b/site/" || image, "width", 200)
else
"no image"
end as img,
json_group_array(
json_object(
"label", targets.name,
"href", "/russian-ads/display_ads?_target="
|| urllib_quote_plus(targets.id)
)
) as targeting
from ads
join ad_targets on ads.id = ad_targets.ad_id
join targets on ad_targets.target_id = targets.id
group by ads.id limit 10
I’m using SQLite’s JSON functions to dynamically assemble the JSON format that datasette-json-html knows how to render. I’m delighted at how well it works.
I’ve turned off arbitrary SQL querying against the main Facebook ads Datasette instance, but there’s a copy running at https://russian-ira-facebook-ads-sql-allowed.now.sh/russian-ads if you want to play with these queries.
Weird implementation details
The full source code for my implementation is available on GitHub.
I ended up using an experimental plugin hook to enable additional custom filtering on Datasette views in order to support showing ads against multiple m2m targets, but hopefully that will be made unnecessary as work on Datasette’s support for m2m relationships progresses.
I also experimented with YAML to generate the metadata.json file as JSON strings aren’t a great way of representing multi-line HTML and SQL. And if you want to see some really convoluted SQL have a look at how the canned query for the faceted targeting interface works.
This was a really fun project, which further stretched my ideas about what Datasette should be capable of out of the box. I’m hoping that the m2m work will make a lot of these crazy hacks redundant.
Analyzing US Election troll tweets with Datasette one month ago
FiveThirtyEight published nearly 3 million tweets from accounts associated with the Russian “Internet Research Agency”, based on research by Darren Linvill and Patrick Warren at at Clemson University.
FiveThirtyEight’s tweets were shared as CSV, so I’ve used my csvs-to-sqlite tool to convert them and used Datasette to publish them in a searchable, browsable interface: https://russian-troll-tweets.datasettes.com/
The data is most interesting if you apply faceting. Here’s the full set of tweets faceted by author, language, region, post type and account category:
The minimal source code for this Datasette instance is on GitHub.