Magical Maps: An Evening at the Art of Mapping Exhibition

I was at a talk last night organised by the Londonist – Magical Maps. It took place in the TAG Fine Arts‘ Air Gallery in Mayfair. TAG Fine Arts currently have an exhibition there – The Art of Mapping, and the audience was surrounded by the various artworks – some made from maps, some as interpretations of maps and some simply with a geographical theme.

The speakers were Stephen Walter, an artist who drew the “Island” London map that was a hit at the British Library’s recent Magnificent Maps exhibition; John Kennedy, a London cabbie, blogger and chronologer of bollards and other London objects; and UCL CASA’s very own James Cheshire, recent PhD, now lecturer.

James was up first and showed some of the recent visualisation work by CASA. You can see most of what he showed in an article here.

Then John explained The Knowledge, taking the audience on a verbal taxi journey, from the gallery to the Elephant and Castle – “the true centre of London” – and back, taking into account various banned turns and other taxi restrictions! His journey was peppered with anecdotes of the various places encountered.

Finally, Stephen outlined his artistic career, showing how he switched from traditional photography and painting, to producing the monochromatic repeated symbols on landscapes, that envolved into maps and finally his famous Island map. He also showed some works in progress, such as a map of London’s underground structures (tube lines, water pipes, the Mail Rail etc).

Matt from the Londonist then chaired a Q&A session with the panel, and finally there was another chance to look around and think about the artworks on the walls. I was most intrigued by the below image, which lists and shows all the bridges across the Thames in London, the top line being Teddington Lock and the bottom being the Thames Barrier – but I don’t know why the lines cross over and wiggle. Perhaps it is just purely artistic, but the data visualiser in me hopes there is a deeper meaning and an embedded infographic…

An excellent evening with three very different but equally engaging speakers and some very interesting things to look at.

The exhibition is open for another week, it’s quite small and free, and there is a complementary exhibition book, so take a visit to Mayfair and have a look! There is a talk tomorrow by some of the artists.

The Tech City Map

Last week the government launched the Tech City map. It is not the first – there have been several previous maps, e.g. by the Guardian, of technology startups in the Old Street/Shoreditch area of London, but this is the highest profile one produced yet. I’m not sure whether a map itself is useful to the community but it serves perhaps to increase the identity and branding of the area – and awareness of it for the general public.

Incidentally, the area covered is also known as the Silicon Roundabout area, it often incorrectly referred to the Westminster set as “East London” (I would more term it as North-East inner-city London). I would consider more for the area around the Olympic Park to be true East London, i.e. east of Charing Cross, north of the river and east of the City – areas such as the East End, Stratford, Barking and West Ham.

As for the cartography, there has been some criticism by some people of the mass of straight blue lines, representing retweets of one company’s message by others. I prefer to see it as a piece of “art” showing the interconnectiveness of the organisations. No one’s trying to do serious quantitative impact work with this, it’s just for visual impact and identification of a new organically-created geographical area.

It’s nice also that the custom style functionality of Google Maps v3 API is being used – giving the streets a nice blue hue rather than the very familiar (and quite distracting) regular colours on Google Maps. Of course Google has recently introduced quite low limits (2500 map loads) for free usage of its custom style functionality, I wonder if this website eventually breaching such a limit if it gets hit with another blaze of publicity?

Perhaps the most exciting news that came out of last week’s announcements was that UCL, Imperial and Cisco are collaborating to build a new Future Cities research facility in the Tech City area, which would be focusing on urban modelling and technology. This is music to my ears – I’m not sure whether UCL CASA, the lab where I work, has plans ever have a presence there but it would be an amazing place to work. Maybe there will be a CASA East in the not too distant future?

WMS and SVG Input with Mapnik and Cairo

A rather techy post from me, just before the Final Project Post for GEMMA (my current project at CASA) is submitted, summing the project and applications up. Why? I wanted to share with you two rather specialist bits of GEMMA that required quite a bit of head-scratching and trial-and-error, in the hope that, for some developer out there, they will be of use in their own work. I’m cross-posting this from the GEMMA project blog.

One of the core features of GEMMA that I have always desired is the ability to output the created map as a PDF. Not just any PDF, but a vector-based one – the idea that it will be razor-sharp when you print it out, rather than just looking like a screen grab. I had written a basic PDF creator, using Mapnik and Cairo, for OpenOrienteeringMap (OOM), an earlier side-project, and because the GEMMA project is about embracing and extending our existing technologies and knowledge at CASA, rather than reinventing the wheel, I was keen to utilise this code in GEMMA.

Most of the layers were quite straightforward – the two OpenStreetMap layers (background and feature) are very similar indeed to OOM, while the Markers and Data Collector layers were also reasonably easy to do – once I had imported (for the former) and hand-crafted (for the latter) suitable SVG images, so that they would stay looking nice when on the PDF. The trickiest layer was the MapTube layer. For the terms of this project, the MapTube imagery is not a vector layer, i.e. we are not using WFS. However I was still keen to include this layer in the PDF, so I turned to Richard Milton (the creator of MapTube) and discovered there is an WMS service that will stitch together the tiled images and serve them across the net. I could combine this requesting the WMS images on the GEMMA server (not the client!), converting them to temporary files, and then using a RasterSymbolizer in Mapnik 2, and an associated GDAL filetype.

The trickiest part was setting georeferencing information for the WMS image. Georeferencing is used to request the image, but it is also needed to position the image above or below the other Mapnik layers. Initially it looked like I would have to manually create a “worldfile”, but eventually I found a possibly undocumented Mapnik feature which allows manual specification of the bounding box.

I’ve not seen this done anywhere else before, although I presume people have just done it and not written it down on the web, so here’s my take, in Python.

First we get our WMS image. MAP_W and MAP_H are the size of the map area on the “sheet” in metres. We request it with a resolution of 5000 pixels per metre, which should produce a crisp looking image without stressing the server too much.

mb = map.envelope()
url = maptube_wms_path + "/?request=GetMap&service=WMS&version=1.3.0"
url = url + &format=image/png&crs=EPSG:900913"
url = url + "&width=" + str(int(MAP_W*5000))
url = url + "&height=" + str(int(MAP_H*5000))
url = url + "&bbox=" + str(mb.minx) + "," + str(mb.miny) + ","
url = url + str(mb.maxx) + "," + str(mb.maxy)
url = url + "&layers=MAPID" + str(maptubeid)

furl = urllib2.urlopen(url, timeout=30)

Mapnik doesn’t work directly with images, but files, so we create a temporary file:

ftmp = tempfile.NamedTemporaryFile(suffix = '.png')
filename = ftmp.name
ftmp.write(furl.read())

Next we set up the layer and style. It’s nice that we can pass the opacity, set on the GEMMA website, straight into the layer in Mapnik.

style = mapnik.Style()
rule = mapnik.Rule()
rs = mapnik.RasterSymbolizer()
rs.opacity = opacity
rule.symbols.append(rs)
style.rules.append(rule)
map.append_style(filename,style)
lyr = mapnik.Layer(filename)

Here’s the key step, where we manually provide georeferencing information. epsg900913 is the usual Proj4 string for this coordinate reference system.

lyr.datasource = mapnik.Gdal(base='',file=filename, bbox=(mb.minx, mb.miny, mb.maxx, mb.maxy)) #Override GDAL
lyr.srs = epsg900913

Finally:

lyr.styles.append(filename)
map.layers.append(lyr)

I’m excited about one other piece of code in the PDF generation process, as again it involves jumping through some hoops, that are only lightly documented – adding an SVG “logo” – the SVG in this case being the GEMMA gerbil logo, that Steve (co-developer) created from Illustrator. Cairo does not allow native SVG import (only export) but you can use the RSVG Python package to pull this in. I’m being a bit lazy in hard-coding widths and scales here, because the logo never changes. There are more sophisticated calls, e.g. svg.props.width, that could be useful.

svg = rsvg.Handle(gemma_path + "/images/logo.svg")
ctx = cairo.Context(surface)
ctx.translate((MAP_WM+MAP_W)*S2P-ADORN_LOGO_SIZE, CONTENT_NM*S2P)
ctx.scale(0.062, 0.062)
svg.render_cairo(ctx)

Note that we are calling render_cairo, a function in RSVG, rather than a native Cairo function that we do for all the other layers in the PDF.

The screenshot above contains data from the OpenStreetMap project.

Olympic Torch Relay – The Unofficial Map

The organisers of next year’s Olympic Games in London, LOCOG, have unveiled their map of the 1000+ places that the Olympic Torch Relay will pass through. The data that the map is built from is readily accessible (as a JSON file that gets downloaded to your computer when you view the map) so I’ve taken the data and built my own (unofficial) map. It has a number of advantages over the official map:

  • The base map is OpenStreetMap, which is much more detailed.
  • The map takes up the whole browser page, allowing for easier panning around.
  • The line that connects each of the places is drawn as a vector, so it still appears as you to zoom right in to see individual villages. (The official map surprisingly uses tiles for the line.)
  • There are Wikipedia links for each of the places. Almost all of these resolve to proper Wikipedia entries, so you can easily find out about the places that have been picked, with the richness of detail that is characteristic of the Wikipedia project.

See it here.

The Ease of Monitoring with Munin

I’m currently using Munin to keep an eye on the various services running on my main server at UCL CASA. Munin is a monitoring package. It is simple to install (on Ubuntu, sudo apt-get install munin on your primary server, sudo apt-get install munin-node on all servers you want to monitor), and brilliantly simple to set up and – most importantly – extend.

Out of the box, Munin will start monitoring and graphing various aspects of your server, such as CPU usage, memory usage, disk space and uptime. The key is that everything is graphed, so that trends can be spotted and action taken before it’s too late. Munin always collects its data every five minutes, and always presents the graphs in four timescales: the last 24 hours, the last 7 days, the last month and the last year.

Extending Munin to measure your own service or process is quite straightforward. All you need is a shell-executable script which returns key/value pairs representing the metric you want to measure. You also need to add a special response for when Munin wants to configure your plugin. This response sets graph titles, information on what you are measuring, and optionally thresholds for tripping alerts.

Here’s a typical response from a munin script “uptime”, this is used by Munin to construct the graph:

uptime.value 9.29

Here’s what you get when you call it with the config parameter:

graph_title Uptime
graph_args --base 1000 -l 0
graph_scale no
graph_vlabel uptime in days
graph_category system
uptime.label uptime
uptime.draw AREA

Munin takes this and draws (or reconfigures) the trend graph, with the data and a historical record. Here’s the daily graph for that:

I have two custom processes being monitored with Munin. The first is my minute-by-minute synchronisation of my database (used by OpenOrienteeringMap and, soon hopefully*, GEMMA) with the UK portion of the master OpenStreetMap database. The metric being measured is the time lag. Normally this is around a minute or less, but if there are problems with the feed at either OSM’s end or (more likely) my end, the graph spikes up and the problem can be spotted and dealt with. Also, the subsequent graphing trend, after such an issue, is useful for predicting how quickly things will be back to normal. I’m using an OSM plugin (part of the Osmosis system, which is also doing the synchronisation) rather than writing my own.

The other process is for monitoring the various feeds I have to around 50 cities around the world, to get their current bikes/spaces information for my Bike Share Map. Munin graphs are useful for spotting feeds that temporarily fail, and then hopefully fix themselves, resulting in distinctive “shark fin” trendlines. If one feed doesn’t fix itself, its shark fin will get huge and I will then probably go and have a look. Above is what the daily graph looks like.

I wrote this plugin myself, in two stages. First, my scripts themselves “touch” a “heartbeat” file (one for each script) upon successful execution. When the feed’s file is touched, its modified timestamp updates, this can then be used as a basis for determining how long ago the last successful operation is.

Secondly, my Munin plugin, every five minutes, scans through the folder of heartbeat files (new ones may occasionally appear – they go automatically onto the graph which is nice) and extracts the name and modified timestamp for each file, and reports this back to Munin, which then updates the graphs.

Because Munin allows any shell-executable script, I was able to use my language du-jour, Python, to write the plugin.

Here it is – latest_dir is an absolute path to the parent of the heartbeats directory, scheme is the name of the city concerned:

#!/usr/bin/python

import os
import time
import sys

filedir = latest_dir + '/heartbeats/'

files = os.listdir(filedir)
files.sort()

if len(sys.argv) > 1:
	if sys.argv[1] == 'config':
		print "graph_title Bike Data Monitoring"
		print "graph_vlabel seconds"
		for f in files:
			fp = f[:-6]
			print fp + ".label " + fp
		exit()

for f in files:
	tdiff = time.time() - os.path.getmtime(filedir + f)
	fp = f[:-6]
	print fp + ".value " + str(tdiff)

The middle section is for the config, the bit that is used every five minutes is just the gathering of the list of files at the top, and the simple measurement at the bottom.

That is really it – there is no other custom code that is producing the graphs like the one at the top of this post – all the colours, historical result storing and HTML production is handled internally in Munin.

My code that updates the heartbeat file is even simpler:


def heartbeat(scheme):
	open(latest_dir + '/heartbeats/' + scheme + '.touch', 'w').close()

Fingers crossed things won’t go wrong, but if they do, I now have a pleasant, graphical interface to spotting them.

* GEMMA will use OpenStreetMap data for the whole world, but currently it takes longer than a minute to process a minute’s worth of OpenStreetMap database updates, such is the level of activity in the project at the moment, so my minutely-updating database only covers the UK. So, for GEMMA, I am just using a “static” copy of the whole world. OpenOrienteeringMap has two modes, UK and Global – only the UK one uses the updating database.

So Just How Many Bikes are there in the Barclays Cycle Hire scheme?

About 6,600 according to the Mayor and cited in this Evening Standard article. But these include the bikes in storage, such bikes are not much help to people wanting to use them on the streets of the capital.

A more accurate figure, that is the number of bikes on the streets of London, that you can actually use, would be about 4900. The number (shown in the graph with the blue dots, indicating maximum values) does fluctuate a lot though:

One interesting feature on the graphs is the four red dots on their own, on the left hand side (i.e. August-December 2010) which are noticeably lower than the others. The red dots show the minimum number of bikes available (i.e. the maximum number of bikes being used) on that day. These four dots in particular, represent a four very busy day for the bike share scheme – they correspond to the four tube strike days.

Occasionally the data feed fails, for a day or so, so the maximum and minimum numbers combine. It is normally reasonable to assume that the maximum value for a particular day shows the number of bikes available on the streets, as probably fewer than 10 bikes will be being ridden at some moment during the day – the usage minimum is typically at around 4am.

V’Lille and Bike in Bath Online – Huge Lille Figures

I’ve added Lille and Bath to the Bike Share Map. Lille’s system, V’Lille, was soft-launched a few weeks ago, while Bike in Bath, a very small system run by Bicincitta, also launched recently. Bath’s uptake so far seems to have been virtually non-existent – with no rides yet today – however Lille’s usage has been nothing short of spectacular – yesterday, nearly 70% of all the bikes in the system were on the streets rather than in the docks. In fact, Lille tops the following table which shows my estimated maximum use of bike share bikes across 17 cities yesterday. In calculating these numbers, I’ve assumed that at some point in the night before/after, no bikes were being used, and that any change in the total number of docks is because of faulty bikes. That last assumption is a conservative one, so the actual %s may well be a little higher. I can’t be sure that these are accurate numbers so these stats should not be regarded as being definitive, just (perhaps) indicative.

Lille leads the way and Seoul’s small system (in two seperate suburbs) also had impressive usage. San Antonio in Texas, another small system, is in a surprise third place, with London fourth. Vienna also had a very good day.

Sunday 2 October – Estimated Maximum Simultaneous Use

City Local Time of Maximum Weather No of Bikes in System Max % In Use
Lille Sunday 16:56 Sunny 678 65.9%
Seoul Sunday 16:16 Sunny 301 40.5%
San Antonio Sunday 11:18 Sunny 109 37.6%
London Sunday 15:52 Sunny 4765 36.9%
Vienna Sunday 16:48 Sunny 878 30.6%
Saragossa Sunday 19:44 Sunny 885 25.9%
Bordeaux Sunday 16:38 Cloudy 1254 19.5%
Tel Aviv (Saturday) Saturday 17:10 Part Cloudy 667 18.1%
Washington DC and Arlington Sunday 12:50 Part Cloudy 852 18.0%
Changzhou (China) Sunday 08:34 Rain 1543 16.3%
Miami Beach (4h interval) Sunday 19:30 Part Cloudy 598 16.2%
Dublin (3rd party data) Sunday 18:10 Rain 448 12.5%
Minneapolis Sunday 11:46 Sunny 987 8.4%
Rennes Sunday 18:00 Sunny 801 7.2%
Toronto Sunday 10:42 Rain 772 6.7%
Montreal Sunday 17:32 Rain 4274 4.0%
Melbourne Sunday 11:48 Part Cloudy 541 2.6%

Thanks as ever to Russell from the Bike-sharing Blog for alerting me to Lille and Bath’s go-live.

Picture from Jason Jones on Flickr.

Mockup of NYC Bikeshare

Following on from yesterday’s post about the forthcoming New York City bikeshare, I’ve created a mockup of how the scheme might look like on my Bike Share Map. The mockup uses the most popular locations voted for by people on the NYC DoT website. It scales each docking station size by the number of votes received, and pseudorandomly decides how empty or full each docking station is, based on the initial time of the suggestion and whether or not the suggesting person said they worked near there. It’s set so that stations near where the person said they worked are more likely to be full – hence the cluster of full stations in Lower Manhattan and Midtown, while much of Brooklyn’s stations are quite empty.

There is a clickable, zoomable version here – I’ve tried to keep to the published boundaries of the scheme. It shows 585 docking stations (announced target 600), and 9802 bikes (target 10000), with a total of 20988 docking points across the stations. Every suggested docking station that had at least five votes as of a couple of hours ago, has been included.

This is a scheme that will have twice as many bikes as London, in an area only around 50% bigger – and there’s more water in the area. So the density of stations does look higher. The average size of each station (35 docking points, assuming roughly 2.2 docking points for each bike) is also around 50% bigger than London’s (23 on average).

New York City Bike Share – Details Revealed

It’s been announced today that the Alta Bicycle Company will be operating the huge New York City bike-share that will be likely launching next summer. An informative press release reveals the area of the scheme, which will be slightly larger than London’s existing area, but with roughly twice as many bikes in the system and 50% more docking stations, it will have a slightly higher density of available bikes and stations than here in London. The bike and dock design is likely to be very similar to London’s so will be very familiar to anyone visiting from across the pond – it’s also the same system used in Montreal, Washington DC/Arlington, Minneapolis, Boston, Toronto, Ottawa and Melbourne.

Interestingly the system will be financed entirely privately. I’m sure this will be an immense challenge, as London’s capital and operating costs are high. However London has demonstrated that advertising can be a very good deal for the advertiser concerned if the scheme is a success. (London’s planning overhead and so capital expense is also almost certainly higher than New York’s.) The scheme will run 24/365. NYC gets some pretty intense snowstorms in the winter, but so does Washington DC’s scheme, which also runs throughout the year – with occasional suspensions when it gets really bad.

NY will doubtless be looking to London closely, as it probably is the scheme most similar to New York’s – the same technology, roughly the same size and area (all of the US and Canada’s schemes are much smaller) and London’s topography is also quite similar – a major river bisecting the scheme, a single major business district (although London will cover two with next year’s extension to Canary Wharf) with a separate commercial centre, and a very large public park. Doubtless NY will see huge popularity for the bikes in Central Park on weekends, as London does in Hyde Park, and a big morning “commuter surge” from Brooklyn into Lower Manhattan, just like London’s from the Waterloo area to the City.

Interestingly the proposed area extends deep into Brooklyn, but on Manhattan Island it extends only up to 79th Street – roughly a third of the way up Central Park. I would be surprised if, on scheme launch, there aren’t some docking stations in Central Park that don’t in fact go north of this line. New York’s density and road layout structure means there are ample opportunities for the scheme to grow in the future, too.

Two websites have also gone live – Alta’s NYC Bikeshare has some nice mock-up pictures of the bikes (from which I’ve stolen the above pic) + an NYC Dept of Transportation website allows you to pick where you would like to suggest a docking station.

Very pleased to see a link to my bike map from the Alta site. With both big operators in North America (B-Cycle and Alta) being tacitly supportive of third-party maps such as my own – a big constrast to continental Europe – and the station data format likely to be the same, I have high hopes that we will see the plethora of mobile apps, maps and visualisations from the community expand to cover NYC.

N.B. The map above is my own estimate based on the press release boundary mentions. The final scheme on launch will not necessarily match these boundaries.