Category Archives: OpenStreetMap

Extracting Feature Geometries from OpenStreetMap

Posted on by
Reply

I’ve recently been extracting some river geometries for major cities around the world. The data needs to be a list of latitude/longitude coordinates, representing the nodes on the shape for the river concerned.

I’m sure there’s easier ways to do this, but here’s my technique, shown here for Minneapolis. Click the images for larger version.

1. Extract the data from OpenStreetMap. Use the Export function, and draw out the area concerned with a bounding box. Choose OpenStreetMap XML as the format. I originally tried SVG, but this presents you with screen coordinates instead of latitude/longitude pairs.

2. Open the resulting file in Quantum GIS (QGIS). I used QGIS 1.9. You need the OpenStreetMap plugin installed, this will allow the OSM file that was created in Step 1 to be read straight in (in fact you could download the file directly from the OSM servers, if you wanted to).

3. Select the feature you are interested in. My river (actually a waterbank polygon) is a “hairy feature” as it extends well beyond the extent of the data that was downloaded. Make sure you are selecting it (feature turns yellow) rather than highlighting it for feature information (feature turns red). Otherwise, the subsequent file is, rather unhelpfully, blank.

4. Do Layer > Save Selection as Vector File. Choose “KML” as the format. You probably don’t need to change the coordinate reference system (CRS) as the data will already be in WGS 84, and this (“normal GPS-style latitude/longitude) is the CRS you want.

5. Edit the resulting file, removing the XML tags, and header/footer, and replace spaces with return characters, to leave a long list of latitude/longitudes, ready for importing into your visualisation code.

Run Every Street in Edinburgh – in Strict Alphabetical Order

Posted on by
Reply

…it sounds like one heck of a lot of running. But Murray Strain, one of Scotland’s top terrain runners, is counting on it for his basic training. He’s logging the whole venture, which is based on his trusty Edinburgh A-Z. If two adjacent streets with very similar names are nonetheless separated in the A-Z index by one on the far side of the city, it means a couple of legs right across the city.

Since he started the exercise last year Murray’s got through all the As, and is currently midway through the Bs. I’ve produce a couple of GEMMA maps, one showing the A-Bs (above, As are red and Bs are orange) and one showing the A-Gs (below, in rainbow order). That’s a lot of streets. N.B. The maps in fact show all linear features in the area in OpenStreetMap, so the odd named cycleway and waterway has crept in there too. But the ~95% of the coloured lines will be the streets that Murray will be run.

In order to produce the map, I’ve added a new feature to GEMMA – it now allows you specify only one desired geometry type, i.e. points, lines OR polygons, when adding an OpenStreetMap layer to your map. Previously, you got all three types, although you could reduce each to a dot if desired. This example also highlights the need for legends on the PDF maps that GEMMA produces – a larger coding change, so one that would make it into a future version 2 of GEMMA.

Main Street UK

Posted on by
Reply

GEMMA is the project I’ve been working on for the last six months, it’s one of the JISCgeo projects and it is now released – although consider it to be beta as there are lots of bugs and UI quirks that we are aware of. More about GEMMA can be found on the project’s blog.

One use of the OpenStreetMap feature highlighter in GEMMA, that was suggested by one of the participants at the JISCgeo Meeting earlier this week where we launched the web application, and augmented by a friend who was trying it out, was mapping the occurrences of the “High Street” road name – and a few regional variations, namely Main Street, Front Street, Market Street, Fore Street and The Street. Using GEMMA, and the high level of completion of OpenStreetMap in the UK and Ireland, allows us to visually show the spatial patterns of such street names.

Here’s a stitched-together screenshot of the GEMMA webpage showing the pattern throughout the UK and part of Ireland:

It turns out that Main Street is popular in the Midlands and in Scotland and Ireland, and Front Street is popular in the North-East of England (around Newcastle) while High Street is used nearly everywhere in the UK – but only sparingly in Ireland. Market Street is popular in the Manchester and Devon areas. Fore Street is popular in Cornwall and The Street very popular in Essex and Kent.

Note that many parts of Northern Ireland and the Republic of Ireland, are not yet well mapped in OpenStreetMap, so the street names will be missing in some parts here. The base-map is copyright Google and the street data is CC-By-SA OpenStreetMap.

You can see the live version of the map here.

Olympic Torch Relay – The Unofficial Map

Posted on by
2

The organisers of next year’s Olympic Games in London, LOCOG, have unveiled their map of the 1000+ places that the Olympic Torch Relay will pass through. The data that the map is built from is readily accessible (as a JSON file that gets downloaded to your computer when you view the map) so I’ve taken the data and built my own (unofficial) map. It has a number of advantages over the official map:

  • The base map is OpenStreetMap, which is much more detailed.
  • The map takes up the whole browser page, allowing for easier panning around.
  • The line that connects each of the places is drawn as a vector, so it still appears as you to zoom right in to see individual villages. (The official map surprisingly uses tiles for the line.)
  • There are Wikipedia links for each of the places. Almost all of these resolve to proper Wikipedia entries, so you can easily find out about the places that have been picked, with the richness of detail that is characteristic of the Wikipedia project.

See it here.

Vienna: State of the Map EU

Posted on by
3

So – I was at the State of the Map EU (SotM) conference in Vienna last weekend – a European-focused conference on the OpenStreetMap project. I travelled with my colleague Steven Gray and presented some screenshots from the GEMMA project I am currently working on at UCL CASA – more about that in a later post. The two of us, and London OpenStreetMappers Shaun and Tom, stayed at the shiny new Wombat Nachtmarkt hostel which was convieneintly a few minutes walk from the venue at Vienna University of Technology (TU Wien). I was impressed that, on walking onto the university campus, my phone connected seamlessly to the Eduroam wireless network, based on my UCL credentials – a feat that was not managed in recent trips to more local academic campuses in Manchester and Imperial.

I was impressed with the number of people at the conference – over 200, which was larger than the global SotM conference I was at in Amsterdam two years ago. According to the stats, 2/3rds of people there were from the German-speaking diaspora (Germany, Austria and Switzerland) which demonstrated the clear demand for a SotM conference based here.

I mainly followed the “Tech” track at the conference. I was particularly interested to know about Mapnik Metawriters, which I’m looking to incorporate into some forthcoming Mapnik-based work. You know when you click on Google Map POI icons (not pins) and you get a tooltip with the name of what you’ve clicked on? It’s similar to that. Another highlight included Andy Allan with a tour of custom cartography of OpenStreetMap data. Andy’s cartographic-focused talks are always a visual feast. Unfortunately my own talk clashed, but I managed to make a quick exit after mine and caught the last bit of his.

Another interesting talk was ESRI’s launch of version 2 of their OpenStreetMap editor for ArcGIS – OSMEditor. Of course, you still have to have a copy of ArcGIS in order to be able to use a plugin – so the non-academic, non-commercial audience is unlikely to be using it. I was slightly surprised the presenter didn’t mention the $100 non-commercial licence that is now available for ArcGIS. The $0 price-point for Quantum GIS (which also has an OSM editor plugin) is still going to be unbeatable, but ESRI is certainly going in the right direction. Their engagement with OSM is not something I would have suspected a couple of years ago, it’s great to see them sponsoring and presenting at a conference like this. Of course, having the OSM layer a click away in ArcGIS as a background layer is a good win for them too. And they even let us call them “esri” these days! :-)

Muki Haklay gave an overview of his team’s completeness analysis for the UK OSM dataset over the years. We used to say we “are good enough”. Now we can say that, subject to qualifications, we are “as good as” some traditional datasets. There was also some similar research presented by Heidelberg University, which used hexagonal cartograms, which was an interesting change from grid squares. I should also mention Steve Coast’s keynote, which was a frank statement of the current state of play of the project – good in many places, but problems with the Australian community feeling disengaged and looking to split from the project were clearly top of his mind.

It was great to meet face-to-face with some major figures in the community – notably Frederick Ramm of GeoFabrik. I managed to sit beside him for half an hour at the conference dinner without twigging who he was. Frederick is one of the authors of the OpenStreetMap book that I reviewed – one of my comments was used as a quotation in the book’s advertising at the conference!

Henk Hoff from the OpenStreetMap Foundation was in fine form, with one of his “poster auctions” at the end of the conference. He also announced the winner of the free trip to the “father” SotM conference in Denver in September being Gregory Marler. Gregory won with his Rebecca Black-esque recording “Fly me to SotM” (I hope he doesn’t mind me saying that!)

The social side of the conference was excellent. Plenty of breaks for networking, and a conference dinner on the Friday night. This involved everyone getting a couple of specially hired 1920s wooden trams (or “Bims” after the sound their bells make) to a suburb of Vienna – via the grand ring-road, past the various palaces and other grand buildings – whereupon we took over most of a restaurant for an Austrian feast of Wiener schnitzel, meat loaf, sauerkraut, picked cucumber, and a dessert of apple strudel. A few resturant-brewery combinations were also visited during the trip – along with some most refreshing lagers, served in proper glasses with handles that make a lovely “clonk”. Vienna was very warm indeed, with a thunderstorm on the first night. It was also eerily quiet – the city is quite grand and spaced out, plus maybe many of the locals were on holiday to the mountains. Certainly the people we met were friendly. I should mention specially the conference organisers, which were flawless and ensured everyone was in the right place at the right time! The organisation of the conference and social events appear to go off without a hitch.

It was a great trip to see what’s going on with the OpenStreetMap development community, present some of our own work at CASA, and explore Vienna.

A Historical Comparison of OpenStreetMap’s Completeness in Britain

Posted on by
Reply

Dr Muki Haklay,UCL CEGE, has been carrying out some quantitative research into OpenStreetMap’s coverage in the UK, comparing road lengths in each square kilometre, with those in a definitive national dataset, Ordnance Survey Meridian 2. He’s updated his findings every few months, from March 2008 until March this year. Some interesting research findings have been found, such as a potential correlation between an area’s affluence and the map’s completeness, a possible reflection of a contributor demographic. On his suggestion I’ve taken his dataset and overlaid the red/blue under/overcompleteness maps on OpenStreetMap (or Ordnance Survey StreetView) itself, allowing the specific towns and villages that are missing the OSM love, to be identified.

The mashup can be viewed here.

These days, OpenStreetMap’s coverage is pretty good -often exceeding Meridian’s, as service roads, private roads and alleys, that don’t exist on Meridian 2 are added in. There’s still (as of March 2011) some significant holes though, particularly in parts of Wales, the North East and East Anglia.

Note the first four maps only cover England. There is an interesting artefact in the first one – a square around London can clearly be seen, corresponding to the extent of aerial imagery, in that area, that was available via a special agreement with Yahoo for tracing. Outside of that area, only 50-year-old (out of copyright) maps and contributor GPS traces were available. Since May last year, the Ordnance Survey OpenData release, and Microsoft Bing Aerial imagery, which became available at roughly the same time, has significantly accelerated work on the map. I presented on the diverse sources of data at the Society of Cartographers annual conference last year, you can see the slides here.

ITO World’s OS Locator is just one of a number of tools that the OpenStreetMap contributor community in the UK is using to “complete” the map, moving towards the goal of a comprehensive free database of the UK’s (and world’s) streets.

Hackney: “Complete” on OpenStreetMap

Posted on by
6

ITO‘s OSM Analysis table is updated daily, showing the number of roads in each district/borough in the UK that are in the OS Locator* dataset that are missing in OpenStreetMap. There is an accompanying choropleth map (you need to login to seet it) showing coverage across the UK. Currently southern England and Scotland’s central belt are looking pretty good, while Wales and parts of northern England still have quite some way to go.

As well as the map and summary table, ITO produce a map showing the approximate location of each missing (or misspelled) road, as a rectangular bounding box. This makes finding the missing roads quite easy. Groundwork is needed to check signposts and confirm the names. Often, discrepancies arise simply from OpenStreetMap not having apostrophes for the street name, and the Ordnance Survey having them, or vice versa. The signposts normally provide the definitive answer, but in quite a few cases, the sign at one end of the street will have an apostrophe and the other end will be missing it – or the names attached to houses will differ from the nearby street signs.

Hackney was around 94% complete a few weeks ago, with around 80 errors flagged up. It’s a small borough (in size, if not population) so I reckoned it was possible to bike around all 80 locations in a day – with a suitable route that would hopefully be a good answer to the Travelling Salesman Problem.

In the end, I managed it in a couple of afternoons. I found quite quickly that in most cases, the street was already in OpenStreetMap but just misnamed or unnamed. In around 5-10% of cases, the Ordnance Survey was wrong, and street signs on the ground suggested either that OSM was correct, or that both sources was wrong. As the OS was in general right, I only stopped to note where this wasn’t the case – so, I was able to cover a lot of ground quite quickly. I still ended up having to cycle nearly 100km within a borough that is roughly 4-5km across.

Perhaps the most interesting part of the exercise is it allowed me to visit all parts of the borough, including places I had never been to before despite having lived there for several years. I discovered that Stoke Newington Church Road is indeed massively gentrified out of all proportion to the surrounding area. I found unexpected gems like The Mothers Square (no apostrophe) and that Hackney Downs is quite a pleasant park. I also found the huge Pembury Estate. This was where one of the Ordnance Survey’s mistakes was – the streets were named in the wrong order. Perhaps the original surveyor didn’t like to hang around.

Of course no map can ever be 100% “complete”. Even with a perfect match to OS Locator, the latter may be out of date, or be missing roads due to missed records. The map may be “complete” in terms of roads but other detail still needs to be added.

Anyway Hackney is now 100% complete with respect to OS Locator, making it the 12th such district in the UK. Another 400-odd to go…

* OS Locator is one of the products in Ordnance Survey’s OpenData release.

Review: OpenStreetMap – Using and Enhancing the Free Map of the World

Posted on by
11

OpenStreetMap, the free wiki world map, is starting to come of age. The project is now six years old, and is gradually becoming noticed in wider circles, with AOL and Mapquest producing their own versions of the map, support from Google and Microsoft, and an ecosystem of companies set up around commercialising the data. Perhaps the highest profile the project has had recently was when, in the days following Haiti’s huge earthquake in January 2010, the country was swiftly mapped remotely in high-detail by contributors from around the world, becoming a useful tool for disaster relief teams. Meanwhile, the project continues to expand, with the availability of high-resolution Microsoft Bing imagery causing a big jump in the detail being added to the map recently, and new countries and areas continuing to be worked on actively.

A couple of OpenStreetMap guides became available just before Christmas, and I have one of them – OpenStreetMap: Using and Enhancing the Free Map of the World. It’s published by UIT Cambridge and available on Amazon. The authors are Frederick Ramm, Jochen Topf and Steve Chilton. The book is in its first English edition – Ramm/Topf wrote the previous German editions of the book, while Steve Chilton has translated the work and updated with the latest developments. The authors are a real authority – Ramm/Topf are well known in the German OpenStreetMap community, running a company Geofabrik which builds on OSM, while Steve Chilton (pictured below at an OpenStreetMap conference) is a professional cartographer who has designed and maintained the “standard” OpenStreetMap map you see at http://osm.org/.

The book has a dual purpose – to act as both a guide for using and getting the most out of OpenStreetMap data, and contributing to it. It runs to over 300 pages and is split into four sections. The first section is an introduction, it outlines the basic structure of the project and the community behind OpenStreetMap. This is followed by the longest section in the book, which details how to contribute to the project, from wandering around your local street to using one of the available editors. It includes a detailed guide to Potlatch, the online editor which many people will use when starting out with contributing to the project. It also introduces several other editors, which is good in terms of balance, although new users do not necessarily need to learn more than one.

The third section is about taking OpenStreetMap data and creating maps from it. Examples of cartographical style sheets are included and carefully dissected. There is some code here, but it is well annotated so shouldn’t prove too intimidating to read. Finally, for the most advanced users, the fourth chapter details getting down under the hood of the OpenStreetMap architecture and “hacking” the data, including documenting the API calls to OpenStreetMap servers to programmatically get and put information.

One of the most interesting parts of the book is actually one of the appendices, detailing country-specific quirks of the project. One of OpenStreetMap’s greatest strengths is that it is a map and spatial database of the whole world – but individual countries have different available data sources, “tagging” customs for features, and community structures, and the appendix makes for insightful reading. Being a UK contributor of a British-founded project, I find it can be easy to overlook regional differences, so it’s good to understand why some countries have mapping features in a certain way.

I found the book extremely comprehensive. I have been a long-time contributor to the project and have used the data in numerous ways, but I was surprised to learn several new details from the book, such as how to set up banned-turn indicators, and a clear discussion of the licensing.

The book is perhaps most valuable over and above other project resources because it has a consistent editing style and level of detail. The “traditional” way of learning OpenStreetMap has been through the online wiki, but this is prone to varying levels of accuracy and detail depending on the enthusiasm of the authors – so having a single-style book like this is a considerable help in fully appreciating a very diverse project. I did feel that the section on contributing is slightly longer than it needs to be – choosing just one of the editors, such as Potlatch, rather than going into multiple editors in detail, many of which duplicate functionality, would have helped shorten this section. As, I think, people are more likely to read this book to understand how to use the project’s data and resources, rather than become advanced contributors, I would imagine many readers will end up skipping over the whole section. I did also think a history of the project would have been a nice inclusion.

Books on fast changing internet projects such as OpenStreetMap are prone to go out of date quickly. With this in mind, the authors have created a special website, http://www.openstreetmap.info/ which will contain updates to the book as the project continues to evolve. As the English edition has just been published, the book itself however is bang up to date and so stands as a definitive reference.

The website also has a PDF version of one of the chapters in the second section, Mapping Practice. You can also download a copy of the country-specific appendix.

The book succeeds in simultaneously being OpenStreetMap for Dummies, OpenStreetMap: The Missing Manual and the O’Reilly OpenStreetMap book – that is to say, complete beginners, intermediate users and enthusiasts/hackers will all get something out of the book. If you are at all interested in the OpenStreetMap project, even if you don’t intend to contribute to the project but are just curious about what it is or what you can do with it, then I recommend this book. It’s as near-perfect as any book can be about one of the web’s, and the geospatial community’s, most exciting projects. More details on Amazon.

The photo of the author Steve Chilton is by Chris Fleming. Below is an example of advanced OpenStreetMap map cartography for part of Birmingham, CC-By-SA OpenStreetMap and contributors – in particular Andy Robinson.

Boris Bikes Flow Video – Now with Better Curves!

Posted on by
9

Dr Martin Austwick and I have produced an updated version of the animation of Barclays Cycle Hire bikes on a typical weekday:

Martin has once again done some programming magic to show the River Thames, Hyde Park/Kensington Gardens and Regent’s Park to add context, plus the trails for the bike “motes” are longer, allowing the road network to be picked out more easily – and the network lines remain as faint “ghosting” in the video. The bikes are also more blue! Although the bridges aren’t specifically marked, their locations quickly become obvious from the volume of bikes crossing them.

I’ve redone the routing, to fix a few problems around Trafalgar Square and a couple of other obvious places. As before, the routing is done using OpenStreetMap data and the Routino routing scripts, optimised for bike usage (i.e constant speeds on all road types, obeying one-way roads and taking advantage of marked cycleways.) I’ve tweaked the desireability of road types, so that trunk and primary roads are now only slightly less desirable than quieter routes. The traffic in most parts of central London is so slow that, based on my own observations, such roads are not such a significant deterrent to cycling. As before, I’m assuming the bikes go along the “best” route, I don’t know where they actually went. Hires that start and end at the same point – popular in Hyde Park – are shown with the motes spinning around the point.

I’ve also included road curves this time. This means bikes don’t go in straight lines between junctions. This was particularly noticeable when they cut the corner of the Thames in the last animation! Watch the bikes as they carefully curve around the kinks of West Carriage Drive in Hyde Park, around the graceful arcs of Regent Street and Aldwych and along the Victoria Embankment. (I don’t think there are many other classic curves in the central London area?)

Expand the video to full-screen, and, if your connection can take it, click the HD button to get a higher-quality with even bluer bikes!

The data for the bikes themselves is from Transport for London, with the Thames, parks and the underlying network being faithfully drawn by OpenStreetMap contributors. One of the great advantages of using OSM data – apart from it being easy to access, is it’s often very up-to-date. For example, you can see the kink at the northern end of Blackfriars Bridge, on the animation, where the road bends around the Blackfriars Station redevelopment site.