OpenStreetMap

CjMalone's Diary

Recent diary entries

Not everybody cares, but we do. We care a lot.

Posted by CjMalone on 18 November 2023 in English. Last updated on 21 November 2023.

We care about having the best data. We care about having best the map. We care about having the best mapping community.

“We” is doing a lot here. Honestly, I don’t care about the render. Sure I’ve wanted some things rendered before, sure I’ve been emotional when it didn’t magically happen. But I’ve only though about rendering for a few minutes. People care a lot more then me, in retrospect, I’m sorry. I should have trusted you.

Honestly, I don’t care about the intricacies of tagging/”schemas”. Sure I’ve gotten emotional on the mailing list, and I’ve even talked about “you” (the list) behind your back. But you’ve thought about this more than me, I should have trusted you. Sorry.

I have thought a lot about the quality of our data, it’s not perfect. About the usability of our data, there are issues. About the accessibility of our data, it could easier. But actually we are doing pretty good. We are even doing better than the commercial datasets.

So trust the people that care the most, support us, encourage us, give us data and tooling, but most importantly, don’t get in our way. We’ve got this.

POI Dataset

Posted by CjMalone on 23 August 2021 in English.

I’ve been working on collecting datasets for various chains across the UK, and the globe. Yesterday I changed the repo to open, so say hello to KAFF Places. It’s mainly got retail chains, but it also some hotel chains, pharmacies and the like. You can see the list here.

This data is first party from their respective websites, interestingly enough there are a lot more quality issues that come with that than I expected. I have reached out to some of them to point out issues, and I’d love to work with all of them to improve their data.

I’m waving any database rights I may have, but because this is first party data each provider holds copyright and/or database rights. I currently do have some rules on which sites I collect data from, the most important rule is that they are publishing the data in Linked Data or Microdata. I personally believe once somebody explicitly publishes data in a machine readable format they want the data to be used. However the law may disagree. This shouldn’t be used as a source for OSM.

Some of this data is already being used by Robert Whittaker in Chain Reaction to help monitor the quality of OSM coverage. I’d also like to see this data being used as an overlay on top of OSM data to increase coverage in under mapped areas, in projects like OsmAnd and Daylight Map Distribution.

Mapping allotments

Posted by CjMalone on 25 November 2020 in English.

I’ve taken on the task of mapping allotments throughout Great Britain. I think there is a similarity between allotment users and OSM contributors, even if it’s not immediately obvious. Allotments are quite often community owned or some form of cooperative, but even the commercially owned ones have roots back to the Diggers when private land was reclaimed for public good and food production.

I’ve been thinking more about the overlap between Open Data, Free Software and the common land lately. I’m starting to think it’s all the same thing, a rebellion against private property and ownership. It’s about common good, and helping your neighbour. It’s about the future and your legacy. It’s about leaving the world slightly better than you found it.

I’ve created a MapRoulette Challange based on a subset of OS Open Greenspace. For some I can add little more than a rectangle with landuse=allotments, some I can see a few more details like paths and tracks. 1 2 3

Quarterly Project with StreetComplete

Posted by CjMalone on 18 August 2020 in English.

One of, if not the, easiest way to contribute to the current Quarterly Project is through StreetComplete. For anyone not aware StreetComplete is an editor on Android, it’s designed to be incredibly simple and help introduce new users to contributing to OSM. It’s how I started regularly contributing.

Some of it’s questions (quests) displayed to users are directly related to the current Quarterly Project like asking if a given road has a cycle path, and categorising types of bicycle parking. So if you were exclusively interested in the Quarterly Project you could disable the other quests and focus on the ones above.

However some of the other quests indirectly help OSM cycling data, such as road surface, whether the way is lit, names, one way, speed limits, crossings and pavements. They are all incredibly useful to downstream users of OSM cycling data. I think I recommend starting with all of the quests but disabling some high traffic ones, such as building types, you can, and should, come back to them, but if the amount of quests is overwhelming there is no harm hiding them until later.

Freedom of Information requests

Posted by CjMalone on 14 August 2020 in English.

I’ve started sending some Freedom of Information requests! This is a new world for me, but seems to be going good and has helped a lot.

The first one I sent was for public phone boxes after BT refused to give me a list. It was successful, and I’m now surveying the locations, as well as the ones already in OSM. The two lists differ quite a lot, and interestingly neither is accurate. It makes me wonder if BTs own list is accurate.

So I’ve been taking a lot of photos of phone boxes for Mapillary and that makes me feel like a trainspotter. I promise I’ve not go a weird thing for phone boxes, it’s just an area of OSM that needed some love. Maybe I’ve got a weird thing for OSM…

My next request was for defibs, they still haven’t responded, but I can’t imagine a reason why they wont provide them. Once they do I’ll start surveying and documenting them. I’ve been adding local defibs to OSM for over a year now so it will be interesting to see which ones I’ve missed.

I think anyone else in the UK should consider using Freedom of Information requests, WhatDoTheyKnow is a great tool and makes it super easy. It probably goes without saying but anything you receive from FOIs is not compatible with OSM, you’ll need to get another source (go out and survey).

Where's the nearest payphone?

Posted by CjMalone on 27 July 2020 in English.

I’ve recently decided to survey the payphone/phone boxes/amenity=telephone in OSM. To my surprise, they don’t exist any more, well 12 out of the 17 surveyed don’t.

It seems BT is in wave of removing them, publicly they are just “restructuring to save costs”, but in reality it seems they are just removing all phone boxes.

They don’t publish locations of payphones but if you contact them they have to give you a list of any in consultation to be removed, or any recent consultations. They’ll give you a PDF with a postcode and a vague-ish location description. You’ll then need to go out and survey them, if they’re gone, delete them, if they still exist add some metadata tags, ensure its operator tag is correct, add it’s incoming phone number.

I’m going to try and survey the 65 remaining payphones in OSM in my county, based on my last survey, it might only be 20 that actually exist! (75%)

StreetComplete may be helping in this area soon, Tobias Zwick is working on map maintenance through the app so I’ll defiantly be recommending this and bus stops to the list of verification quests.

OpenStreetMap and Linux are pretty similar in a lot of ways. The licences are both copyleft, require attribution, and allow modification and re distribution. Both projects have a foundation to look over them, but are relatively hands of on day to day operations. Both foundations have corporate sponsorship to fund them.

However when it comes to edits, the projects differ. Linux is now hugely dominated by paid developers, OSM is almost exclusively volunteers. One reason for this may be because most downstream users of OSM use it exclusively as a base map, they use there POI data on top and then almost laughably link to Google Maps for directions. They are only using a tiny portion of OSM. (And they are missing out!)

Lots of Linux contributors are paid employees of a business, usually tasked with fixing or implements a specific thing. As far as I know only Amazon does this in OSM, paying employees to fix routing issues. Sometimes this is closer to having a benefactor, where the business pays the employee to work on what ever they want and the benefactor trusts it’s in the best interest over all. Mark Shuttleworth ran Ubuntu as a loss for a long time, they contributed to the whole ecosystem during that time. I guess Mapbox might have the finances to be able to be a benefactor, but other than them there doesn’t seem to be a lot of money around.

Some developers are founded directly through other people on platforms like Patreon. I don’t think this is a massive in the kernel space, although a few high profile developers for other projects do it. You would then get the freedom to spend your time how you wanted, working on projects that interest you. This is similar to having a benefactor although you have to spend a significant amount of time and effort marketing yourself, and being your own PR. I don’t know if OSM has a big enough crowd to support this.

Some developers work as contractors just implementing or fixing things for one off payments, bug bounties are a form of this. This is less popular in the kernel than other projects, possibly because of the amount of paid employees contributing, but it still exists. This maybe the a viable option for paid edits in OSM, if a company wants to ensure that all there stores are listed and of high quality in OSM they could hire someone from the community to do so. Somebody who already knows the tools, somebody who knows the community, somebody who knows the how to integrate there data into OSM.

It’s no secret that I want to turn OSM editing into my job, employee, contractor or my own business. I just don’t know where to start.

Asda stores county wide

Posted by CjMalone on 29 April 2020 in English.

As some people have noticed last week I edit lots of Asda stores across the UK. I have been adding branch specific contact:website as well as brand data (brand=Asda brand:wikidata=Q297410 brand:wikipedia=en:Asda). The stores that didn’t already have address data, now should have data provided by FHRS Open Data.

Across the way I’ve made some mistakes, when I started I was also adding brand social media (contact:facebook=https://www.facebook.com/asda/ contact:twitter=@asda). trigpoint pointed out this is wrong because some (most?) Asda branches have there own social media accounts that would be more accurate, and user agents can still get the brand accounts via wikidata.

Throughout this project I have been prioritising adding the the specific webpage rather than a generic homepage. This has multiple benefits.

  • The most obvious being users get precise data when they click the link.
  • Another benefit is that the Asda website supports microdata, to publish metadata.
    • User agents can use this data to get more up to date or missing data than OSM provides, even though relatively few Asda stores have contact:phone that data is now available and can be used. The same goes for opening_hours, addresses and even the GeoCoordinates.
    • Mappers can use it similarly to improve OSM.
    • Data scientists can now easily compare OSM data to Asda data.

While doing this I was careful not to use data from the Asda website, due to copyright concerns. So I ignored opening_hours and contact:phone, this data is valuable and I would love for it to be in OSM but I didn’t want to compromise OSM. For addresses I used another source, FHRS Open Data, it is an amazing data set and in my opinion every OSM mapper in the UK should know about it. Using it in OSM has some awesome benefits.

  • The addresses added to nodes improve navigation.
  • fhrs:id means user agents can display the food hygiene ratings helping there users decide if they want to visit a given establishment.
  • Tracking new FHRS Open Data means we can get notified when a new place opens in an area and inform local mappers where to survey and add data to OSM.
  • The same goes for any invalid fhrs:id in OSM, it can be used to notify us that an establishment may have closed and in that case we can remove it from OSM.

Most of what I have been doing is related to tags and so “invisible”, at least to osm.org rendering. However I did add some stores/petrol stations and fix several mistakes along the way, but there is still work to be done.

ASDA Brierley Hill

Deleting a bunch of car parks.

ASDA Brynmawr

More.

ASDA Mansfield

Deleting duplicate stores. I also turned these 2 separate buildings into building:part and added a building outline.

ASDA Woking Sheerwater

Another 3D building but this time it wasn’t even a supermarket.

ASDA Bexleyheath Crook Log

Traced buildings and car parks.

ASDA Bucknall

More.

ASDA Hayes

Removing duplicate labels, in this case it was a separate node for a supermarket and a name on building.

ASDA Cheadle

I deleted a car park, moved a name to old_name, and improved tagging of a sub station. The mapper before used areas for the walkways instead of ways, so it looks a bit odd. But I think it’s valid.

ASDA Stoneycroft

This one was a hairdressers, for 8 years.

During my edits last week I only added FHRS data to stores without address data. So there are still plenty Asda stores without fhrs:id. I plan to go back over the stores and add this, at some point I should also add them to the cafes and pharmacies in stores.

I picked Asda, probably mainly because it’s at the top of the alphabet, but also because the wiki said we could use data from their website. I couldn’t find a source for that and so I didn’t use the data. However while doing this I tracked it down to domdomegg who provided me with a copy of the email that gives OSM permission to use the website data.

I now want to do a bulk programmatic edit of OSM going back over the stores that I manually linked and add opening hours and phone numbers to the stores without them. I wont do this to the amenity=fuel due to there opening hours being a little confusing, it seems to be for the little store/kiosk while some places have 24/7 card only pumps. I could also do it for alt_name to hopefully improve search.

Asda supermarket being: nwr["brand:wikidata"="Q297410"]["shop"="supermarket"]["contact:website"];

Asda Living being: nwr["brand:wikidata"="Q297410"]["name"="Asda Living"]["contact:website"];

I don’t think I’m in favour of opening_hours:covid19 but I will use it and then at some point in the future when store opening hours have changed again I will propose another mass edit where I just update opening_hours and remove opening_hours:covid19.

This does mean that every Asda Living store gets a tag of opening_hours:covid19=Mo-Su off.

Here in the UK, and probably most of the world, schools are closing, which means parents and students are looking for contact information for the schools. If any of them look for that data through OSM.org or downstream clients it is really useful to have that data up to date. So if anyone has a bit of time to kill, please help.

Robert Whittaker generates reports about schools, it’s a great tool to use. I don’t have time to do a proper write up, my work hasn’t closed, and is incredibly busy.

But vaguely,

  • Pick an area
  • Go to the table of schools matched by location
  • The first link is to edubase/Get Information About Schools.
  • Copy the contact details to OSM

It would be nice to also add the id as ref:edubase and the address, but I think the contact details should be prioritised at the moment. Other contact links like Facebook and Twitter would also be great but take longer to find as they aren’t part of Edubase.

I’ve done the PO and SO postcodes so far, I’ll be doing more as I can.

Bus stops in the UK

Posted by CjMalone on 23 January 2020 in English.

As with everything in OSM, quality varies massively by location. Around my area, and lots of the UK, the state of bus stops has been neglected. Around a decade ago there was the NaPTAN import, with a few nit picks I believe it was a good thing. OSM needed bus stops, NaPTAN had bus stops, the answer is obvious, import.

However we have let that data go stale and get out dated. A surprising amount of the bus stops, haven’t been touched in the 10 years+- since the import. They still have naptan:verified=no. They have old names. Take a look at https://www.openstreetmap.org/user/NaPTAN/history, pick a changeset, pick a node. Was it last edited about 10 years ago by NaPTAN? That’s an issue. Maybe somebody realigned it based on satellite imagery, that’s still an issue.

The NaPTAN import was a static thing. A one time event. But the NaPTAN data isn’t, they record and publish when a bus stop is changed, it’s name changed, it gets deleted, moved. Why can’t we keep on top of this data? I know OSM is very against automatic edits, for one reason or another. So I’ll just flat out ignore that, even though I disagree.

So we should be manually keeping in sync with NaPTAN, one way to do this is to download the NaPTAN data for a region and sort it by newest modified date. Compare it to the OSM data, does it have a new name? Has it location changed? Has it plain been removed due to route changes? Do we need re survey the bus stop? To download NaPTAN see that page.

Please, everyone who reads this, do an overpass query in your local area, the bus stops need your help.

Not verified:

node["highway"~"bus_stop"]["naptan:verified"="no"];

No NaPTAN ref:

nwr["highway"~"bus_stop"][!"naptan:AtcoCode"];
nwr["public_transport"="platform"]["bus"="yes"][!"naptan:AtcoCode"];

For the ones that aren’t verified, verify it. Change it to naptan:verified=yes and probably add public_transport=platform and bus=yes. There are a bunch of other tags that can help, but not required e.g. shelter=yes/no, bench=yes/no, bin=yes/no, tactile_paving=yes/no.

If there isn’t a naptan:AtcoCode tag, you can try and add it, it might be written on the bus stop board. It can be as little as 6 numbers but it seems to more often be 12 numbers, maybe with some letters at the end. If you can’t find the code there is no harm in adding survey:date=YYYY-MM-DD to point out that the node was physically verified.