OpenStreetMap

JesseFTW's Diary

Recent diary entries

Update!

As of July 25 – joto did the manual override! So the problem is resolved, at least for now. We still need to deal with the “but MY bay isn’t really a coastline” recurring problems, but now we can deal with them individually, rather than trying to address a six month backlog.


So, I accidentally wrote a wall of text yesterday in the IRC channel, and rory asked me to copy it here.

It was prompted by this question:

15:28 what is the rio plata thing about?

I responded:

johan_: So, the coastline rendering process used for openstreetmap.org is complex. and involves multiple semi-independent groups.

One group produces the planet.osm file, containing all the OSM data.

A different group (well, it seems to mainly just be joto), runs an independent server at https://osmdata.openstreetmap.de that processes the planet.osm data into a set of shapefiles defining the world coastlines.

Then a third group (back within osm.org) uses that data to render the map displayed by default on openstreetmap.org, known as Carto.

The osmdata script that does the processing includes an automatic cut-off that marks the produced shapefiles as invalid if the maganitude of changes since the last valid results is beyond a certain amount specifically, 0.0000015 (I’m not sure in what units, exactly). https://github.com/fossgis/osmdata/blob/master/scripts/coastline/compare-coastline-polygons.sh#L13

And the third group independently choses to obey this marking, and only apply shapefiles that have been blessed by this script.

There is an option to manually override the cutoff (by simply deleting the data it is comparing to), but joto, reasonably enough, will only do that with a clear, well-documented demonstration of a consensus among mappers that such a larger change is needed.

This mostly works.

Recently, specifically on January 10, 2020 – it didn’t.

A mapper noticed a large error in the coastline, and, reasonably enough, fixed it, all at once. This was too large for the automatic cutoff, so it halted updates. This would still have been OK if the large error had been totally uncontroversial and obvious – the mapper who fixed it could have reached out to joto, showed him the obvious correction, and he would have overriden the cutoff, and things would have continued to be fine. But, sadly, the error wasn’t so obvious and uncontroversial. It was, in fact, a subtle local political decision, that isn’t at all obvious or self-evident to a busy, uninvolved gis developer from far away. And the mapper who corrected it also didn’t realize they need to reach out to joto and get a manual override. So, instead, things just sat there. Blocked.

Over time, additional (small) corrections/updates to the coastline were made, by other mappers – some of whom noticed that their work didn’t seem to show up on openstreetmap.org, while lots of others didn’t even think to check. Eventually, someone tried reverting the large correction, in the hope that doing so would unlock the blockage and enable the other, uncontroversial updates to go thru. But it was too late – there were so many other tiny updates, all across the world, that their combined size alone hit the automatic cutoff. That’s where things stand today, to our collective embarassment.


There are two paths forward, that I see:

1) Revert enough of the changes since Jan 9 to bring us back under the cutoff, then slowly re-apply them, making sure to stay below the limit of changes per day.

2) Appeal to joto to do a manual override (with or without the large correction).

johan_: I would LOVE for you to try and work on whichever of those appeals to you more! What, you may ask, does any of this have to do with the Rio de la Plata?

Basically, nothing.

But the connection is that Plata is the name of the area where the large correction applied. That’s it.

Hope this wall of text was either interesting, or at least, not too frustrating to scroll past.