A little over a year ago, I wrote about Observe — a new field mapping tool for OpenStreetMap that works offline. There was a lot of interest from the community to use Observe and we got a lot of great feedback. Ever since, we have slowly been working on adding new features and improving the overall performance.

Few weeks ago, we released a new version of Observe. I’m excited to invite you all to test this version and give us feedback. I’ll highlight some of the changes in this release.

You can download the development version of Observe for Android and iOS from here. Note that this uses the OSM Dev API. If you don’t have an account, you can create one. You can read about all Observe features and how to use them in the user manual.

Taking photos and recording traces

Two important features that we wanted to add to improve the field mapping experience of Observe is being able to capture photos and record traces. Photos are critical when it comes to mapping outdoors, as we tend to take pictures of features so we can refer to them later on to create additional tags. For example, I take pictures of the sign boards of businesses so I can enter address and opening hours when I’m back at my computer.

1. Taking geotagged photos. 2. Taking photos associated to features

Observe allows users to take photos that are geotagged, but more importantly, photos can be associated with existing OpenStreetMap features or a feature that you just created.

Recording the path a field mapper took during the mapping exercise can be really useful to compare routes from different mappers, use it for accountability, or turning those into roads or sidewalk geometries later on. While mapping, users can turn on recording and Observe will prepare a GeoJSON line string of the entire path followed.

Photos and traces captured through Observe Mobile will be available offline and uploaded to the Observe Dashboard when the user comes online.

Observe Dashboard

To manage photos and traces generated from Observe Mobile and to allow for further integrations of these auxiliary data, we built Observe Dashboard.

Photos can be downloaded and traces can be opened directly in JOSM using the remote control. We hope this integration will make it easier for mappers to introduce additional tags to features they added during the field exercise.

At the moment, any user can login to Observe Dashboard with their OSM dev account to see all photos and traces uploaded to the API. Since this is the development version, data that you send to the dashboard could be deleted anytime.

Editing and creating way geometries

When we released Observe last year, users could only add and edit point geometries. A lot of people expressed interest to have basic geometry editing functionality. We know way geometry editing can be problematic especially when modifying shared nodes or editing features that are members of relations.

Our approach has drawn inspiration from tools like GoMap!! and Vespucci, and rely on using map styles and offering the user several decision points before changes are made to feature geometries. Users can draw lines and polygons, modify existing lines and polygons, add and delete nodes, snap to existing nodes, and merge nodes.

We don’t allow users to delete nodes and ways that are part of relations, or change the preset / feature type of such features. When editing features that are members of relations, there’s a warning that communicates to the user the impact of what they are about to do.

These steps safeguards against accidental edits, but is definitely not a comprehensive approach. We look forward to hearing more ideas and feedback from you.

Future

There has been a lot of interest in using Observe outside of adding geometries to the map. One of the strong ideas we’ve been working on is adding capabilities to design surveys and campaigns. We might also consider integrating with existing tools like Tasking Manager.

Observe is still in a development mode and there are no production builds for OpenStreetMap. If your project could use this tool and are interested in collaborating, please reach out to us (sajjad@developmentseed.org). Figuring out a roadmap to get Observe on the app stores is something that we are working on right now, though, there’s no timeline for this.

Over the last few months, we’ve been building an offline-first field mapping tool for the OpenStreetMap ecosystem called Observe. Observe makes field surveying and verification easy for mappers, and works on iOS and Android.

Motivation

Field verification is an important part of keeping OSM data accurate. So far, field mapping exercises are largely manual, cumbersome, or require internet connectivity. OpenStreetMap has an active mobile editing ecosystem, but doesn’t offer the same editing experience as iD for beginners. Most often, mapping campaigns need a tool as good as iD that allows edits from the field to verify existing data and improve data quality.

Observe is a cross-platform, offline-first field mapping tool for OpenStreetMap, perhaps the first of its kind. Our primary goal was to build an application that makes field observation easy, and to provide a comparable experience to iD on Android and iOS — with some success. Observe focuses on browsing OSM data and allows users to add new points or verify existing information. The edits made offline are stored on the phone and uploaded when the mapper goes online. Observe is a product of several iterations of user research and a couple years of conceptualization.

Using Observe

Today, we’re launching development builds for both Android and iOS. You can download the Android APK from here or request for access if you’re on iPhone (you’ll have to submit a request with your email, then download and install a profile that’s sent to your email, we’ll then let you know when you can install the app). The app will be configured to use the OSM development API by default, as opposed to the production API. If you don’t have an account on the development API, create one. We want to let community members test Observe thoroughly and to give us valuable feedback before releasing a version for the production API.

Functionality

Observe allows mappers to add new points, move/delete existing points, and edit attributes of existing ways. Observe does not support complex way drawing operations or relation editing at the moment. To learn more about how to use Observe, read the user manual. Observe creates one changeset per feature. Past experience has led us to make this choice - it’s easy to revert the single defective changeset without losing many good changes. The app also supports a very rudimentary resolution of conflicts by allowing to see the changes upstream, and allowing to continue uploading.

Presets

Observe uses presets from iD. We’ve tried to include as many as possible, but there could still be certain combinations missing since this is an MVP. To keep the MVP performant, a handful of features are not available for editing. These will still be available on the base map.

Offline mapping

You can download areas of interest for editing offline. Observe will allow you to download areas of up to about 400 square kilometers, but it’s still dependent on your device, storage space, and quality of network. Edits made while offline are stored on the phone, and uploaded when the device is back online. The status of each upload can be seen in the contributions screen. When creating an offline pack, Observe downloads OSM data, base map tiles, and satellite imagery tiles for that area. When the user pans the map, while connected to the internet, these areas are automatically refreshed to fetch latest data.

Use-cases for Observe

We think Observe is excellent for the following use-cases:

• Quickly map POI features - Observe is best for adding POIs. As you walk down the street, it only takes a few seconds to add a shop, bus stop, or any point of interest to OpenStreetMap.

• Community mapping parties and field mapper exercises - Observe will work great particularly for mapping parties that identify clearly what feature to map beforehand. For instance, gathering a group of people to map trees in a neighborhood could be a really easy task with Observe.

• Verify names of streets, shops, or other features - A lot of times we want to verify and correct tags of existing features. Observe allows us to edit tags and to add new tags to ways.

Development and future

Observe is built using react-native, and Mapbox GL. There are some really cool ideas on the list, and we’re exciting to see what feedback you have. On the roadmap are:

1. Clear outstanding bugs, technical debt, and validate decisions. These are documented on GitHub.
2. Tasking Manager integration
3. Validation - both preset level and geometry level
4. Notes - add a layer of OSM notes, and potentially add notifications
5. Integrate Nominatim for search

Feedback

We’re looking for feedback. You can open a ticket in http://github.com/developmentseed/observe and let us know any bugs or issues you come across while using Observe. We’ve only tested Observe on a handful of devices, so if it doesn’t work on your phone, let us know. Any reasonably performing phone with Android 8+ should work well, as well as iOS 10.

It’s important to see what exactly happened to features in a changeset. This means identifying the state of each feature, the history, including geometry and tags that changed. The OSM changeset page doesn’t give you a clear idea of what happened in a changeset - you see a list of features that changed, and the bounding box of the changeset.

The changeset XML from OpenStreetMap only has current version of the features that changed in the changeset.

Overpass offers augmented diffs between two timestamps that contains current and previous versions of each feature that changed in that period. We put together an infrastructure that queries Overpass minutely, prepares changeset representation as a JSON, and stashes them on S3. The augmented diffs are also cached on S3. This means that the load to Overpass instance can reduce drastically while many of us are looking at the same changeset.

This is directly used in changeset-map - a utility to visualise OSM changesets.

JSON

The cached changeset JSONs are available here: https://s3.amazonaws.com/mapbox/real-changesets/production/changeset-id.json. The JSON looks like this for a changeset by user Rezhin Ali.

This is inspired by the work Development Seed did with Planet Stream. We use osm-adiff-parser to convert the augmented diff to changeset JSON.

// 20170411184718
// https://s3.amazonaws.com/mapbox/real-changesets/production/47656996.json

{
  "elements": [
    {
      "id": "4787752634",
      "lat": "36.1823442",
      "lon": "44.0158941",
      "version": "2",
      "timestamp": "2017-04-11T13:12:35Z",
      "changeset": "47656996",
      "uid": "5323129",
      "user": "Rezhin Ali",
      "old": {
        "id": "4787752634",
        "lat": "36.1823442",
        "lon": "44.0158941",
        "version": "1",
        "timestamp": "2017-04-11T08:02:21Z",
        "changeset": "47649032",
        "uid": "5323129",
        "user": "Rezhin Ali",
        "action": "modify",
        "type": "node",
        "tags": {
          "name": "ێەبد مەنان",
          "name:ar": "ێەبد مەنان",
          "shop": "car"
        }
      },
      "action": "modify",
      "type": "node",
      "tags": {
        "name": "Abd Manan",
        "name:ar": "Abd Manan",
        "shop": "car"
      }
    }
  ],
  "metadata": {
    "id": "47656996",
    "created_at": "2017-04-11T13:12:34Z",
    "open": "true",
    "user": "Rezhin Ali",
    "uid": "5323129",
    "min_lat": "36.1823442",
    "min_lon": "44.0158941",
    "max_lat": "36.1823442",
    "max_lon": "44.0158941",
    "comments_count": "0",
    "tag": [
      {
        "k": "created_by",
        "v": "MAPS.ME ios 7.2.3"
      },
      {
        "k": "comment",
        "v": "Updated a car shop"
      },
      {
        "k": "bundle_id",
        "v": "com.mapswithme.full"
      }
    ]
  }
}

Empty changesets

It’s possible that certain changesets are empty. They could have been opened, but failed to upload changes due to unreliable network, and eventually gets closed in 60 minutes. Empty changesets are not cached.

Long changesets

Changesets can also remain open for a long time. For example this one from user Manuchehr was opened 36 mins. Experienced users like to survey outdoors, and upload data in bulk. Some editors also don’t close changesets automatically. Idle changesets get closed eventually after 60 mins.

When features of changeset comes through in a later minutely diff, we update the cache on S3. This will ensure, changeset remain complete.

Database transactions and augmented diffs

A changeset being closed doesn’t mean that all features that changed have been committed to the OSM database, and appear in the minutely diff right after. Some features may take longer to commit to the database, we handle these by updating the augmented diff from S3, and then recreating the changeset JSON. You can read more about this case here.

Missing changesets

Changesets that are after March 1, 2017 are cached. We are considering doing a slow backfill, but this is entirely dependent on Overpass. If you see something missing, or unclear, please open a ticket and let us know!

I’ll be at State of the Map in Brussels this week presenting all of our work on validating OpenStreetMap data. At Mapbox, we spent the last few months looking at changes that happen in OpenStreetMap closely - geometry, tags, users and the community. I’m really excited to share what we learned, and to open the conversation on how the community can focus on keeping the map from breaking while growing. I’ll be presenting the tools we have been building, insights about problematic changes, response, mechanics of communication and more during my talk.

Our team just published an approach to validating OpenStreetMap data - talking about identifying problematic changes, inspecting them, communicating and eventually fixing. Let me know what you think!

What makes OpenStreetMap special is the community. The community is what makes OpenStreetMap a truly self-healing map. The community is the map.

Map of recently reverted changesets

If you haven’t already, watch Sanjay Bhangar’s presentation at State of the Map US.

Looking forward!

I’m a software developer at Mapbox based in Bangalore, India, co-leading our data team. Prior to Mapbox, I’ve been part of projects at Karnataka Learning Partnership, Moabi, and organizations like the Center for Internet and Society and Tactical Technology Collective. What makes writing this note special is that I’m in Doha this week, talking about OpenStreetMap data, tools and running a Digital Humanitarian workshop along with Heather Leson and the Qatar Red Crescent Society. In doing so, I’ve been able to engage entrepreneurs, engineers and scientist - introducing them to the largest living map, and OpenStreetMap as a project that is truly “moving the map.” What makes me believe in what we do at HOT and OpenStreetMap is the power of bringing people together.

I have been part of OpenStreetMap since 2008. Since the Haiti activation, I regularly lurked on the HOT mailing list and occasionally mapped. More than mapping, I was drawn to how the OpenStreetMap software infrastructure worked and the idea of mapping anything and everything. I started off my career building data infrastructures and advocated the use of OpenStreetMap software for other use cases. The recent Nepal activation is when I truly got involved in HOT. Right after the earthquake, our team in Bangalore got together to start mapping priority areas. I was communicating with folks at the Kathmandu Living Labs and relaying requests to map, make map data available for download, and working with actors on the ground to design maps for print. I also got a chance to help and learn about imagery acquisition through Mapbox and the Indian Space Research Organisation.

India doesn’t have a strong OpenStreetMap community. We managed to build momentum and groups across different cities - New Delhi, Pondicherry, Ahmedabad, Hyderabad and Mumbai to contribute to the Nepal mapping tasks. Arun and I did training over Skype, phone and hangouts. It was a surreal experience to have been in the middle of a crisis at the same time I realise from the conversations with everyone at KLL that it was very real. I’ve been part of the most recent activations in Mexico and Afghanistan, rallying communities to map and getting in touch with people on the ground who could tell us more about what’s important. I have run events, trained individuals to map and spoken to journalists and the media, in India, about what we do at HOT.

I think HOT is in an excellent position to do something that will change how crisis response works, from data acquisition and communication to controlling and supporting distribution of aid. Our community is strong, technology is robust and we are continuously improving, as we learn from each activation. In my mind, HOT’s challenges are more than drumming up people to map. In times of crisis, people want to help. What’s missing is the link between traditional disaster response agencies and HOT. During the Nepal and Afghanistan activations, I have reached out to local aid agencies in India like the Red Cross chapters, Goonj, Indian Airforce - they were all responding to Nepal - but I failed to make any connection and failed to see our maps being used by them. All the journalists I spoke to were interested in the story how a group of digital humanitarians are helping map Nepal and were not going to help speak about how these maps need to be used by local agencies. In Afghanistan, I heard firsthand that the agents on mission have practically no communication channels. HOT has work to do to break these silos, work with partners to and make our maps known and used. I’d like to start small, understand how local aid agencies work and make our ways flexible to collaborate closely.