Thank you Iyan, the pleasure is all mine, it’s such a privilege to have you on the team!

Thanks all for the kind words! Many apologies for the delays, as you may have guessed I’m currently swamped with misc commitments. I’m collecting all feedback in preparation for a further analysis pass sometime in March, so all your suggestions are much appreciated. I’m particularly interested in your suggestions about how to safeguard the analysis: for an observational study like this, it’s important to measure carefully, and to take into account any confounding factors that may introduce a systemic bias in outcomes.

Marion Barry: I’m including both automated messages (required when rejecting a task) and manual @mentions, but only consider messages that have been sent by a validating user at the time of validation. I ‘m not currently considering whether people actually read the message, but agree that it would be a good addition; it can increase our confidence in the causal link.

Tallguy – I agree, this seems to be a good summary: “there needs to be a message”.

BushmanK: yes, this is about HOT contributors only, I do not observe OSM activity outside of HOT. I edited the intro to make it clear that I’m talking about HOT. It’s feasible that the OSM community might operate quite differently; it’s also likely a much more complex space than HOT.

tekim – correct, I don’t go into detail in the short writeup above, but I see that many mappers don’t return after their first contribution, and likely never see any messages. As you suggest, this introduces a potential for confounding factors, which I try to address by introducing a set of carefully selected control variables that can serve as broad proxies for a propensity to remain engaged. Please let me know if you have ideas for other factors to control for. However, this is still only an observational study, and there is a risk that we identify spurious causes. We would have to run experiments to confirm these relationships.

mapeadora – interesting, thanks for these examples! I appreciate when people share such stories, because currently I don’t think many people have reflected very deeply about the contributor experience from this side. Everyone’s experiences will be different, but the sharing of personal experiences helps a lot to make our understanding of the process more concrete.

tekim – I’ve seen some validators use OSM direct messages in addition to TM messages; unfortunately I don’t have access to OSM DMs, so can’t include them in the analysis.

Jorieke – intriguing thought! There definitely are different “styles” of validation, but so far I’ve not compared their impact; in part because we might not yet have enough observational data available for such a fine-grained analysis. Maybe in another 1-2 years… :)

Thanks Tyler! I’m looking forward to the next TM as well :) The design discussions around it are definitely moving in the right direction!

Thanks Nate! Yes I thought I should start sharing these before the TM3 team starts committing to specific goals…

tyr_asd: ah, a very good point. I added a paragraph to the end of the labelling section to explain the statistical method. Please let me know if it’s still unclear; I guess I initially left it out because it’s quite hard to avoid using overly technical language when explaining it. (Maybe someone else has a better way of expressing it?)

RebeccaF: Thank you! Yes, I don’t go into the details here, but I found that the effect of feedback diminishes with each additional day of delay; this is independent of the kind of feedback that is given. Luckily, these days most feedback arrives quickly: as I mention above, 50% of all feedback was given within 28 hours. I expect that with our maturing validator community, this will get even quicker over time. Please let me know if that didn’t clarify it!

mapeadora, thanks for your comment! I definitely agree – while large-scale studies can reveal some general tendencies, in the end these are very personal matters, and everyone will have their own curiosities and sensitivities, and respond to feedback in their own way. I also find it significant that you’re not the first person to mention the question of gender in this setting to me; there is a universe of social and contextual considerations worth exploring. However, the OSM edit history provides no information about the demographics of mappers… I hope other researchers will take on such questions, possibly using ethnographic methods.

Amazing, Selene! I’m very happy to hear about your initiative. And also Jennings – thanks for your kind words!

Hallo Super-Map! Just briefly, in case I’ve given the wrong impression: plenty of the newcomers joining HOT turn into highly engaged mappers. The top 5% of mappers contribute for 18 hours of more over their lifetime (slide 10) – that’s 1,600 people! The core community may not grow as quickly as the overall contributor number, but it’s certainly growing. (My research is explicitly focused on the newcomer experience, so I don’t spend a lot of time exploring what happens to the many who decide to stick around.)

Oh, these are great questions. To evaluate this we would need a list of attendees of such events, or at least an approximation. Previously I’ve estimated mapathon attendees based on known event dates/times and the mapped project – are there such records for GeoWeek?

And of course we’d have to wait at least a couple of months to then observe retention.

Have there been similar large events we could look at in the past? E.g. are there attendee records of last year’s 100 mapathons?

(I’m currently looking at validation impact on engagement, and then plan to start my PhD thesis write-up; but this looks like a great evaluation opportunity for if I can find some time on the side, provided we can find a good attendance estimate.)

Hey bdiscoe, that must have been a momentary reduction in project activity; I’m applying some simple heuristics to identify the key activity period, and they probably were just below my minimum activity threshold. Iirc I’m only tracking activity for periods where at least 5 people a day contribute.

In an updated chart I made last week, MapLesotho is again shown as an active project. Will update the chart here when I get some spare time. (I’ve since also changed some of the project classifications, so an update is due anyway.)

Max, have a look at the draft spec I linked to – mapping experience is indeed merely a first measure because it is easy to compute. As you suggest, there’s loads more that could be done that more directly relates to map data quality.

Shame I missed this by less than 24h :) Hope you’re doing more, looks like a fun event!

az09 – hah I’d lost track of OSM node IDs, but I’m not surprised we have over 3 billion now!

Thanks BushmanK! Will give that a go for the next iteration.

aHaSaN – I used QGIS for this. (At some point I’ll start doing web-based visualisations too. I’ve never produced OSM tiles, no idea what that would entail or whether it’d be worth the effort.)

(Click on the image for a high-res version.)

Woop woop! :)

Oooh, good one! I should also have mentioned universities, there are quite a few student groups organising mapathons already.

Free food does probably help as a motivator :) At our evening events, many mapathon attendees come straight from work.

Hah indeed, that likely does prevent conversations from ever going very far.

Haha nice suggestion. Unfortunately I won’t have time to explore it – have a busy few weeks ahead with misc deadlines. But you’re right, there’s some further observational data we could make use of.

GOwin – it’s indeed interesting to ponder the progression over time! I also expect it depends a lot on the specific setting.

I started looking at this for London events recently. At our monthly mapathons we now make an effort to promote JOSM use to people who are already comfortable using iD. This came from a realisation a few months ago that we need to build more expert capacity in our community; and it already has made an impact both anecdotally and empirically.

At our most recent mapathon, around 40 people were in the iD section; by far most of them were first-time attendees. Around 20 were at JOSM tables, these mostly had been at previous mapathons.

Empirically I found that the more often people return to our mapathons, the more likely they are to eventually learn JOSM. It’s unfortunately too early to put specific numbers on this – I need to take some time to develop this analysis further, and decide how best to capture the effect. It’s also a scenario where we don’t yet have too much observational data (because the JOSM group is still comparatively small), so I might let things develop for a few more months before I come back to this.

As a preview, here’s a visualisation that shows editor use per mapathon attendance for a subset of attendees (people with absolutely no prior OSM experience who attended at least twice):

The attendance history for every first-time attendee is visualised from left to right. The left-most column is the first mapathon someone attends, every subsequent block towards the right is an attendance at a subsequent mapathon. Blocks are shaded red when people use JOSM during their session.

This shows that most people do start out with iD, but most long-time attendees eventually switch.

Again, this is mostly anecdotal, and directly influenced by how the London mapathons are organised.

LivingWithDragons – it was personal correspondence, Joost had contacted me by DM.

Vincent – I would love a random assignment study! It requires some discipline to set up correctly though, plus it introduces artificial side-effects; e.g. people may get separated from their friends. It’s hard to do social studies well.

Pete and Harry – some very insightful comments, thank you both!

Warin61 – unfortunately I have no idea how to calculate a coverage factor for survival curves! I’m using the lifelines Python library which doesn’t have that feature. But pointers at how to compute it will be appreciated. (Note that the data is not normal distributed, which will likely complicate things.)

RobJN – I’ve heard HOT people discuss custom iD interfaces many times, I expect there would be similar interest in a HOT-specific JOSM distribution. And I’m sure there are plenty of customisations already in the wild. Let’s see what the Antwerp group will produce!

Richard – very true, nice examples.

Ralph also emailed a longer reply with some training experiences in London, I’ll ask him if he wants to post it here.