“Reclama” Experiment report

Intro

FOSDEM 2015 is a premier open source developers’ event in Europe. Mozilla is heavily participating in the event for over 10 years now, with booths and dev-rooms. This year the Community Development team decided to run an experimental approach on recruiting new contributors by promoting good-first bugs.

Scope

Given the highly technical nature of the audience in FOSDEM 2015, the approach decided has a straight-forward promotion of bugs and prizes for them, for people to track and get involved.

Following sign-off meeting with the stakeholders (William Quiviger, Francisco Piccolini and Brian King) the specifications agreed for the first iteration of the experiment were as follows:

1. A straight-forward interface displaying hand-selected good first bugs, ready to be worked on
2. Next to bugs, prizes should be displayed.
3. Viewing from mobile devices should be also accounted for.
4. Public access to all content. Ability to edit and add events on selected accounts (hardcoded for v1)
5. The interface has to be properly Mozilla branded (high visibility and promotion in an event)
6. It needs to be future-proof for event to come (easily add new events and/or bugs after deployment)
7. Solution URL should be short and easily memorable and digestible.
8. It needs to be delivered before the start of FOSDEM 2015
9. A report should be compiled with usage statistics and impact analysis for the experiment

Development

Given the extremely short timeframe (less than 3 days), ready, quick or off-the-shelf solutions were evaluated like:

• An online spreadsheet (gdocs)
  • Not meeting requirements #3, #5, #6, #7
• A bugzilla query
  • Not meeting requirements #2, #7, #6
• A scrappy HTML plain coded solution
  • Not meeting requirements #4, #6

Thus, given the expertise of the group it was decided to create a django application to meet all requirements in time.

Following 2 days of non-stop development (fantastic work from Nikos and Nemo!), testing and iteration we met all requirements by developing the app which is codenamed “reclama” (Italian -shortof-  for claim).

Code can be found here: https://github.com/mozilla/reclama

Deployment

In order to meet requirement #7 (short and memorable URL) and given the timeframe, we decided to acquire a URL quickly ( mozbugsprints.org ) and deploy the application.

For usage statistics, awstats was deployed on top of the app to track incoming traffic.

Usage statistics

During the weekend of FOSDEM 2015, 500 people with almost 5000 hits visited the website. That’s almost 10% of the event participants.

Saturday 31-Jan-2015 traffic analysis

Booths and promotion of the experiment started at 9:00 as expected. With mid-day (noon) pick which is consistent with increased traffic in booths area of the event.

Traffic continues to flow in steadily even after the end of the event, which indicates that people keep the URL and interact with our experiment substantial time after the face to face interaction with our booth. Browsing continues through the night, and help might be needed (on call people/mentors ?) during that too.

Sunday 1-Feb-2015 traffic analysis

Second day in FOSDEM 2015 included a Mozilla dedicated dev-room. The assumption that promotion through the dev-room would increase the traffic in our experiment proved to be false, as the traffic continued on the same levels for day 2.
As expected there was a sharp cut-off after 16:00 (when the final keynote starts) and also people seem to not come back after the event. Thus the importance of hyper-specific event-focused (and branded) challenge seems to be high, as people relate to that understanding the one-off nature of it.

Impact

32 coding bugs from different product areas were presented to people. 9 of them were edited (assigned, commented, worked on) during or immediately (one day) after FOSDEM 2015. Out of those 9 , 4 ended up on first patch submission (new contributors) and 3 received no response (blocked contributor) from mozilla staff (or core contributors).

Recommendations

• Re-do experiment in a different cultural environment, but still with related audience, so we can cross-compare.
• Continue the experiment by implementing new functionality as suggested by the stakeholders (notes, descriptions etc).
• Experiment random sorting of bugs, as current order seemed to affect what has been worked on.
• Submission of bugs to be featured on an event should be coordinated by event owner and related to event theme and topics.
• A/B test how prize allocation affects what bugs are worked on.
• Expand promotional opportunities throughout the event. (booth ideas?)
• On call developers on promoted bugs would eliminate the non-answered section of our bugs

PS. Special thanks to Elio for crafting an excellent visual identity and logo once again!

http://pierros.papadeas.gr/?p=422