Buildduty is the friendly face of Mozilla release engineering that contributors see first. Whether you need a production machine to debug a failure, your try server push is slow, or hg is on the fritz, we’re the ones who dig in, help, and find out why. The buildduty role is almost entirely operational and interrupt-driven: we respond to requests as they come in.

We think it’s important for everyone in release engineering to rotate through the role so they can see how the various systems interact (and fail!) in production. It also allows them to forge the relationships with other teams — developers, sheriffs, release management, developer services, IT — necessary to fix failures when the occur. The challenge has been finding a suitable tenure for buildduty that allows us to make quantifiable improvements in the process.

Originally the tenure for buildduty was one week. This proved to be too short, and often conflicted with other work. Sometimes a big outage would make it impossible to tackle any other buildduty tasks in a given week. Some people were more conscientious than others about performing all of the buildduty tasks. Work tended to pile up until one of those conscientious people cycled through buildduty again. There were enough people on the team that each person might not be on buildduty more than once a quarter. One week was not long enough to become proficient at any of the buildduty tasks. We made almost no progress on process during this time, and our backlog of work grew.

In September of last year, we made buildduty a quarter-long (3 month) commitment. This made it easy to plan quarterly goals for the people involved in buildduty, but also proved hard to swallow for release engineers who were more used to doing development work than operational work. 3 months was too long, and had the potential to burn people out.

One surprising development was that even though the duration of buildduty was longer, it didn’t necessarily translate to process improvements. The volume of interrupts has been quite high over the past 6 months, so despite some standout innovations in machine health monitoring and reconfig automation, many buildduty focus areas still lack proper tooling.

Now we’re trying something different.

For the next 3 months, we’ve changed the buildduty tenure to be one month. This will allow more release engineers to rotate through the position more quickly, but hopefully still give them each enough time in the role to become proficient.

To address the tooling deficiency, we also created an adjunct role called “buildduty tools.” The buildduty person from one month will automatically rotate into the buildduty tools role for the following month. While in the buildduty tools role, you assist the front-line buildduty person as required, but primarily you write tools or fix bugs that you wish had existed when you were doing the front-line support the month before.

Hopefully this will prove to be the “Goldilocks” zone for buildduty.

Without further ado, here’s the buildduty schedule for Q2:

• April: Massimo, with Callek in buildduty tools
• May 1-22: Selena, with Massimo in buildduty tools
• May 25-June 19: Kim, with Selena in buildduty tools
• June 22-30: me. I’m not going to Whistler, so I’ll be back-stopping buildduty while everyone else is in BC.

This is also in the shared buildduty Google calendar.

Callek will be covering afternoons PT for Massimo in April because, honestly, that’s when most of the action happens anyway, and it would be irresponsible to not have coverage during that part of the day.

Massimo starts his buildduty tenure tomorrow. It’s hard but rewarding work. Please be gentle as he finds his feet.

http://coop.deadsquid.com/2015/03/the-changing-face-of-buildduty/