Happy Friday once again, releng enthusiasts!

The release engineering and operations teams are heads-down this week trying to get quarterly deliverables done *before* heading off to Whistler for a Mozilla-wide work week. There’s lots of work in-flight, although getting updates has occasionally been like pulling teeth.

Because almost everyone will be in Whistler next week, next week’s update will focus less on completed or in-progress work and more on what releng team members took away from their time together in Whistler.

tl;dr

Taskcluster: Morgan got 32-bit Linux builds working! Rail reports that funsize update generation is ready to go, pending an AWS whitelist update by IT. Ted reproduced mshal’s previous work to get OS X builds cross-compiling in one of Morgan’s existing desktop build containers.

Puppetized Windows in AWS: Jake and Rob are working on additional puppet modules for Windows. Q is running performance tests on jobs in AWS after the networking modifications mentioned last week.

Operational: MozillaBuild 2.0 is out! Mark deployed NSIS 3.0b1 to our windows build/try pools. Kim and Netops have finished up the SCL3 switch upgrades. Jake rolled out changes to enable statsd on our POSIX systems. Dustin’s talk on fwuint was accepted to LISA 15. Dustin merged all of the relengapi blueprints into a single repository and released relengapi 3.0.0.

Whistler: There’s been a bunch of planning around Whistler, and props to catlee, naveed, and davidb for getting our stuff into the sched site (the tag for releng/relops/ateam/relman is platform-ops). Be sure to take a look and pick some planning, presentation, and hacking sessions to go attend! http://juneworkweekwhistler2015.sched.org/type/platform-ops

Thank you all!

…

And here are all the details:

Taskcluster

• Ben has been doing a bunch of work adding S3 support to release automation. The two parts he focused on this week are the uploading partner repacks to S3 (bug 1173384), and generating checksums out of files in S3, and uploading them back there (bug 1174145).
• Work continues on moving the spidermonkey builds from buildbot to TaskCluster. Anhad, our intern, was able to easily get the existing script running in a novel container type. After talking with Morgan and Rail, he’s now trying to make changes to the scripts and process to allow using an existing container. One side effect of the migration to TaskCluster is that we may be able to provide build artifacts from the spidermonkey test process which is something we’ve never had before.
• The new S3 buckets for updates still need to be whitelisted, but Rail reports that funsize is otherwise ready to generate updates in TaskCluster. Here’s a sneak peek at what it will soon look like in treeherder, hopefully with less orange: https://treeherder.allizom.org/#/jobs?repo=mozilla-central&revision=a3f280b6f8d5&filter-searchStr=balrog
• After battling with the base containers and investigating workarounds, Morgan has 32-bit Linux builds working in TaskCluster.
• Ted dusted off mshal’s cross-compilation work from last year and was able to get OS X builds cross-compiling locally in one of Morgan’s existing desktop build containers. The next step is to hook it up to TaskCluster, and then begin investigating packaging and symbols which were the two outstanding issues we needed to address from our first attempts last year.

Puppetized Windows in AWS

• The networking changes we made to Windows may have potentially had a large impact on our performance in AWS, so Q is running more jobs to get new numbers to compare to our hardware performance.
• There are still a few bits of software that we have managed by the domain (AD+GPO) that we don’t yet have in puppet. Rob is working on getting nxlog support to match the remote system logging we have in the datacenter. Jake is doing the same for metric collective, the program he wrote to gather system statistics and send them to graphite.

Operational

• MozillaBuild 2.0 is out! Callek helped the sheriffs make the latest version of the Windows build environment bootstrapping tool available for developers.
• Now that the new version of MozillaBuild is out, Mark has deployed NSIS 3.0b1 on our build/try machines. This will fix several stub installer and full installer bugs (bug 989531).
• Kim and Netops have been working together to finish up the last of the SCL3 switch upgrades. For several weeks, we’ve been quietly doing rolling upgrades while keeping the trees open so there’s minimal developer impact.
• With the collectd upgrades in place, Jake rolled out changes to enable the statsd collectd module so that we we have a hook to get application metrics into graphite as well as system metrics.
• Q and Morgan have pushed out the Windows runner code to the a number of the try systems, and things are looking quite promising. Next step is to roll it out to all of the try pool next week.
• Dustin’s talk on fwunit (https://github.com/mozilla/build-fwunit), his software to perform unit tests for network flow rules, was accepted to LISA 15 (https://www.usenix.org/lisa15). He’ll be presenting at the conference on November 12th.
• In preparation for some of the security work inherent in the move to TaskCluster in Q3, Hal met with the OpSec group to review the state of releng security issues.
• Hal was also able to troubleshoot some data issues with the VCS-sync mapper, which then allowed him to start using it to map changesets for the gecko-dev repo. This is part of ongoing work to transfer ownership of the VCS-sync tools and process to the new dev-services team. Other cutovers are planned, but Hal is wisely holding off on flipping the switch until *after* the Whistler work week.
• Dustin provided a very detailed review of Jordan’s work to provide archives of mozharness on-demand (bug 1154795). The scope has increased somewhat to allow for the creation of archives for other build repos as well. Between of this work for archives on-demand, the new hg bundleclone extension we’ve been deploying, and various existing releng pro(x)xy solutions, we will have many options for reducing bandwidth consumption and network overhead going forward.
• Jordan re-landed a correctness fix that ensures we always use the newly-cloned talos repo for tests rather than a cached version on disk (bug 1112773). Thanks to Jordan for taking this work across the finish line after inheriting it from a community member who had to step away due to school commitments.
• Mach build stats are being collected in Influxdb for try builds now, thanks to Mike Shal: https://stats.taskcluster.net/grafana/#/dashboard/db/Grafana Mike notes that there are still some issues with the stats for Windows builds, but those will get ironed out in bug 1175895.
• Dustin merged all of the relengapi blueprints into a single repository and released relengapi 3.0.0. This simplifies development of relengapi by removing a number of workarounds that we needed to straddle multiple repositories. It creates a shared resource that positions the project for longer term and deeper adoption by the team.

See you next week!

http://coopcoopbware.tumblr.com/post/121936734490