Major highlights:

• Aki is working on changing how we chunk our unittest builders. Because of some limits in the way we currently scale with Buildbot, this has became extremely important.
• Kim enabled a bunch of new Firefox for Android tests on emulators. These will let us move some on device tests into more scalable and reliable solutions.
• Andrew moved a ton of Mozharness test configs in-tree. Having them there reduces a lot of pain by giving us greater flexibility in branch specific configuration, and making it easier for test changes to follow code merges.
• Catlee changed our job scheduling in Buildbot to coalesce a maximum of 3 jobs together. This should help reduce the bisection windows when bad pushes are made. He wrote a great blog post detailing it here.
• Anhad (our first intern of the year), started detailing a plan to build a partial MAR generation service that we’ve wanted for a long time. When completed, it will allow us to serve many more users more efficient updates.

Completed work (resolution is ‘FIXED’):

• Buildduty
• No Tarako Build Today on 5/15/2014
• Please block/disable/stop updates to today’s Thunderbird nightly builds
• running out of disk space during linux64_gecko-debug jobs
• upload https://github.com/vitillo/python_cache_flusher to http://pypi.pub.build.mozilla.org/pub
• 10.5 hour wait time for mtnlion try test builds due to around 50 mtnlion slaves breaking last night
• Respin B2G Trunk Hamachi Builds Due to Critical Regressions from Bug 906164
• A lot of XP machines out of action
• General Automation
• The b2g desktop /latest/ directory contains way out of date builds
• Self-serve should be able to request arbitrary builds on a push (not just retriggers or complete sets of dep/PGO/Nightly builds)
• nuke lockfiles in .repo dir in b2g_build.py
• upload mars for other b2g device builds that need updates
• upload flame gecko/gaia mars to public ftp
• make it possible to post b2g mar info to balrog
• Mozharness test configs should live in-tree
• Jetpack jobs failing with “IOError: [Errno 13] Permission denied: ‘/etc/instance_metadata.json’”
• Don’t do Thunderbird builds on comm-* branches for non-Thunderbird pushes
• Stop running tests on 10.8 on B2g28-v1.3 and B2g26-v1.2
• Need Tarako 1.3t FOTA updates for testing purposes
• Move Android 2.3 reftests to ix slaves (ash only)
• Changes introduced for Bug 970918 break mozharness code when mock_target is not defined
• Reduce log retention on buildbot masters from 200 twistd.log files to 100 twistd.log files
• We need to pull in merge commit refs for gaia-try runs
• Limit coalescing
• Get e10s tests running on inbound, central, and try
• Make it possible to run gaia try jobs *without* doing a build
• inari eng nightlies failing because stage-update.py can’t find a mar
• Don’t enable sccache on PGO builds
• Disable branches to reduce builder count
• blobber TinderboxPrints could be tidier
• Run more Android 4.0 Debug tests on Cedar
• Schedule Android 2.3 crashtests, js-reftests, plain reftests, and m-gl on all trunk trees and make them ride the trains
• add b2gbld password to BuildSlaves.py
• Add MySQL specific database schema dumps to the BuildAPI repo
• Configure Elm for specific builds
• Loan Requests
• Slave loan request for a VS2013 build machine
• Slave loan request for a t-w732-ix machine
• Loan glandium a (the) linux build slave in scl3
• Other
• Switch in house try builds from ceph to reverse-proxied S3
• S3 bucket for Servo
• Platform Support
• mh and device changes for relocated p3 panda racks
• Create a Windows-native alternate to msys rm.exe to avoid common problems deleting files
• Release Automation
• Release tagging should use purge_builds.py
• release jobs don’t pay attention to jacuzzi allocations
• release repacks needs to submit data to balrog
• Repos and Hooks
• Need project branch for building the Mulet
• Fix documentation for hgrc [extension] paths for Windows
• Tools
• Create a ‘reftest’ alias for ‘reftest-1' and ‘reftest-2' linux-android unittest suites

In progress work (unresolved and not assigned to nobody):

• Buildduty
• Reconfigs should be automatic, and scheduled via a cron job
• General Automation
• Run Android 2.3 tests against armv6 builds, on Ash only
• make mozharness test scripts easier to run standalone
• kill inari, leo builds
• Provide B2G Emulator builds for Darwin x86
• Figure out the correct path setup and mozconfigs for automation-driven MSVC2013 builds
• set-up initial balrog rules for b2g updates
• Add a MOZ_AUTOMATION environment to all builds
• Manage repo checkout directly from b2g_build.py
• Triggering arbitrary jobs gets branch wrong
• b2g balrog submission should point at dated dirs, not latest-*
• Create 4 more linux64 tests masters
• hamachi device builds submitting bad urls to balrog
• [Meta] Some “Android 4.0 debug” tests fail
• /builds/slave is read only for cltbld
• allow mozharness scripts to call other mozharness scripts
• [Flame] Please update the blobs to use blobs from 10F-3
• Add the build step or else process name to buildbot’s generic command timed out failure strings
• switch b2g builds to use aus4.mozilla.org as their update server
• Run Gaia unit oop and reftest sanity oop for b2g desktop on trunk
• Stop running cppunit and jittest on OS X 10.8 on all branches
• eng+noneng nightlies upload MARs (and maybe other files) that stomp on each other
• Please add non-unified builds to mozilla-central
• Do debug B2G desktop builds
• change how we chunk in desktop_unittest.py
• include device in fota mar filenames
• Add ‘hsb’ to the Firefox build
• Upload the list of all functions from hazard analysis
• Start doing mulet builds
• Don’t require puppet or DNS to launch new instances
• Put ccache on SSDs
• Add support for webapprt-test-chrome test jobs & enable them per push on Cedar
• Loan Requests
• Need a bld-lion-r5 to test build times with SSD
• Other
• Deprecate tinderbox-builds/old directories for desktop & mobile
• [tracking] infrastructure improvements
• Switch in house try builds to S3 for sccache
• Platform Support
• Cleanup temporary files on boot
• evaluate mac cloud options
• cancelled 2.3 mochitest jobs put ix slaves into weird state
• signing win64 builds is busted
• Windows slaves often get permission denied errors while rm’ing files
• slave pre-flight tasks
• Run unittests/talos on OS X 10.9 Mavericks
• Update version of pip installed on automation machines from 0.8.2 to 1.5.4+
• Release Automation
• release automation can’t update balrog blobs during the update step
• Figure out how to offer release build to beta users
• Releases
• tracking bug for build and release of Firefox and Fennec 30.0
• Trim rsync modules (May 2014 OMG we still have to do this edition)
• Repos and Hooks
• Create post-push hook to print cset url for QA repositories on hg.mozilla.org
• Tools
• b2g tagging script
• AWS Sanity Check lies about how long an instance was shut down for…
• implement “disable” action in slaveapi
• tegra/panda health checks (verify.py) should not swallow exceptions
• Setup docker apps for buildbot, buildapi and redis
• Blobber upload files not served with correct content type

http://hearsum.ca/blog/this-week-in-mozilla-releng-may-16th-2014/