Highlights:

• Rail spent some time optimizing spot instance bidding, which helps lower our AWS bill.
• Massimo improved one of our major AWS usage reports, which will also help us be more efficient with our usage.
• Ahal got us uploading qemu.logs to blobber, making it easier to debug some types of failures.
• Mike got his patch for “hg purge” landed, which has improved try build times!
• Callek moved all of our in-house test machines onto in-house masters, which takes additional load off of our AWS VPN link (which reduces the chance of tree closures due to it failing).
• Dan continued his work on moving MozBase tests out of the build jobs, which improves build and test turnaround time.
• John started figuring out how to make our signing servers support 64-bit Windows binaries, which is one thing (out of many) that we need to do before shipping 64-bit Windows builds.
• Catlee continued work on Jacuzzi implementation, making good progress on some of the Buildbot implementation that we need.
• Nick made additional progress on fixing our release automation to support shipping Release builds to Beta users for extra testing before we ship for real.

Completed work (resolution is ‘FIXED’):

• Buildduty
• mac-signing4's nagios is having issues
• Report impaired AWS non slave machines
• upload a new talos.zip to capture recent talos changes
• Add swap to linux build machines with <12GB RAM
• builds-4hr.js.gz not updating, all trees closed
• Please upload mozfile-1.1 to http://pypi.pub.build.mozilla.org/pub
• some test machines unable to connect to their masters
• All trees closed due to timeouts as usw2 slaves download or clone
• General Automation
• Need helix-eng builds for test automation
• Rooting analysis mozconfig should be in the tree
• Periodic PGO and non-unified builds shouldn’t be running again on pushes that already have them
• Implement ghetto “gaia-try” by allowing test jobs to operate on arbitrary gaia.json
• Make Pine use mozharness production & limit the jobs run per push
• Save qemu.log to blobber
• decommission final set of KVM buildbot masters back to IT
• Compression for blobber
• Temporarily revert the change to m3.medium AWS instances to see if they are behind the recent increase in test timeouts
• Do our own hg purge
• Lots of Command Queue nagios alerts from new AWS masters
• Loan Requests
• Please loan t-xp32-ix- instance to dminor
• Loan glandium a m3.medium test slave
• Loan jgilbert a Linux M1/R test slave
• Need a slave for bug 818968
• Loan Win8 test slave to Jim Mathies for soft keyboard investigation work
• Slave loan request for a tst-linux64-ec2 vm
• Other
• Grant pkewisch access to dev-master1
• Platform Support
• Intermittent Android x86 We have not been able to establish a telnet connection with the emulator
• Release Automation
• release builds not using space requirement of platform config
• Releases
• tracking bug for build and release of Firefox 27.0.1
• Releases: Custom Builds
• Clean up MSN bundles and restore MSN add-on to the correct URL
• Repos and Hooks
• Determine why github mozilla/mozilla-central has been removed from service by github
• Tools
• Self-serve nightly wildcard matching is not strict enough (can’t retrigger UX nightlies)
• Create a staging environment for buildapi

In progress work (unresolved and not assigned to nobody):

• Balrog: Backend
• API for manipulating balrog rules
• Buildduty
• Setup tegras that are returning from loan
• Deploy python 2.7.3 to all build machines
• rewrite watch pending to cope better with spot requests that aren’t being fulfilled
• General Automation
• mozilla-central should contain a pointer to the revision of all external B2G repos, not just gaia
• Don’t reset mock environments if we don’t have to
• Self-serve should be able to request arbitrary builds on a push (not just retriggers or complete sets of dep/PGO/Nightly builds)
• Schedule multimedia b2g mochitests for emulator-jb on Cedar
• Provision enough in-house master capacity
• Provide B2G Emulator builds for Darwin x86
• Do debug B2G desktop builds
• keep buildbot master twistd logs longer
• [tracking bug] Run desktop unittests on Ubuntu
• Add support for add-if-not instruction added by bug 759469 to the mar generation scripts
• Configure holly like Aurora properly
• [Meta] Some “Android 4.0 debug” tests fail
• b2g build improvements
• Run mozbase unit tests from test package
• fx desktop builds in mozharness
• Move Firefox Desktop repacks to use mozharness
• add –dump-config and –dump-config-hierarchy to mozharness
• port MockReset to mozharness
• allow for multiple basedirs to be passed to PurgeMixin.purge_builds()
• allow mozharness’s tbox_print_summary() to be used outside of unittests
• Use spot instances for regular builds
• Create SpiderMonkey builds for Windows on TBPL again
• Do nightly builds with profiling disabled
• Windows ix test machines don’t always start buildbot
• Don’t clobber the source checkout
• Add Linux32 debug SpiderMonkey ARM simulator build
• aws sanity check shouldn’t report instances that are actively doing work as long running
• Allow the possibility of spidermonkey builds on more platforms
• [tarako][build]create “tarako” build
• move off of dev-master01
• Schedule JB emulator builds and tests on cedar
• [tracking bug] migrate hosts off of KVM
• Run jit-tests from test package
• Intermittent Linux spot builder “command timed out: 2700 seconds without output, attempting to kill” while trying to install mock
• Loan Requests
• Loan felipe an AWS unit test machine
• need a loaner bld-linux64-ec2 instance
• Slave loan request for Matt Woodrow
• Loan an ami-6a395a5a instance to Aaron Klotz
• Please lend Andreas Tolfsen a linux64 EC2 m3.medium slave
• Please loan dminor instance to build Android 2.3 Emulator
• Loan :sfink a linux64 bld-centos6-hp-005
• Slave loan request for a tst-linux64-ec2 machine
• Other
• [tracker] Machine move SCL1 -> SCL3
• [tracking] infrastructure improvements
• s/m1.large/m3.medium/
• Platform Support
• Migrate ESR branches to win64-rev2
• Setup in-house buildbot masters for remaining in-house testers
• signing win64 builds is busted
• Windows slaves often get permission denied errors while rm’ing files
• Deploy ndk-stack on foopies
• do initial planning on “chunks” of machines to move.
• slave pre-flight tasks
• Deploy patched version of Mesa 8.0.4
• Release Automation
• Support modifying update-settings.ini when doing update verify
• Create SSL products in bouncer as part of release automation
• Figure out how to offer release build to beta users
• cache MAR + installer downloads in update verify
• Releases: Custom Builds
• Update Rambler pack with updated extension 4.9.3
• Repos and Hooks
• Request for a /gaia-l10n/lt repository
• disable & delete mirror of gaia-ui-tests
• Tools
• [Tracking bug] – Assisted/Auto Landing from Bugzilla to tip of $branch
• Store the timing of actions in slaveapi
• Move trychooser hg extension into the new https://hg.mozilla.org/hgcustom/version-control-tools repo
• Use MozillaPulse from pypi
• cut over build/* repos to the new vcs-sync system
• slaveapi still files IT bugs for some slaves that aren’t actually down
• Replace dev-master01 for dev-master1 in our repos

http://hearsum.ca/blog/this-week-in-mozilla-releng-february-21th-2014/