Major Highlights:

• John Zeller made it possible to trigger builds/tests on arbitrary revisions. This is a feature with HUGE impact — it will allow us to bisect problems much more easily, and possibly even do so automatically.
• Kim is happy to be back from parental leave and has been preparing to spread the word of Mozilla RelEng to EclipseCon next week and at Google’s RelEng Workshop in April.
• John Hopkins has been continuing his work on helping get tests on 64-bit Windows EC2 machines green, which will help us scale to the cloud for those tests and gets us closer to shipping a 64-bit Windows binary.
• Armen enabled mozilla-inbound desktop browser-chrome and b2g reftests on EC2 test machines. While they’re still hidden for now, it’s gotten us much closer to making sure we keep turnaround times great for those tests.
• Catlee figured out that l10n nightlies were a big part of the reason we’ve had tree closures due to pending jobs, and we’ve moved those builders to jacuzzis to fix that.
• Taras and Rail wrote a better algorithm for bidding on EC2 spot instances. Because of that we’re now running more jobs than ever on spots, and well under 1% of them are killed off because of a higher bid.

Completed work (resolution is ‘FIXED’):

• Balrog: Backend
• /rules/:id endpoint needs to return data_version/csrf token
• exception in getLocaleOrTopLevelParam since latest balrog push to prod
• Balrog: Frontend
• need to be able to delete rules from admin ui
• balrog revisions pages never load sometimes
• fix balrog admin ui title not to read “AUS3''
• Buildduty
• Heavy Tegra Backlog on all trees
• servo linux slaves couldn’t successfully run puppet
• Reimage remaining win64-rev1 machines as win64-rev2
• nightly builds failing because of switch to ftp-ssl
• Trigger b2g Hamachi nightly build
• General Automation
• remove old, now unused balrog credentials from BuildSlaves.py
• Make B2G device builds periodic
• Schedule multimedia b2g mochitests for emulator-jb on Cedar
• b2g_bumper can add extra comments
• add jacuzzis for linux l10n builders and more b2g-inbound builders
• gdb not installed in idle-mock-hw
• Schedule mochitest-browser-chrome on EC2 slaves on all branches
• Compression for blobber
• mozharness should use –manifest instead of –test-manifest for emulator-jb mochitests
• emulator-kk builds
• Add support for add-if-not instruction added by bug 759469 to the mar generation scripts
• Make TBPL gaia-integration download node modules from git mirror
• Remove the version 1 manifest from the mar creation scripts
• Loan Requests
• Request for tst-linux64-ec2-XXX machine to run b2g-emulator mochitests
• Please loan dminor Android 2.3 Emulator test instance
• loan ec2 linux32 and linxu64 slaves to jmaher
• Slave loan request for a t-xp32-ix machine
• Loan of linux64 test machine to Steve Fink for M-oth
• Need a slave for bug 818968
• Other
• Recent builds missing from http://inbound-archive.pub.build.mozilla.org
• Platform Support
• The clock of some Fedora machines is out of sync
• AWS machines should run b2g emulator reftests with GALLIUM_DRIVER=softpipe
• Intermittent Android x86 We have not been able to establish a telnet connection with the emulator
• Please update libxcb on the linux test slaves to a more recent release
• Upgrade ec2 test instances mesa versions from 8.0.2-0ubuntu3(32-bit hosts)/8.0.4-0ubuntu0.2(64-bit hosts) to 8.0.4-0ubuntu0.6mozilla1
• Ensure way to create VM that can generate our current Android 2.3 armv7 emulator
• Releases
• Add SeaMonkey 2.25 Beta 2 to bouncer
• Releases: Custom Builds
• Fix outdated and duplicated landing pages for MSN/Bing builds
• move inactive partner builds out of the build process
• Partner repack changes for Onet
• Update Rambler pack with updated extension 4.9.3
• Repos and Hooks
• New git repositories to mirror (for tarako)
• Tools
• beta2release_l10n.sh does not handle merged locales properly
• linux_gecko should be linux32_gecko in trychooser
• Spot bidding should be able to filter out AZs with bad conditions
• b2g_bumper should check tree status
• Integrate slaveapi into slave heath

In progress work (unresolved and not assigned to nobody):

• Balrog: Frontend
• balrog’s rules page is excessively wide
• General Automation
• support desktop build builders being created from Mozharness (SigningScriptFactory) or through Buildbot (MBF)
• Self-serve should be able to request arbitrary builds on a push (not just retriggers or complete sets of dep/PGO/Nightly builds)
• Tracking bug for 17-mar-2014 migration work
• Schedule Android 4.0 Debug M4,M5,M6,M7,J1,J2,J3 (only) on all trunk trees and make them ride the trains
• Stop running mochitest-browser-chrome on B2G branches
• Pass tooltool_script into misc.py, and make it work
• Requesting OMTC Reftest (Ro) variant for Windows
• Add tooltool support to Windows builds
• [tracking bug] Run desktop unittests on Ubuntu
• [Meta] Some “Android 4.0 debug” tests fail
• [tracker] run Android 2.3 test jobs on EC2
• b2g_build.py steps aren’t independent
• Please schedule linux jit-tests in two chunks
• take ownership of and fix relman merge scripts
• Run desktop mochitests-browser-chrome on Ubuntu
• Rename cppunittest to cppunit in mobile_config.py
• Do not use spot instances for some builders
• Do nightly builds with profiling disabled
• Install ant on builders
• prep buildbot.py for desktop builds in mozharness
• Run additional hidden 3 debug mochitest-browser-chrome chunks on m-i, and then switch on all trunk branches
• Please add non-unified builds to mozilla-central
• Cloning of hg.mozilla.org/build/tools and hg.mozilla.org/integration/gaia-central often times out with “command timed out: 1200 seconds without output, attempting to kill”, “command timed out: 1800 seconds without output, attempting to kill”
• Prepare infra to handle spot instances
• Fix compression for non-compressed uploads
• Bengali in 1.3t builds
• submit release builds to balrog automatically
• [tarako][build]create “tarako” build
• Intermittent Linux spot builder “command timed out: 2700 seconds without output, attempting to kill” while trying to install mock
• Loan Requests
• Slave loan request for a windows 7 32 bit machine
• Loan felipe an AWS unit test machine
• loan linux32 (hardware) slave to jmaher
• Need a slave for bug 818968
• Loan talos-linux64-ix-001 to dminor, ctalbert & gbrown
• Please loan dminor t-xp32-ix- instance
• Slave loan request for a tst-linux64-ec2 machine
• Slave loan request for a t-w864-ix machine
• Slave loan request for a t-w732-ix machine
• Loan an ami-6a395a5a instance to Dan Glastonbury
• Other
• Make AWS node type available to graphite & build metadata
• s/m1.large/m3.medium/
• Allow non-RelEng folk to enable/disable snippets and move/copy them between channels
• Platform Support
• [tracker] Move away from the rev3 minis
• Update builders to Android SDK platform 17
• Experiment with other instance types for Android 2.3 jobs
• Add msttcorefonts package to Servo buildbots
• Upgrade Windows 8 Nvidia drivers to 335.23
• figure out why test machines can’t set resolution sometimes
• Release Automation
• Thunderbird Windows L10n repack builds failed with pymake enabled
• staging release runner instructions don’t work with new dev-master1/ssh key management
• Update firefox-latest, firefox-stub, firefox-latest-euballot bouncer aliases as a part of post-release builder (and same for beta)
• Releases
• do a staging release of Firefox and Fennec 29.0b1
• tracking bug for build and release of Firefox and Fennec 29.0
• Add SeaMonkey 2.25 Beta 3 to bouncer
• Yandex partner repack changes for Fx 28 release
• tracking bug for build and release of Firefox and Fennec 28.0
• Tools
• Store the timing of actions in slaveapi
• more vcs-sync email tweaks
• Add an option to trychooser to select Talos profiling options
• self-serve agents eat exceptions, causing stalls
• self-serve agents’ error handling needs more care
• db-based mapper on web cluster

http://hearsum.ca/blog/this-week-in-mozilla-releng-march-14th-2014/