Major Highlights:

• Catlee wrote a great blog post about the size of our updates and how we can reduce them.
• Joel discovered that test screenshots on some of our Linux test machines were broken, and Armen deployed a change to fix it. Debugging hard to reproduce failures one these machines should be easier again.
• Aki enabled nightly debug b2g emulator builds, which give QA much smaller regression windows for some types of failures. You can find these builds in directories like this.
• Kim has been working on enabling a few new types of tests.
• I enabled early testing of our new update server (Balrog) for Beta releases. If you’d like to give it a whirl following these instructions.
• Catlee enabled builds for our new 128MB phone (Tarako).
• Callek landed his work on SlaveAPI to support Panda boards. With this done, sheriffs have more tools to deal with flaky test machines.

Completed work (resolution is ‘FIXED’):

• Buildduty
• Set value of lastgood to False by default
• kmoir needs access to AWS console
• /data over 85 full (85%) on ip-10-39-70-110.ec2.internal
• Windows 8 tests failing across all branches
• watch pending and misc.py scheduling make different decisions about which slave to use for a retried job
• Android 2.3 tests not starting on Ash
• Integration Trees closed, high number of pending linux/android compile jobs about 1 hour backlog
• Please upload new xre.zip to tooltool
• Please upload new xre.zip to tooltool
• Determine RFO for 5 hour tree closure: No Linux64 test slaves running, 3+ hour pending
• All trees closed – mozpool issues with “We could not request the device: mozpool status not ok, code 500''
• nightly builds failing because of switch to ftp-ssl
• General Automation
• Stop setting JAVA_HOME in buildbot and mozharness to a dir that does not exist
• [tracking bug] Use AWS spot instances
• ASan is the only Linux platform running the separate jittest suite on try
• Tracking bug for 17-mar-2014 migration work
• Increase number of b2g reftest on EC2 chunks from 10 to 15
• Pass tooltool_script into misc.py, and make it work
• Increase free space requirement for Thunderbird l10n builds on central & aurora
• [tracking bug] Run desktop unittests on Ubuntu
• gaia-ui tests need to dump a stack when the process crashes
• b2g build improvements
• Please schedule linux jit-tests in two chunks
• take ownership of and fix relman merge scripts
• Drop the lastgood requirement for nightlies off m-c
• bld-lion-r5-037 debug failures may be showing a problem with bug 969689
• [tracking bug] Run b2g emulator unittests on Ubuntu
• mozharness reports success when build fails in some cases
• B2G desktop builds with git tryserver syntax in gaia.json
• generate signing token before doing build
• ccache sometimes hits fatal permission denied errors when trying to create temp files
• Add support for non-unified builds in mozharness desktop builds
• prep buildbot.py for desktop builds in mozharness
• Enable ASAN tests on the Jetpack tree
• [tarako][build]create “tarako” build
• Speed up reconfigs on linux test masters
• Improve prioritizeBuilders
• Add web-platform-tests to buildbot config
• tag_spot_instances.py exceptions
• Prepare infra to handle spot instances
• Increase maxtime for browser-chrome-3
• submit release builds to balrog automatically
• Stop running builds and tests on pushes which only include changes to the b2g or mobile or browser directories
• Loan Requests
• Loan an ami-6a395a5a instance to Dan Glastonbury
• borrow a linux ix multinode host to test CentOS-6.5
• Please loan dminor t-xp32-ix- instance
• Other
• Use pymake by default on Windows
• Merge git build-cloud-tools to official hg cloud-tools
• Platform Support
• Disable deja-dup-monitor and jockey-gtk services from autostarting on Linux ubuntu slaves
• Switch 32-bits linux builds to x86-64 mock environments
• Update builders to Android SDK platform 17
• Release Automation
• run clobbers after release
• release runner attempts to tag with the same tag multiple times in some situations
• make it possible to control whatsnew page showing or not via ship it, per-locale
• Releases
• Show Australis tour page upon update for en-US on Beta
• Repos and Hooks
• b2g branching automation
• Tools
• Slaveapi should abstract out Machine from Slave
• Slaveapi should support mobile devices for reboots
• slaveapi should have a seperate knob for concurrency on buildapi calls alone

In progress work (unresolved and not assigned to nobody):

• Balrog: Frontend
• balrog’s rules page is excessively wide
• Buildduty
• Reconfigs should be automatic, and scheduled via a cron job
• General Automation
• Make blobber uploads discoverable
• Add ssltunnel to xre.zip bundle used for b2g tests
• Enable all mozharness desktop build linux variants on Cedar
• Add a TURN server for running mochitest automation under a VPN
• Requesting OMTC Reftest (Ro) variant for Windows
• Add tooltool support to Windows builds
• decommission (releng side) aws buildbot masters migrated to scl3
• Show SM(Hf) builds on mozilla-aurora, mozilla-beta, and mozilla-release
• Add configs for debug B2G desktop builds & enable on mozilla-central and mozilla-aurora
• Please remove hamachi.zip and the out folder from the ril build for hamachi
• Mozharness test configs should live in-tree
• Bengali in 1.3t builds
• Make –enable-debug gecko device builds for B2G
• fx desktop builds in mozharness
• Run desktop mochitests-browser-chrome on Ubuntu
• Do nightly builds with profiling disabled
• Install ant on builders
• prioritize l10n nightlies below everything, regardless of branch
• Please schedule desktop jit-tests on all trunk trees and make them ride the trains
• Run additional hidden 3 debug mochitest-browser-chrome chunks and b2g reftests
• Please add non-unified builds to mozilla-central
• Cloning of hg.mozilla.org/build/tools and hg.mozilla.org/integration/gaia-central often times out with “command timed out: 1200 seconds without output, attempting to kill”, “command timed out: 1800 seconds without output, attempting to kill”
• Schedule Android 4.0 Debug M4,M5,M6,M7,J1,J2,J3 (only) on all trunk trees and make them ride the trains
• Updates not properly signed on the nightly-ux branch: Certificate did not match issuer or name.
• Schedule (mostly) B2G tests on Mozilla-B2g28-v1.3t
• Schedule desktop tests only on Holly
• Split up mochitest-bc on desktop into ~30minute chunks
• Run jit-tests from test package
• Loan Requests
• Loan t-w864-ix-003 to jrmuizel
• Please loan shu OS X 10.6 test runner
• Slave loan request for a t-w864-ix machine
• Other
• Use mozmake for some windows builds
• Upgrade ASan Clang in Q1
• [tracking] infrastructure improvements
• s/m1.large/m3.medium/
• Platform Support
• [tracker] Move away from the rev3 minis
• evaluate mac cloud options
• Experiment with other instance types for Android 2.3 jobs
• aws_create_ami.py should generate HVM and PV AMIs
• Releng Config/Automation support of staging run of scl1->scl3 move
• Upgrade Windows 8 Nvidia drivers to 335.23
• Release Automation
• Update firefox-latest, firefox-stub, firefox-latest-euballot bouncer aliases as a part of post-release builder (and same for beta)
• Releases
• tracking bug for build and release of Firefox and Fennec 29.0
• Releases: Custom Builds
• FF 28 config change for yahoo
• Repos and Hooks
• Request for a new git repository: “fuzz-tools” (public)
• Tools
• update relman scripts based on notes from first usage
• Add jittest to trychooser
• Store the timing of actions in slaveapi
• Setup docker apps for buildbot, buildapi and redis
• Add Windows support to aws_stop_idle.py
• Check mozilla bitbucket account is functioning correctly and that access key is still valid
• Create a Comprehensive Slave Loan tool

http://hearsum.ca/blog/this-week-in-mozilla-releng-march-28th-2014/