Major Highlights:

• Pete has been working away at getting a production grade hg <-> git revision mapper to replace the current quick&dirty one. This is an important service because we how we build B2G out of a combination of Mercurial and Git repositories.
• Jordan has been making great strides on moving Firefox Desktop builds to Mozharness, and Aki has been helping him keep pace with quick reviews. Once these are moved, RelEng Firefox builds will be much easier to reproduce outside of Buildbot, making it easier to debug problems that only happen on our machines.
• Kim presented her “Built to Scale” talk at EclipseCon!
• Armen enabled B2G reftests on EC2 machines, getting us one step closer to retiring our aging Fedora hardware.
• Hal helped us take ownership of and improve the merge day scripts. This should help merge days happen more efficiently in the future, and eventually be fully automated.
• Callek made great strides in getting SlaveAPI to support panda boards. When finished, sheriffs will have more control to deal with problematic pandas.
• Nick added support for locale-specific actions for updates. This will allow us improve user experience by doing things like only showing users “what’s new” pages if it’s been translated to their locale.

Completed work (resolution is ‘FIXED’):

• Balrog: Backend
• Updates not found since 2014-03-13 Firefox for Android Nightly/Aurora
• Support multiple products in a permission options
• Buildduty
• Integration Trees closed, high number/time backlog of pending jobs
• buildbot-master54's buildbot process got OOM’ed
• General Automation
• Enable EC2 b2g reftests and Ubuntu EC2 debug mochitest-b-c on try
• support desktop build builders being created from Mozharness (SigningScriptFactory) or through Buildbot (MBF)
• Self-serve should be able to request arbitrary builds on a push (not just retriggers or complete sets of dep/PGO/Nightly builds)
• Desktop UnitTests should not rely on PyPi as a repo
• submissions to balrog shouldn’t silently fail
• Parallelize spot instance tagging
• Turn off PGO, talos and debug builds on UX project branch
• b2g emulator tests not running since ~9am 20140319, all trunk and b2g-building release trees closed
• aws sanity check shouldn’t report instances that are actively doing work as long running
• Non-Fast Forward tag operation on git.mozilla.org/releases/gecko.git
• take ownership of and fix relman merge scripts
• Allow to run retried jobs on spot instances
• Bump MAX_BROKER_REFS to 4096
• ril repack failure – merge day failure
• Fix compression for non-compressed uploads
• Do not use spot instances for some builders
• Stop running builds and tests on pushes which only include changes to the b2g or mobile or browser directories
• emulator-kk builds
• Request: Nightly FirefoxOS engineering builds for nexus-4, please
• Increase the 9900 second time limit for browser-chrome
• Loan Requests
• Slave loan request for a t-w732-ix machine
• Slave loan request for gbrown
• Please lend Andreas Tolfsen a linux64 EC2 m3.medium slave
• Loan felipe an AWS unit test machine
• Other
• please add a –bootstrap cli flag to reftest/crashtest/jsreftest for android
• Platform Support
• deploy ccache 3.1.9 to linux builders
• Migrate ESR, b2g26_v1_2, b2g28_v1_3 branches to win64-rev2
• Update builders to Android SDK platform 17
• Add support for win64+gpo platform
• Release Automation
• Create SSL products in bouncer as part of release automation
• staging release runner instructions don’t work with new dev-master1/ssh key management
• Installer bouncer entries should be ssl-only
• turn off js/src/config/milestone.txt bumping for tb beta + add comment in esr24
• Releases
• tracking bug for build and release of Firefox and Fennec 28.0
• Thunderbird SSL-only Bouncer product
• do a staging release of Firefox and Fennec 29.0b1
• 29.0b1 failures for some mac ‘or’ locale
• Wrong link latest-esr/. Points to 24.3.0 instead of 24.4.0
• Stub Installer is serving Firefox 27.0.1
• Disable Aurora 30 daily updates until merge to mozilla-aurora has stabilized
• Please sign the hotfix to temporarily disable the malware blocklist (v20140319.01)
• Add SeaMonkey 2.25 to bouncer
• Releases: Custom Builds
• Version bumps for FF 28 builds
• Partner repack changes for web.de, gmx, and mail.com (Firefox 28)
• Repos and Hooks
• Create a script to import one Mercurial repository into a subdirectory of another, with linear history
• Pushing to projects/metro fails due to “Error accessing https://treestatus.mozilla.org/metro?format=json”
• Tools
• Remove android-noion from trychooser
• Add emulator-kk builds to trychooser
• Clean up trychooser’s tryload
• update merge_helper.py to work in releng environment
• Add gaia-integration to trychooser
• update beta2release to current working version
• Update b2g cronjobs to keep the next 12 weeks worth of builds after 3/17's merge day
• Add b2g desktop (win32)/win32_gecko to trychooser
• Spot bidding should be able to filter out AZs with bad conditions

In progress work (unresolved and not assigned to nobody):

• Balrog: Frontend
• balrog’s rules page is excessively wide
• Buildduty
• Reconfigs should be automatic, and scheduled via a cron job
• Determine RFO for 5 hour tree closure: No Linux64 test slaves running, 3+ hour pending
• General Automation
• Stop setting JAVA_HOME in buildbot and mozharness to a dir that does not exist
• Make blobber uploads discoverable
• ASan is the only Linux platform running the separate jittest suite on try
• Tracking bug for 17-mar-2014 migration work
• Enable all mozharness desktop build linux variants on Cypress
• Add a TURN server for running mochitest automation under a VPN
• Pass tooltool_script into misc.py, and make it work
• Requesting OMTC Reftest (Ro) variant for Windows
• Add tooltool support to Windows builds
• [tracking bug] Run desktop unittests on Ubuntu
• Improve prioritizeBuilders
• b2g build improvements
• [tracker] run Android 2.3 test jobs on EC2
• Use standard release channel names for B2G builds, remove slashes
• Run desktop mochitests-browser-chrome on Ubuntu
• Do nightly builds with profiling disabled
• Install ant on builders
• make mozharness test scripts easier to run standalone
• B2G desktop builds with git tryserver syntax in gaia.json
• Run additional hidden 3 debug mochitest-browser-chrome chunks on m-i, and then switch on all trunk branches
• Please add non-unified builds to mozilla-central
• Cloning of hg.mozilla.org/build/tools and hg.mozilla.org/integration/gaia-central often times out with “command timed out: 1200 seconds without output, attempting to kill”, “command timed out: 1800 seconds without output, attempting to kill”
• Speed up reconfigs on linux test masters
• Schedule Android 4.0 Debug M4,M5,M6,M7,J1,J2,J3 (only) on all trunk trees and make them ride the trains
• Turn off Metro Browser Chrome tests on inbound, central, cedar, ash, fx-team, etc.
• Mozharness test configs should live in-tree
• Avoid unnecessary buildbot builds for more parts of the tree
• add failure_config parameter to mozharness’s run_command()
• Bengali in 1.3t builds
• submit release builds to balrog automatically
• Split up mochitest-bc on desktop into ~30minute chunks
• [tarako][build]create “tarako” build
• Loan Requests
• Loan an ami-6a395a5a instance to Aaron Klotz
• Loan an ami-6a395a5a instance to Dan Glastonbury
• Loan an ami-3ac4aa0a instance to Tim Abraldes
• Slave loan request for a t-w864-ix machine
• Other
• [tracking] infrastructure improvements
• s/m1.large/m3.medium/
• Merge git build-cloud-tools to official hg cloud-tools
• Platform Support
• [tracker] Move away from the rev3 minis
• Ensure that AWS Windows Server 2012 instances are running the Windows Default theme
• signing win64 builds is busted
• Experiment with other instance types for Android 2.3 jobs
• win32file import failure on win64
• disable deja-dup-monitor and jockey-gtk services from autostarting on linux ubuntu hardware slaves
• [tracker] Determine how to support and roll out darwin-on-linux in production
• Releng Config/Automation support of staging run of scl1->scl3 move
• Upgrade Windows 8 Nvidia drivers to 335.23
• Setup Diamond to send metrics to hosted graphite
• AWS Windows Server 2012 tests failing due to start screen
• Release Automation
• make it possible to control whatsnew page showing or not via ship it, per-locale
• Update firefox-latest, firefox-stub, firefox-latest-euballot bouncer aliases as a part of post-release builder (and same for beta)
• Releases
• tracking bug for build and release of Firefox 24.4.0 ESR
• tracking bug for build and release of Firefox and Fennec 29.0
• Show Australis tour page upon update for en-US on Beta
• Releases: Custom Builds
• FF 28 config change for yahoo
• Tools
• Buildduty Report sorting should not error out (JS) if a section has no bugs
• Slaveapi should abstract out Machine from Slave
• Slaveapi should support mobile devices for reboots
• slaveapi should have a seperate knob for concurrency on buildapi calls alone
• permanent location for vcs-sync mapfiles, status json, logs
• Move mqext into the new https://hg.mozilla.org/hgcustom/version-control-tools repo
• self-serve agents eat exceptions, causing stalls
• db-based mapper on web cluster

http://hearsum.ca/blog/this-week-in-mozilla-releng-march-21st-2014/