I’ve heard from a few people that the bug long list of bugs I’ve been provided for the past few weeks is a little difficult to read. Starting with this week I’ll be supplementing that with some highlights at the top that talk about some of the more important things that were worked on throughout the week.

Highlights:

• Aki did some work on improving our documentation about things that need to happen on merge days. Historically, this has been a hairy process, and cleaning this up will help us complete it more quickly and without mistakes.
• Armen worked on extracting data to show how much CPU is used on EC2 machines. This data is a step on the road to more efficient use of them.
• Hal started work to enable the tree closure to trees that don’t yet have it. Enabling that will help sheriffs minimize tree closures of large branches (eg, mozilla-inbound) by having better control over incoming load.
• Catlee, Callek, Mike, Simone, and Massimo all attended TRIBE Session 1.
• Nick made it possible for us to ship Firefox 27.0 to our Beta users before we ship it to the Release channel. This gives us a better opportunity to find hard to reproduce bugs such as bug 865701 before we ship.
• Many people both in and outside of RelEng helped debug and fix network load issues that caused massive tree closures. Catlee wrote an in-depth blog post on this for those interested.
• Rail switched many of our EC2 machines from m3.xlarge to the cheaper (but just as fast) c3.xlarge instances.
• I did some experiments with smaller pools of build machines which is a start towards more intelligent build machine selection.

Completed work (resolution is ‘FIXED’):

• Buildduty
• Jobs not being scheduled
• Re-purpose mw32-ix-slave##, linux-ix-slave##, linux64-ix-slave##, bld-linux64-ix-05[1-3], mw32-ix-ref and linux-ix-ref as b-2008-ix-#### (rev2) machines
• Jobs not being scheduled
• Please update version of mock available on internal pypi
• Self-serve is down
• Hamachi nightlies on b2g18 are burning with “Error: Couldn’t find /builds/slave/b2g_m-b18_ham_ntly-00000000000/build/out/host/linux-x86/bin/fs_config”
• Revision mapper down
• deploy gcc 4.7.2 with newer binutils to tooltool server
• nightly builds failing because of switch to ftp-ssl
• Deploy and updated tools check out to all foopies
• upload a new talos.zip file
• General Automation
• b2g merge day work
• B2G emulator tests are broken on Pine and Cedar
• [tracker] deal with machines that will be obsoleted after the esr17 EOL
• Tracking bug for 3-feb-2014 migration work
• No Firefox linux nightly l10n builds since Jan 30
• limit pvtbuilds uploads
• Add ‘latest’ directory for mozilla-central TBPL b2g builds
• android nightlies failing to submit to balrog because of forbidden domain
• Compression for blobber
• Expose system resource usage of jobs
• Schedule debug emulator tests and builds on pine
• Schedule gaia-ui-tests on cedar against emulator builds
• Switch update server for Buri from OTA to FOTA by using the solution seen in bug 935059
• [Tracking bug] automation support for B2G v1.3.0
• Make gecko and gaia archives from device builds publicly available
• Loan Requests
• Please loan t-xp32-ix- instance to dminor
• Slave tst-linux64-ec2 or talos-r3-fed for :marshall_law
• Please loan t-w864-ix- instance to dminor
• Other
• Add AWS networks to inventory
• Try harder to get a fast slave
• recreate bad instances in us-west-2
• a bunch of usw2 slaves failed to puppetize
• Platform Support
• s/m3.xlarge/c3.xlarge/
• Build and deploy a patched gcc 4.7.3
• The clock of some Fedora machines is out of sync
• Migrate mozilla-release and comm-release to win64-rev2
• Releases
• Add SeaMonkey 2.24 Beta 1 to bouncer
• Manually create SSL products for Firefox 27
• tracking bug for build and release of Firefox 24.3.0 ESR
• tracking bug for build and release of Thunderbird 24.3.0
• Disable Aurora 29 daily updates until merge to mozilla-aurora has stabilized
• Add SeaMonkey 2.24 to bouncer
• Generate partial updates from 27.0build1 (27.0rc) to 28.0b1
• Yandex partner repack changes for Fx 27 release
• Releases: Custom Builds
• Update Bing favicon to match current branding for partner builds and add-ons
• Clean up MSN bundles and restore MSN add-on to the correct URL
• Repos and Hooks
• Clean up the repos that use the push_printurls hook
• Tools
• slaveapi’s reboot action shouldn’t create a bug for a slave unless it was unable to reboot it
• Support for instance type changes for aws_watch_pending.py
• Don’t display “long loans” in aws sanity report twice
• getting ambigous results for long running instances -> tst-w64-ec2-001 and tst-w64-ec2-002
• Teach slave health to file tracking bugs about b-2008-ix-*
• Mirror https://github.com/mozilla/gaia-ui-tests to hg
• Make git-hg mapfile public
• Loaner EC2 instances should not auto startup
• Standardize on single quotes in slave health
• # of constructors no longer being measured correctly
• Move js into libs as much as possible

In progress work (unresolved and not assigned to nobody):

• Balrog: Backend
• API for manipulating balrog rules
• Balrog shouldn’t serve updates to older builds
• Buildduty
• Testpool connection issues
• All trees closed due to timeouts as usw2 slaves download or clone
• General Automation
• Create two S3 buckets and make them available from build slaves
• Provision enough in-house master capacity
• Tracking bug for 17-mar-2014 migration work
• Provide B2G Emulator builds for Darwin x86
• Get existing metrofx talos tests running on release/project branches
• Create IAM roles for EC2 instances
• Rooting analysis mozconfig should be in the tree
• gaia-ui tests need to dump a stack when the process crashes
• Sometimes building files more than once on mac and linux
• Make Pine use mozharness production & limit the jobs run per push
• take ownership of and fix relman merge scripts
• Run desktop mochitests-browser-chrome on Ubuntu
• Use spot instances for regular builds
• switch to https for build/test downloads and hg
• split Android 4.0 robocop (rc) into 5 chunks
• Run win64 unit tests at bootup on tst-w64-ec2-xxx fork
• Implement ghetto “gaia-try” by allowing test jobs to operate on arbitrary gaia.json
• Remove ‘update_files’ logic from B2G unittest mozharness scripts
• Please add non-unified builds to mozilla-central
• Cloning of hg.mozilla.org/build/tools and hg.mozilla.org/integration/gaia-central often times out with “command timed out: 1200 seconds without output, attempting to kill”, “command timed out: 1800 seconds without output, attempting to kill”
• Flags passed to jit-test from mozharness should match flags in make check.
• final verification should report remote IP addresses
• b2g builds failing with AttributeError: ‘list’ object has no attribute ‘values’ | caught OS error 2: No such file or directory while running ['./gonk-misc/add-revision.py', '-o', 'sources.xml', '--force', '.repo/manifest.xml']
• keep buildbot master twistd logs longer
• Build the Gecko SDK from Firefox, rather than XULRunner
• Run unittests on Win64 builds
• submit release builds to balrog automatically
• Split up mochitest-bc on desktop into ~30minute chunks
• [tracking bug] migrate hosts off of KVM
• Run jit-tests from test package
• [tarako][build]create “tarako” build
• Use mach to invoke printconfigsetting.py
• Loan Requests
• request for a Windows 7 build slave
• Loan felipe an AWS unit test machine
• loan request for graydon [Ubuntu 64]
• Slave loan request for a talos-mtnlion-r5 machine
• Please lend Andreas Tolfsen a linux64 EC2 m3.medium slave
• loan linux slave to jmaher
• Slave loan request for a tst-linux64-ec2 machine
• Need a slave for bug 818968
• Other
• Create a long term archive of opt+debug builds on M-I for Regression Hunting
• s/m1.medium/m3.medium/
• Add b2g-inbound, fx-team, and mozilla-central to regression archive
• [tracking] infrastructure improvements
• Set up cron jobs to copy mozilla-inbound dep builds to S3
• s/m1.large/m3.large/
• Use mozmake for some windows builds
• Don’t clobber unrelated trees
• [tracking] move services from cruncher to production
• The stage.m.o:/mnt/cm-ixstore01 mount is filling up
• Platform Support
• [tracker] move jobs away from obsoleted machines
• AWS machines should run b2g emulator reftests with GALLIUM_DRIVER=softpipe
• Windows slaves often get permission denied errors while rm’ing files
• Determine number of iX machines to request for 2014
• Run Windows 8 unit tests on Date branch
• Create a Windows-native alternate to msys rm.exe to avoid common problems deleting files
• Release Automation
• Figure out how to offer release build to beta users
• Releases
• tracking bug for build and release of Firefox and Fennec 28.0b1
• Set up testing updates for 27.0 and older betas to 27.0 build1
• tracking bug for build and release of Firefox and Fennec 27.0
• Add SeaMonkey 2.25 Beta 1 to bouncer
• Releases: Custom Builds
• figure out strategy for custom build support under Metro
• Repos and Hooks
• Mercurial server should encourage best practices
• Add a hook to detect changesets with wrong file metadata
• Add every tree to treestatus, and add the treeclosure hook to every tree’s repo
• Tools
• tracker to retire legacy vcs2vcs
• Report Bug # in AWS loan report
• update merge day documentation
• Add an option to trychooser to select Talos profiling options
• permanent location for vcs-sync mapfiles, status json, logs
• cut over build/* repos to the new vcs-sync system
• slaveapi still files IT bugs for some slaves that aren’t actually down
• db-based mapper on web cluster
• Create a Comprehensive Slave Loan tool

http://hearsum.ca/blog/this-week-in-mozilla-releng-february-7th-2014-new-format-again/