Reducing Treeherder’s time-to-deploy

Up until September we had been using code merges from the master branch to the production one to cause production deployments.

A merge to production would trigger few automatic steps:

1. The code would get tested in the Travis CI (10 minutes or more)
2. Upon success the code would be built by Heroku (few minutes)
3. Upon success a Heroku release would happen (less than a minute)

If a regression was to be found on production we would either `git revert` a change out of all merged changes OR use Heroku’s rollback feature to the last known working state (without using Git).

Using `git revert` to get us back into a good state would be very slow since it would take 15–20 minutes to run through Travis, a Heroku build and a Heroku release.

On the other hand, Heroku’s rollback feature would be an immediate step as it would skip steps 1 and 2. Rolling back is possible because a previous build of a commit would still be available and only the release step would be needed .

The procedural change I proposed was to use Heroku’s promotion feature (similar to Heroku’s rollback feature). This would reuse a build from the staging app with the production app. The promotion process is a one-click button event that only executes the release step since steps 1 & 2 had already run on the staging app. Promotions would take less than a minute to be live.

This change made day to day deployments a less involved process since all deployments would take less that a minute. I’ve been quite satisfied with the change since a deployment requires less waiting around to validate a deployment.

http://feedproxy.google.com/~r/armenzg_mozilla/~3/GmmF8zSOX18/reducing-treeherders-time-to-deploy-a15c383ca5cd