Raptor on Gaia-CI Refactored

A lot has happened in the world of Raptor in the last couple of months. The Raptor tool itself has been migrated out of Gaia, and is now available as it’s own Raptor CLI Tool installed as a global NPM package. The new Raptor tool has several improvements including the addition of Marionette, allowing Raptor tests to use Marionette to drive the device UI, and in turn spawn performance markers.

The automation running Raptor on Gaia-ci has now been updated to use the new Raptor CLI tool. Not only has the tool itself been upgraded, but how the Raptor suite is running on Gaia-ci has also been completely refactored, for the better.

In order to reduce test variance, the coldlaunch test is now run on the base Gaia version and then on the patch Gaia version, all on the same taskcluster worker/machine instance. This has the added benefit of just having a single Raptor app launch test task per Gaia application that the test is supported on.

New Raptor Tasks on Treeherder

The Raptor tasks are now visible by default on Treeherder for Gaia and Gaia-master, and appear in a new Raptor group as seen here:

All of the applications that the coldlaunch performance test were performed on are listed (currently 12 apps). The Raptor suite is run once by default now (on both base and patch Gaia revisions) and results calculated and compared all in the same task, per app. If an app startup time regression is detected, the test is automatically retried up to 3 more times. The app task is marked as an orange failure if all 4 runs show a 15% or greater regression in app startup time.

To view the coldlaunch test results for any app, click on the corresponding app task symbol on Treeherder. Then on the panel that appears towards the bottom, click on “Job details”, and then “Inspect Task”. The Task Inspector opens in a new tab. Scroll down and you will see the console output which includes the summary table for the base and patch Gaia revs, and the overall result (if a regression has been detected or not). If a regression has been detected, the regression percentage is displayed; this represents the increase in application launch time.

Retriggering the Launch Test

If you wish to repeat the app launch test again for an individual app, perhaps to further confirm a launch regression, it can be easily retriggered. To retrigger the coldlaunch test for a single app, simply click the task representing the app and retrigger the task how you normally would on Treeherder (login and click on the retrigger job button). This will start the Raptor coldlaunch test again just for the selected app, on both the gaia base and patch revisions as before. The versions of Gaia and the emulator (gecko) will remain the same as they were in the original suite run, when the test is retriggered on an individually-selected app.

If you wish to repeat the coldlaunch test on ALL of the apps instead of just one, then you can retrigger the entire Raptor suite. Select the Raptor decision task (the green “R“, as seen below) and click the retrigger button:

Note: Retriggering the entire Raptor suite will cause the emulator (gecko) version to be updated to the very latest before running the test, so if a new emulator build has been completed since the Raptor suite was originally run, then that new emulator (gecko) build will be used. The Gaia base and patch revisions used in the tests will be the same revisions used on the original suite run.

For more information drop by the #raptor channel on Mozilla IRC, or send me a message via the contact form on the About page of this blog. Happy coding!

http://robwood.zone/raptor-on-gaia-ci-update/