Earlier this year, I wrote a blog post introducing Mutation Testing in JavaScript using the Grunt Mutation Testing framework. But as their NPM README said,

We will be working on (gradually) migrating the majority of the code base to the Stryker.

So I’ll update my post to use the latest Stryker framework. The following will be the updated post with all the code example migrated to the Stryker framework:

• Code example: github link

Last November (2015) I attended the EuroStar Software Testing Conference, and was introduced to a interesting idea called mutation testing. Ask yourself: “How do I ensure my (automated) unit test suite is good enough?”. Did you miss any important test? Is your test always passing so it didn’t catch anything? Is there anything un-testable in your code such that your test suite can never catch it?

Introduction to Mutation Testing

Mutation testing tries to “test your tests” by deliberately inject faults (called “mutants”) into your program under test. If we re-run the tests on the crippled program, our test suite should catch it (i.e. some test should fail.) If you missed some test, the error might slip through and the test will pass. Borrowing terms from Genetics, if the test fails, we say the mutant is “killed” by the test suite; on the opposite, if the tests passes, we say the mutant survived.

The goal of mutation testing is to kill all mutants by enhancing the test suite. Although 100% kill is usually impossible for even small programs, any progress on increasing the number can still benefit your test a lot.

The concept of mutation testing has been around for quite a while, but it didn’t get very popular because of the following reasons: first, it is slow. The number of possible mutations are just too much, re-compiling (e.g. C++) and re-run the test will take too long. Various methods has been proposed to lower the number of tests we need to run without sacrifice the chance of finding problems. The second is the “equivalent mutation” problem, which we’ll discuss in more detail in the examples.

Mutation Testing from JavaScript

There are many existing mutation testing framework. But most of them are for languages like Java, C++ or C#. Since my job is mainly about JavaScript (both in browser and Node.js), I wanted to run mutation testing in JavaScript.

I have found a few mutation testing frameworks for JavaScript, but they are either non-open-source or very academic. The most mature one I can find so far is the Stryker framework, which is released under Apache 2.0 license.

This framework supports mutants like changing the math operators, changing logic operators or even removing conditionals and block statements. You can find a full list of mutations with examples here.

Setting up the environment

You can follow the quickstart guide to install everything you need. The guide has a nice interactive menu so you can choose your favorite build system, test runner, test framework and reporting format. In this blog post I’ll demonstrate with my favorite combination: Vanilla NPM + Mocha + Mocha + clear-text.

You’ll need node and npm installed. (I recommended using nvm).

There are a few Node packages you need to install,

sudo npm install -g --save-dev mocha
npm install --save-dev stryker styker-api stryker-mocha-runner

Here are the list of packaged that I’ve installed:

"devDependencies": {
	"mocha": "^2.5.3",
	"stryker": "^0.4.3",
	"stryker-api": "^0.2.0",
	"stryker-mocha-runner": "^0.1.0"
}

A simple test suite

I created a simple program in src/calculator.js, which has two functions:

// ===== calculator.js =====
function substractPositive(num1, num2){
  if (num1 > 0){
    return num1 - num2;
  }
  else {
    return 0
  }
}

function add(num1, num2){
  if (num1 == num2){
    return num1 + num2;
  }
  else if (num1  num2){
    return num1 + num2;
  }
  else {
    return num1 + num2;
  }
}

module.exports.substractPositive = substractPositive
module.exports.add = add;

The first function is called substractPositive, it substract num2 from num1 if num1 is a positive number. If num1 is not positive, it will return 0 instead. It doesn’t make much sense, but it’s just for demonstrative purpose.

The second is a simple add function that adds two numbers. It has a unnecessary if...else... statement, which is also used to demonstrate the power of mutation testing.

The two functions are tested using test/test_calculator.js:

var assert = require("assert");
var cal = require("../src/calculator.js")

describe("Calculator", function(){
  it("substractPositive", function(){
    assert.equal("2", cal.substractPositive(1, -1));
  });

  it("add", function(){
    assert.equal("2", cal.add(1, 1));
  });
})

This is a test file running using mocha, The first verifies substractPositive(1, -1) returns 2. The second tests add(1,1) produces 2. If you run mocha in your commandline, you’ll see both the test passes. If you run mocha in the commandline you’ll see the output:

% mocha

  Calculator
    

https://shinglyu.github.io/testing/2016/10/11/Mutation_Testing_in_JavaScript_Using_Stryker.html