Anders Tornblad

All about the code

Monthly archive for January 2013

JavaScript unit test framework, part 2

While adding new features to a project, you're supposed to add only one or a few unit tests at a time, see them fail, and then implement as little code as possible to make it/them pass. The current version of the unit test framework only shows the number of passing/failing tests, so you cannot be sure about which test is actually failing – especially after making some code changes. Also, there is no way of seeing the actual error message thrown when running a failing unit test. I really need to improve on this, so I introduce the failures property, which will contain names and error messages of all failing tests.

There is also the case of testing asynchronous code, such as timers, event handlers, AJAX requests, CSS transitions and so on. The way the run method is currently implemented, there is no way of properly waiting for it to complete if some unit test contains event handlers or timers.

New requirements

  1. The engine should keep track of failing tests and their error messages
    • after eng.run() is done, the eng.failures should contain a list of objects containing the name and error message of each failing test
  2. Each test function should be called with a ut.TestContext object as its first argument
    • testContext.actAndWait(timeout, actFunc) should run the actFunc function, and wait timeout milliseconds before continuing
    • if the actFunc function crashes, the test is marked as a failed test
    • the actFunc function should receive a ut.TestContext object as its first argument
    • calling testContext.actDone() from within an asynchronous test function stops the waiting immediately
    • the testContext.actAndWait method should return the test context object itself, for call chaining purposes
    • calling testContext.thenAssert(assertFunc) should make the assertFunc function get called after the actFunc function is either timed out, or marked as done using the actDone() method
    • the assertFunc function should receive a ut.TestContext object as its first argument
  3. The framework should provide a function hook to be called after all unit tests have been run
    • the run() method of ut.Engine should return the engine itself
    • the then(func) method should register a function to be run after all unit tests are done running, passing the engine as the first parameter to the func function
    • If none of the registered unit tests contain any asynchronous code, calling run() should run all tests before returning, and the caller of run() shouldn't need to use the then() method

Fifth requirement

engine.add("The engine should keep track of which tests fail/succeed", function() { var eng = new ut.Engine(); var failFunc = function() { throw "I did crash!"; }; eng.add("Failing", failFunc);    // Act eng.run();    // Assert if (eng.failures.length !== 1) { throw "failures.length should be 1, but is " + eng.failures.length; } if (eng.failures[0].name != "Failing") { throw "failures[0].name should be 'Failing', but is " + eng.failures[0].name; } if (eng.failures[0].message != "I did crash!") { throw "failures[0].message should be 'I did crash!', but is " + eng.failures[0].message; } });

The implementation is pretty simple, and after this refactoring, I can use the failures property to list failing tests and their error messages.

run : function() { this.failures = []; for (var name in this.tests) { var testfunc = this.tests[name]; try { testfunc.call(); ++this.successCount; } catch(e) { this.failures.push({ name : name, message : e.toString() }); ++this.failureCount; } } } engine.run();    console.log(engine.failureCount + " failures and " + engine.successCount + " successes, out of " + engine.testCount + " tests");    for (var i = 0; i < engine.failures.length; ++i) { var fail = engine.failures[i]; console.log(fail.name + ": " + fail.message); }

Sixth and seventh requirement

Requirements for an API or a framework are often expressed better in code. This is how I want to be able to use the features described in requirements 6/7:

var eng = new ut.Engine();    eng.add("Asynch test", function(testContext) { // Arrange var someObj = new SomeClass(); var result = null;    // Act testContext.actAndWait(1000, function(tc) { someObj.someAsyncMethod({ success : function(r) { result = r; tc.actDone(); } }); }). // < -- notice the dot here... call chaining!    // Assert thenAssert(function(tc) { if (result == null) { throw "Timeout!"; } if (result.foo !== "bar") { throw "Expected 'bar', but found " + result.foo; } }); });    eng.run().then(function(engine) { // Display test results });

Requirements 6/7 – first batch

The first batch of eight unit tests addresses every piece of the sixth requirement except for the actDone() function.

engine.add("Test functions should be called with a ut.TestContext as its first argument", function() { // Arrange var innerTc; var eng = new ut.Engine(); eng.add("set inner test context", function(tc) { innerTc = tc; });   // Act eng.run();   // Assert if (!(innerTc instanceof ut.TestContext)) { throw "innerTc is not a ut.TestContext object"; } });   engine.add("testContext.actAndWait should return the testContext itself", function() { // Arrange var innerTc; var returnedTc; var eng = new ut.Engine(); eng.add("set inner test context", function(tc) { innerTc = tc; returnedTc = tc.actAndWait(1, function() {}); });   // Act eng.run();   // Assert if (innerTc !== returnedTc) { throw "actAndWait did not return the testContext itself"; } });   engine.add("actAndWait(timeout, actFunc) should run the actFunc, and wait (at least) timeout milliseconds", function(testContext) { // Arrange var timeout = 100; var calledAt = 0, endedAt = 0; var eng = new ut.Engine(); var actFunc = function() { calledAt = new Date().getTime(); } var testFunc = function(tc) { tc.actAndWait(timeout, actFunc); }; eng.add("actAndWait should wait correct amount of ms", testFunc);   // Act testContext.actAndWait(timeout + 100, function() { eng.run().then(function() { endedAt = new Date().getTime(); }); }).   // Assert thenAssert(function() { if (calledAt == 0) { throw "Did not call the actFunc function"; } if (endedAt == 0) { throw "Did not finish running the tests properly"; } // Minor timing issue: one or two milliseconds off is not a big deal if (endedAt < calledAt + timeout) { throw "Did not wait enough ms (waited " + (endedAt - calledAt) + " ms"; } }); });   engine.add("thenAssert(func) should called the assert function after (at least) the registered number of milliseconds", function(testContext) { // Arrange var timeout = 100; var calledAt = 0, assertedAt = 0; var eng = new ut.Engine(); var actFunc = function() { calledAt = new Date().getTime(); }; var assertFunc = function() { assertedAt = new Date().getTime(); }; var testFunc = function(tc) { tc.actAndWait(timeout, actFunc).thenAssert(assertFunc); } eng.add("thenAssert should wait correct amount of ms", testFunc);   // Act testContext.actAndWait(timeout + 100, function() { eng.run(); }).   // Assert thenAssert(function() { if (calledAt == 0) { throw "Did not call the actFunc function"; } if (assertedAt == 0) { throw "Did not call the assertFunc function"; } if (assertedAt < calledAt + timeout) { throw "Did not wait enough ms (waited " + (assertedAt - calledAt) + " ms"; } }); });   engine.add("if the actFunc for actAndWait crashes, the test should be failed", function(testContext) { // Arrange var eng = new ut.Engine(); eng.add("This should crash", function(tc) { tc.actAndWait(100, function() { throw "Crashing!"; }); });   // Run testContext.actAndWait(200, function() { eng.run(); }).   // Assert thenAssert(function() { if (eng.failures.length !== 1) { throw "Did not register exactly one failure"; } }); });   engine.add("then(func) should run func immediately if there are no asynchronous unit tests", function() { // Arrange var thenCalled = false; var eng = new ut.Engine(); eng.add("no-op test", function() { });   // Run eng.run().then(function() { thenCalled = true; });   // Assert if (!thenCalled) { throw "the thenFunc was not called"; } });   engine.add("then(func) should NOT run func immediately if there are some asynchronous unit test", function() { // Arrange var thenCalled = false; var eng = new ut.Engine(); eng.add("async test", function(tc) { tc.actAndWait(100, function() { }); });   // Run eng.run().then(function() { thenCalled = true; });   // Assert if (thenCalled) { throw "the thenFunc was called, but shouldn't!"; } });   engine.add("then(func) should run func after all asynchronous tests are done", function(testContext) { // Arrange var thenCalled = false; var eng = new ut.Engine(); eng.add("async test", function(tc) { tc.actAndWait(100, function() { }); });   // Run testContext.actAndWait(200, function() { eng.run().then(function() { thenCalled = true; }); }).   // Assert thenAssert(function() { if (!thenCalled) { throw "the thenFunc wasn't called"; } }); });   // new way of printing successes/failures engine.run().then(function() { console.log(engine.failureCount + " failures and " + engine.successCount + " successes, out of " + engine.testCount + " tests");   for (var i = 0; i < engine.failures.length; ++i) { var fail = engine.failures[i]; console.log(fail.name + ": " + fail.message); } });

This takes a pretty big piece of refactoring. I'm essentially transforming a sequential traversal of the tests property into a "wait-and-continue" loop using window.setTimeout to save engine state, halt the unit test engine, let a test run its course, then continue with the assert function or the next test.

First, the new ut.TestContext class:

var utTestContext = function(engine) { this.engine = engine; };   utTestContext.prototype = { actAndWait : function(timeout, actFunc) { this.engine.actAndWaitFunc = actFunc; this.engine.actAndWaitContext = this; this.engine.actAndWaitTimeout = timeout; return this; },   thenAssert : function(assertFunc) { this.engine.thenAssertFunc = assertFunc; this.engine.thenAssertContext = this; } };   window.ut = { "Engine" : utEngine, "TestContext" : utTestContext };

Then the new ut.Engine implementation:

var utEngine = function() { this.tests = {}; this.testCount = this.successCount = this.failureCount = 0; };   // private function, not exposed var runOneTestOrAssertFunc = function(engine, func, context) { try { func.call(null, context); if (!engine.actAndWaitFunc) { ++engine.successCount; } } catch(e) { engine.failures.push({ name : engine.currentTestName, message : e.toString() }); ++engine.failureCount; } };   utEngine.prototype = { add : function(name, testfunc) { this.tests[name] = testfunc; ++this.testCount; },   run : function() { if (this.initialized !== true) { this.initialized = true;   this.failures = [];   this.testNameIndex = 0; this.testNames = []; for (var name in this.tests) this.testNames.push(name); }   this.running = true;   if (this.actAndWaitFunc) { runOneTestOrAssertFunc(this, this.actAndWaitFunc, this.actAndWaitContext);   delete this.actAndWaitFunc; var self = this;   // pause the engine for a number of milliseconds this.actAndWaitTimeoutId = window.setTimeout(function() { self.run(); }, this.actAndWaitTimeout);   return this; }   if (this.thenAssertFunc) { runOneTestOrAssertFunc(this, this.thenAssertFunc, this.thenAssertContext);   delete this.thenAssertFunc; delete this.thenAssertContext; }   while (this.testNameIndex < this.testNames.length) { var name = this.testNames[this.testNameIndex++]; var testFunc = this.tests[name]; var context = new ut.TestContext(this); this.currentTestName = name;   runOneTestOrAssertFunc(this, testFunc, context);   if (this.actAndWaitFunc) { var self = this; window.setTimeout(function() { self.run(); }, 0); return this; } }   this.running = false;   if (this.thenFunc) { this.thenFunc.call(null, this); }   return this; },   then : function(thenFunc) { if (this.running) { this.thenFunc = thenFunc; } else { thenFunc.call(null, this); } } };

The unit test framework is starting to be really useful now, but is still only just over a hundred lines of production code, and about 350 lines of test code.

Requirements 6/7 – second batch

If you have lots of asynchronous unit tests where you need a large timeout value, but the tests still might finish quickly, it doesn't really feel good to always wait for the maximum expected timeout before moving on to the next test. You should be able to move on instantly if a test finishes early. That's why I add the actDone() method to tell the engine to move on to assertion and/or the next test instantly.

engine.add("actDone() should move immediately to the thenAssert assert function", function(testContext) { // Arrange var calledAt = 0, assertAt = 0; var eng = new ut.Engine(); eng.add("10ms func with 10000ms timeout", function(tc) { tc.actAndWait(10000, function() { calledAt = new Date().getTime(); window.setTimeout(function() { tc.actDone(); }, 10); }).thenAssert(function() { assertAt = new Date().getTime(); }); });   // Act testContext.actAndWait(500, function() { eng.run(); }).   // Assert thenAssert(function() { if (assertAt === 0) { throw "Assert wasn't called!"; } if (assertAt > (calledAt + 100)) { throw "Assert took way too long to get called!"; } }); }); The solution is to add this to the ut.TestContext prototype: actDone : function() { var engine = this.engine; if (engine.actAndWaitTimeoutId) { window.clearTimeout(engine.actAndWaitTimeoutId); delete engine.actAndWaitTimeoutId;   window.setTimeout(function() { engine.run(); }, 0); } }

There it is. Sixteen unit tests have rendered 137 lines of production code, which actually makes a pretty decent unit test framework. What is missing is a group of convenient helper functions and hooks for asserting and pretty output. If you want to use this in a automated testing environment it is already pretty much good to go. In the then() function, you could add a jQuery.ajax call to automatically post the test results to some server. Then just add a post-build script to your CI environment of choice to run your unit tests in a number of different browsers.

Next part will focus on assert helper functions and output hooks. Then I will look into mocking and some inversion of control magic.

The finished code can be found on github at /lbrtw/ut.

JavaScript unit test framework, part 1
JavaScript unit test framework, part 2 (this part)
JavaScript unit test framework, part 3
JavaScript unit test framework, part 4

JavaScript unit test framework, part 1

I have been involved in lots of agile and non-agile development projects, so I have experienced enough benefits of agile test-driven development to see a pattern. Smaller codebase, higher quality code, less bugs, more fun, lower startup threshold for adding new developers to the team, more efficient refactoring, maintainability, etc.

One key tool in achieving all of that is a unit test framework, which is why my first LBRTW project is to develop such a framework in JavaScript. Also, I will use the very unit test framework I'm writing to unit-test the framework itself.

The intended users of this framework are developers, so I can use pretty technical language in the specs, but I will still focus on keeping requirements short and concise. The first group of requirements look like this:

  1. eng = new ut.Engine() should create a new unit testing engine
    • upon creating a new engine, the eng.testCount should be zero
  2. eng.add(name, testfunc) should add a new unit test to the engine
    • eng.tests[name] should point to the testfunc function
    • the eng.testCount property should be increased
  3. eng.run() should run the test function of each added unit test once
    • if running the test function throws an exception, that indicates a failed unit test
    • all unit tests should always run, even if some unit test function crashes (or even all of them)
  4. The engine should keep track of the number of failed/succeeded tests
    • after eng.run() is done, the eng.successCount and eng.failureCount should contain the number of succeeded/failed unit tests respectively, and the sum of them should be the same as eng.testCount

First requirement

After establishing this, writing and running the first test is easy:

var engine = new ut.Engine();    engine.add("new ut.Engine should create a new unit testing engine", function() { // Act var eng = new ut.Engine();    // Assert if (eng.testCount !== 0) { throw "Did not set testCount to zero"; } });    engine.run();

Of course, when I try to run this, I will get a reference error saying ut is not defined. Also, I won't be able to actually run any tests before both add and run are somewhat implemented, so here is iteration zero of ut.Engine:

var utEngine = function() { this.tests = []; };    utEngine.prototype = { add : function(name, testfunc) { this.tests.push(testfunc); },    run : function() { for (var i = 0; i < this.tests.length; ++i) { var testfunc = this.tests[i]; testfunc.call(); } } };    window.ut = { "Engine" : utEngine };

Now it's possible to add and run unit tests, so it actually produces the first failing unit test output, albeit only visible in the developer console: Did not set testCount to zero. One small code change, and no errors are thrown:

var utEngine = function() { this.tests = []; this.testCount = 0; };

Second requirement

The next requirement deals with adding test functions to the tests collection and increasing the testCount property. This is what those tests look like:

engine.add("add() should set tests[name] to func", function() { // Arrange var eng = new ut.Engine(); var bar = function() {};    // Act eng.add("foo", bar);    // Assert if (eng.tests["foo"] !== bar) { throw "tests.foo does not point to bar"; } });    engine.add("add() should increase testCount", function() { // Arrange var eng = new ut.Engine(); var func = function() {};    // Act eng.add("foo", func);    // Assert if (eng.testCount !== 1) { throw "Did not increase testCount"; } });

The first test is made to pass by refactoring the tests array into an anonymous object, changing the add method to add by name instead of by index, and the run method to traverse the object using for ... in. The second test passes after a small refactoring of the add method:

var utEngine = function() { this.tests = {}; this.testCount = 0; };    utEngine.prototype = { add : function(name, testfunc) { this.tests[name] = testfunc; ++this.testCount; },    run : function() { for (var name in this.tests) { var testfunc = this.tests[name]; testfunc.call(); } } };

Third requirement

If this one isn't satisfied, any failing test will stop all concurrent tests from running, which will only allow us to deal with one failing test at a time.

engine.add("run() should run each added test once", function() { // Arrange var eng = new ut.Engine(); var called = [0, 0]; var func1 = function() { called[0]++; }; var func2 = function() { called[1]++; }; eng.add("func1", func1); eng.add("func2", func2);    // Act eng.run();    // Assert if (called[0] !== 1) { throw "Did not call func1"; } if (called[1] !== 1) { throw "Did not call func2"; } });    engine.add("run() should run all tests even when crash", function() { // Arrange var eng = new ut.Engine(); var called = 0; var func = function() { ++called; throw "Crash!"; } eng.add("Going once", func); eng.add("Going twice", func);    // Act eng.run();    // Assert if (called !== 2) { throw "Did not call both added tests"; } });

The first test of this requirement already passes, but the second one does crash. It doesn't even run all the way to the assertion part. It's the test function itself that produces the developer console output – not the unit test framework. To make the test pass, I simply wrap calling test functions in try ... catch.

run : function() { for (var name in this.tests) { var testfunc = this.tests[name]; try { testfunc.call(); } catch(e) { } } }

Fourth requirement

When writing the unit test for this requirement, I also adopt the new way of checking for failing unit tests. Instead of just letting the developer console print out the exception message, I print out the values of failureCount, successCount and testCount after run() has been called.

engine.add("The engine should count successes and failures", function() { // Arrange var eng = new ut.Engine(); var failFunc = function() { throw "Crash!"; }; var successFunc = function() { }; eng.add("One fail", failFunc); eng.add("Two fails", failFunc); eng.add("Three fails", failFunc); eng.add("One success", successFunc);    // Act eng.run();    // Assert if (eng.successCount !== 1) { throw "successCount should be 1, but is " + eng.successCount; } if (eng.failureCount !== 3) { throw "failureCount should be 3, but is " + eng.failureCount; } });    engine.run();    console.log(engine.failureCount + " failures and " + engine.successCount + " successes, out of " + engine.testCount + " tests"); When running all unit tests now, the output simply says undefined failures and undefined successes, out of 6 tests, simply because the engine does not yet count failures or successes. A small refactoring of the constructor and the run method later: var utEngine = function() { this.tests = {}; this.testCount = this.successCount = this.failureCount = 0; };    // inside prototype definition: run : function() { for (var name in this.tests) { var testfunc = this.tests[name]; try { testfunc.call(); ++this.successCount; } catch(e) { ++this.failureCount; } } }

There it is – the first iteration of my JavaScript unit test framework. Feel free to use it. There is still lots of important stuff to do, like a way of knowing which tests are failing and not just how many of them.

The finished code can be found on github at /lbrtw/ut.

JavaScript unit test framework, part 1 (this part)
JavaScript unit test framework, part 2
JavaScript unit test framework, part 3
JavaScript unit test framework, part 4