Here are some examples:

[Expect int:[myArray count] toEqual:4];

[Expect object:user.name toEqual:@"Fred"];

[Expect array:validTokens toContainObject:user.token];

[Expect block:^{
    [fluxCapacitor rethread];
} toThrowExceptionWithReason:@"Uncharged capacitors cannot be rethreaded."];

[Expect blockToNotThrowException:^{
    [fluxCapacitor charge];
    [fluxCapacitor rethread];
}];

and here are some screenshots of the code completion in action:

Check out the GitHub repo or the public Pivotal Tracker project.

Often, clients who are new to writing automated tests start out by writing big integration tests (with tools like Selenium). This is an easy way to start testing an app that wasn’t written with testability in mind.

Unfortunately, these sorts of tests tend to be slow and brittle, and when they fail they don’t always point directly to a broken line of code in the way that a unit test would.

Over the years, I’ve come across a number of ways to help clients speed up and otherwise improve their test suites and I thought I’d enumerate them here.

Better Feedback

Continuous Builds

Continuous builds are vital to any tested application. I’ve used a few of them, but the simplest one I’ve used is CruiseControl.rb, a version of CruiseControl written in Ruby. It can run tests written in any language, and it is about 1000 times easier to deal with than the legacy version of CruiseControl.

I’ve also had a few clients who wrote their own continuous build systems. Two of them were simple (in a good way), and one seemed horribly complex. If you’re going to write your own, keep it simple!

Visible Continuous Build Monitor

If you’re lucky enough to have all your developers in a single room, you should consider displaying the continuous build results somewhere. Some clients have set up some hardware to show the status: red/green lava lamps, “pass”/”fail” neon signs, and Ambient Orbs that change color from red to green. A simpler solution is to display the status on a big TV screen or monitor. An old iMac seems like a really good way to display the status. There are now digital picture frames that can be updated over wifi — I wonder if those might be a cheap solution.

Fast-Failing Continuous Build

I helped one client who had written their own continuous build system to modify it so that the build turned red as soon as any test failed. They were usually able to check in a fix for the failing test before the suite finished running so that the fix was included in the very next build.

Encourage Unit Tests

While I believe that integration tests are important, the majority of a project’s tests should be unit tests. Unit tests are faster, less brittle, and when they fail, they often point directly at the code that’s not working.

When A Bug Is Found, Write a Unit Test

One way to build up a suite of unit tests for a program that doesn’t have very many is to commit to writing a unit test to expose any bug that’s found in the system. Sometimes in a non-test-driven codebase, it’s hard to write a really good unit test for a bug, but just adding coverage for part of the issue is a good step.

When An Integration Test Fails, Write A Unit Test

When an integration test fails, it’s often hard to tell exactly what went wrong. This is a good opportunity to write a unit test so that the next time something similar fails, you’ll have a very narrow test that finds it.

Ashcroft

There is a tool for Java called Ashcroft which will cause your test to fail if it does things that a strict unit test shouldn’t: start a thread, access the filesystem or network, etc.

I’ve helped clients separate their test suite into a “slow test” suite and a fast Ashcroft-enforced unit test suite. On the projects that have had this, developers would often try first to make an Ashcroft-compliant test and only settle for a slower integration test if they ran into trouble. (Sometimes on a project that wasn’t completely test driven, certain unit tests are just going to take too long to write.)

The Ashcroft suites usually run thousands of tests in a few seconds, whereas the slow suites usually run hundreds of tests in an hour.

Fake Services

You can often speed up tests by replacing some services with fake versions. For example, you might use an in-memory database such as HSQL or Mayfly. Or you might fake out a custom service with a faster one. For example, I have an app that stores files on Amazon S3 by calling a simple storage class I wrote. The tests use a different storage class that implements the same interface but stores the files in memory.

Remove Unnecessary Waiting

I’ve run across many Selenium tests that perform some action, wait a number of seconds, and then perform another action. All that waiting really adds up and leads to very long test suites.

An easy solution is to use Selenium’s waitForCondition command. For example, if your test clicks on a button that fires an asynchronous event and then shows the result on the page, instead of clicking, waiting 5 seconds and then asserting that the result is visible, just click wait for the result to become visible.

A similar approach is to create a waitForCondition-type method in your own code which calls a function repeatedly until it is true or until a timeout occurs.

Distribute Tests

One way to speed up a test suite is to throw hardware at it.

Multiple Builds

The simplest way to throw hardware at the problem is to split your continuous build into multiple suites, each of which run on its own server. You could write a script to check the results of each continuous build and tell you if the whole project is red or green, or just look at all the results and consider the entire project red if any of the suites are red. (Hopefully you have some sort of multiple-project continuous build dashboard like this one.)

MapReduce

One of my former clients distributes their tests using MapReduce. I don’t think they use Amazon’s MapReduce service, but that does seem like an easy way to throw a bunch of computers at your problem.

Selenium Farm

One reason that Selenium tests are so slow is that they run tests in a browser. Often, a team will want to test against different browsers at once, resulting in an even longer test run. I’ve helped clients speed up their test suites by putting together a farm of servers that run different browsers, and then running a bunch of tests in parallel. I found that on an ordinary developer box, I could easily run tests in 4 threads if the work of launching and running browsers was offloaded onto the farm.

I’ve also helped a couple clients run their JsUnit tests in Selenium to let them take advantage of the Selenium infrastructure. Basically, it amounts to writing a Selenium test that opens the JsUnit test runner page, clicks “run” and parses the results.

Setting up a Selenium farm is not trivial. Luckily, some folks have written and released an open source library called Selenium Grid to help. I haven’t used it, but it looks promising.

Test Running Services

A company called Sauce Labs is working on a service that runs Selenium tests. It doesn’t seem to be available yet, but the idea sounds very promising.

Conclusion

You can greatly improve your test suite run time, especially if you combine multiple solutions, if you don’t mind putting some time and money into the problem. Considering the costs of a slow test suite, I think it’s almost always worth the investment.

Jasmine is a new Javascript testing framework from my pals at Pivotal Labs.

Jaxer is an Apache module that wraps Firefox’s Javascript and rendering engine, providing a server-side Javascript environment that’s fast and compatible with any JS that works in Firefox (including all kinds of DOM manipulation and libraries like Prototype, jQuery, etc.).

Running Jasmine Tests Through Jaxer

Part of an app I’m writing uses Jaxer and I wanted to write my tests in Jasmine, so I wrote a test runner and reporter to bridge the two, and then I wrote a rake task to run the tests.

So now I can run “rake spec” and my JS tests will run, in Firefox, but without having to actually have Firefox open up (or be installed).

So:

rake spec

returns:

.    foo should do something.
.    foo should do something else.
.    foo should do yet another thing.

3 of 3 specs passed

And the whole thing takes 0.37 seconds. It should be noted that those are trivial tests, but running a JS test suite with only 0.37 seconds of overhead is pretty neat.

My rake task looks like:

task :spec do
  results = `curl -s http://127.0.0.1:8081/jaxer-service/jasmine-jaxer`
  print results
  fail "spec failures" if results.match(/FAILURES!/)
end

Download

The JS code that hooks Jasmine to Jaxer is here: jasmine-jaxer.js.

Sending mouse events such as click and mouseover in JsUnit tests can be really hard.

More Bad News

Prototype doesn’t make it any easier. Sam Stephenson says:

We would very much like to support it in the future. It’s fairly complicated to implement native event firing across all supported browsers, so in 1.6.0, fire works with custom events only.

YUI To The Rescue

YAHOO.util.UserActions can simulate some user actions. Unfortunately, calls to YUI can look a bit clunky in a Prototype-heavy codebase:

var element = new Element("div").insert("Hi");
var offset = element.cumulativeOffset();
YAHOO.util.UserAction.click(element, { shiftKey: true });

YUI + Prototype FTW

A little mixin magic:

Element.addMethods({
  simulateClick: YAHOO.util.UserAction.click.bind(YAHOO.util.UserAction),
  simulateDblClick: YAHOO.util.UserAction.dblclick.bind(YAHOO.util.UserAction),
  simulateMousedown: YAHOO.util.UserAction.mousedown.bind(YAHOO.util.UserAction),
  simulateMouseup: YAHOO.util.UserAction.mouseup.bind(YAHOO.util.UserAction),
  simulateMouseover: YAHOO.util.UserAction.mouseover.bind(YAHOO.util.UserAction),
  simulateMouseout: YAHOO.util.UserAction.mouseout.bind(YAHOO.util.UserAction),
  simulateMousemove: YAHOO.util.UserAction.mousemove.bind(YAHOO.util.UserAction)
});

and now our test code looks nicer:

var element = new Element("div").insert("Hi");
var offset = element.cumulativeOffset();
myElement.simulateClick({ shiftKey: true });

So in IE, JsUnit has to resort to searching the text of all scripts for strings that look like test function names. Unfortunately, it doesn’t understand comments so commented-out test functions will end up running anyway.

Recently, Dean McNamee and Dan Pupius alerted Edward Hieatt and me to an undocumented Internet Explorer Javascript function called RuntimeObject and suggested JsUnit might be able to use it to discover test functions.

I replaced the text searching with RuntimeObject and tagged it as version 2.2 alpha 25. (I also added an assertEqualsIgnoringOrder function while I was making changes.)

Browse the new version at SourceForge or check it out like this:

Try it out and let us know if there are any problems.

Show Log

There’s now a “Show Log” button that opens a window with a log of what JsUnit has been up to.

New log button (click image for larger view)

New log window (click image for larger view)

New URL Parameters

There are three new URL parameters:

• suppressDialogs=true will suppress JsUnit’s dialogs in interactive mode
• setupPageTimeout=n will change the timeout for the setupPage call
• pageLoadTimeout=n will change the page load timeout

Easier Suite Creation

The jsUnitTestSuite constructor now accepts test pages and subsuites as parameters. So instead of this:

function suite() {
  var suite = new jsUnitTestSuite();
  suite.addTestPage("/foo/barTest.html");
  suite.addTestPage("/foo/bazTest.html");
  return suite;
}

you can do this:

function suite() {
  return new jsUnitTestSuite("/foo/barTest.html", "/foo/bazTest.html");
}