Erik A. Hanson's Weblog

As a proponent of test driven development, I’m naturally a big fan of writing automated tests.

Often, clients who are new to writing automated tests start out by writing big integration tests (with tools like Selenium). This is an easy way to start testing an app that wasn’t written with testability in mind.

Unfortunately, these sorts of tests tend to be slow and brittle, and when they fail they don’t always point directly to a broken line of code in the way that a unit test would.

Over the years, I’ve come across a number of ways to help clients speed up and otherwise improve their test suites and I thought I’d enumerate them here.

Better Feedback

Continuous Builds

Continuous builds are vital to any tested application. I’ve used a few of them, but the simplest one I’ve used is CruiseControl.rb, a version of CruiseControl written in Ruby. It can run tests written in any language, and it is about 1000 times easier to deal with than the legacy version of CruiseControl.

I’ve also had a few clients who wrote their own continuous build systems. Two of them were simple (in a good way), and one seemed horribly complex. If you’re going to write your own, keep it simple!

Visible Continuous Build Monitor

If you’re lucky enough to have all your developers in a single room, you should consider displaying the continuous build results somewhere. Some clients have set up some hardware to show the status: red/green lava lamps, “pass”/”fail” neon signs, and Ambient Orbs that change color from red to green. A simpler solution is to display the status on a big TV screen or monitor. An old iMac seems like a really good way to display the status. There are now digital picture frames that can be updated over wifi — I wonder if those might be a cheap solution.

Fast-Failing Continuous Build

I helped one client who had written their own continuous build system to modify it so that the build turned red as soon as any test failed. They were usually able to check in a fix for the failing test before the suite finished running so that the fix was included in the very next build.

Encourage Unit Tests

While I believe that integration tests are important, the majority of a project’s tests should be unit tests. Unit tests are faster, less brittle, and when they fail, they often point directly at the code that’s not working.

When A Bug Is Found, Write a Unit Test

One way to build up a suite of unit tests for a program that doesn’t have very many is to commit to writing a unit test to expose any bug that’s found in the system. Sometimes in a non-test-driven codebase, it’s hard to write a really good unit test for a bug, but just adding coverage for part of the issue is a good step.

When An Integration Test Fails, Write A Unit Test

When an integration test fails, it’s often hard to tell exactly what went wrong. This is a good opportunity to write a unit test so that the next time something similar fails, you’ll have a very narrow test that finds it.

Ashcroft

There is a tool for Java called Ashcroft which will cause your test to fail if it does things that a strict unit test shouldn’t: start a thread, access the filesystem or network, etc.

I’ve helped clients separate their test suite into a “slow test” suite and a fast Ashcroft-enforced unit test suite. On the projects that have had this, developers would often try first to make an Ashcroft-compliant test and only settle for a slower integration test if they ran into trouble. (Sometimes on a project that wasn’t completely test driven, certain unit tests are just going to take too long to write.)

The Ashcroft suites usually run thousands of tests in a few seconds, whereas the slow suites usually run hundreds of tests in an hour.

Fake Services

You can often speed up tests by replacing some services with fake versions. For example, you might use an in-memory database such as HSQL or Mayfly. Or you might fake out a custom service with a faster one. For example, I have an app that stores files on Amazon S3 by calling a simple storage class I wrote. The tests use a different storage class that implements the same interface but stores the files in memory.

Remove Unnecessary Waiting

I’ve run across many Selenium tests that perform some action, wait a number of seconds, and then perform another action. All that waiting really adds up and leads to very long test suites.

An easy solution is to use Selenium’s waitForCondition command. For example, if your test clicks on a button that fires an asynchronous event and then shows the result on the page, instead of clicking, waiting 5 seconds and then asserting that the result is visible, just click wait for the result to become visible.

A similar approach is to create a waitForCondition-type method in your own code which calls a function repeatedly until it is true or until a timeout occurs.

Distribute Tests

One way to speed up a test suite is to throw hardware at it.

Multiple Builds

The simplest way to throw hardware at the problem is to split your continuous build into multiple suites, each of which run on its own server. You could write a script to check the results of each continuous build and tell you if the whole project is red or green, or just look at all the results and consider the entire project red if any of the suites are red. (Hopefully you have some sort of multiple-project continuous build dashboard like this one.)

MapReduce

One of my former clients distributes their tests using MapReduce. I don’t think they use Amazon’s MapReduce service, but that does seem like an easy way to throw a bunch of computers at your problem.

Selenium Farm

One reason that Selenium tests are so slow is that they run tests in a browser. Often, a team will want to test against different browsers at once, resulting in an even longer test run. I’ve helped clients speed up their test suites by putting together a farm of servers that run different browsers, and then running a bunch of tests in parallel. I found that on an ordinary developer box, I could easily run tests in 4 threads if the work of launching and running browsers was offloaded onto the farm.

I’ve also helped a couple clients run their JsUnit tests in Selenium to let them take advantage of the Selenium infrastructure. Basically, it amounts to writing a Selenium test that opens the JsUnit test runner page, clicks “run” and parses the results.

Setting up a Selenium farm is not trivial. Luckily, some folks have written and released an open source library called Selenium Grid to help. I haven’t used it, but it looks promising.

Test Running Services

A company called Sauce Labs is working on a service that runs Selenium tests. It doesn’t seem to be available yet, but the idea sounds very promising.

Conclusion

You can greatly improve your test suite run time, especially if you combine multiple solutions, if you don’t mind putting some time and money into the problem. Considering the costs of a slow test suite, I think it’s almost always worth the investment.