Digital Achievement

edit

System.out deemed Unreliable

Posted by Tim Freund Wed, 15 Nov 2006 04:47:00 GMT

Actually, System.out fulfills its purpose perfectly. Data goes in through print methods, and goes out to the console every single time. Before you complain about the misleading title, let me finish. Although great at printing, System.out is a completely unreliable test framework.

Who in their right mind uses System.out as a test framework? At no place in the javadocs does it mention code testing as an alternative use of System.out. How about a quick show of hands. How many people have written code like this:

System.out.println("widget.name == " + session.getWidget().getName());

I have, I admit. Quite often, actually. Printing a few statements is often times the easiest way to become familiar with a new API. When faced with particularly confusing libraries, especially closed source libraries, it is often helpful to do a little bit of detective work. Often times this detective work morphs into a tiny self standing program:

public class CogAdapter{
  /*
   * other code goes up here...
   */ 

  public static void main(String[] args){
    // the intuitive naming scheme in the catalog is awesome... ugh.
    Cog c = CogswellCogFactory.getCog(CogswellCatalog.X43RNT);
    Sprocket s = SpacelySprocketFactory.getSprocket();

    try{
      c.attach(s);
      System.out.println("cog.attach(sprocket) worked");
      System.out.println("cog.attachedObjects.length == " + 
                          c.getAttachedObjects().size());
    } catch(RatioException re){
      System.out.println("cog & sprocket ratios incompatible");
      System.out.println("cog.ratio == " + c.getRatio());
      System.out.println("sprocket.ratio == " + s.getRatio());
    } catch(JetsonException je){
      System.out.println("manufacturing defect in sprocket: " + je.getMessage());
    }
  }
}

Right, right, cogs don’t really attach directly to sprockets. That’s why this example is in the CogAdapter class. Duh. I didn’t want to pass up a fun Jetsons reference just because sprockets and cogs are not directly interchangeable. The object model of Fred Flintstone’s rock quarry just wasn’t a good fit for use in this example.

At this point System.out has evolved into the cornerstone of a miniature testing framework. Not a good testing framework, but one none the less. This output driven test framework requires labor intensive manual intervention for the tests to have any value. A person must manually run the tests and then manually verify the printed results with the intent of the code.

When I would write tests like this, I would get the test to work, satisfy my original curiosity, and then the test would never run again. At least not until the vendor upgraded their library and changed behavior. Even then, I had to actually remember that I wrote such a test when first working on the project before that test would again be called into service.

I saw the answer to these problems for years before I acted to remedy the situation. These little System.out driven test programs were perfect candidates to become JUnit tests, but I didn’t make that connection. I had only ever heard the “Test First” mantra, and I was obviously already knee deep in source code, so I thought that unit tests weren’t for me. Watching two of Mike Clark’s presentations at a NFJS conference in 2004 got me excited about integrating unit tests into my existing code.

How does JUnit improve upon our System.out test framework?

• JUnit tests can run one at a time, or all at once
• Every major Java IDE understands how to run JUnit
• Every major build system supports running JUnit tests, either through built in plugins or trivial customization.
• The mind numbing task of verifying results is left up to the computer.

Let’s transform the Sprocket/Cog test above into a real JUnit test.

public class CogAdapterTest extends TestCase {
  public void testAttach(){
    Cog c = CogswellCogFactory.getCog(CogswellCatalog.X43RNT);
    Sprocket s = SpacelySprocketFactory.getSprocket();

    try {
      c.attach(s);
      assertNotNull(c.getAttachedObjects());
      assertEquals(1, c.getAttachedObjects().length());
    } catch(RatioException re){
      assertEquals(c.getRatio(), s.getRatio());
    } catch(JetsonException je){
      fail("Incompetent Manufacturer");
    }
  }
}

No System.out references in sight. Now the computer will only produce output if something goes wrong, and once JUnit is scheduled to run as a part of the build process, your project will never go untested again.

System.out is great at printing data, but it makes a lousy test framework.

Ready to learn more? Check out these resources:

JUnit

Pragmatic Unit Testing in Java with JUnit or in C# with NUnit.

A Dozen Ways to Get the Testing Bug in the New Year by Mike Clark.