Testing

Regression tests are tests that compare new results with a known results and report any differences. Regression tests are a good first step towards creating software that will survive multiple developers over time.

Coverage

Regression tests are of little use without a measure of coverage. Note that 100% code coverage does not mean the code is completely tested, but high level of code coverage is a start.

Java developers can use JavaScope from Sun, which formerly was freely available to schools, but now lists at roughly $800/seat. The educational pricing is unknown at this time. JavaScope can generate HTML reports.

C++ developers can use tools like PureCov from Rational.

Tcl

The Ptolemy group has had good results using Tcl as a glue language for test suites.

Using scripting to write tests for Java or C/C++ is quick and easy

Writing tests is much more of an incremental process than writing system code - a scripted language makes sense

Being able to easily modify tests, and then run them from an interpreter makes test case development faster

Java and Tcl

Java classes can be accessed via Jacl and Tcl Blend. Jacl is a 100% Java implementation of a subset of Tcl, which is contained in one 675k jar file. Tcl Blend is an extension that can be loaded at runtime into a standard C based Tcl interpreter. The Ptolemy project is currently using Jacl for the test bed.

The Tcl command to instantiate a Java object: si

set a [java::new classname]

This returns a handle, like java0x4. We can then call Java methods on the handle:

$a toString

C++ authors can use Swig (http://www.swig.org) to generate wrappers for C++ classes.

Tcl Testing Framework

The Tcl testing framework was first implemented by Mary Ann May-Pumphrey of Sun Microsystems

The Tcl testing framework is fairly straight forward, test files consist of blocks of code like

test testname { testcomment} {
	# Do the test

	# The return value from the last line is compared with the 
	# known result
} {known result}

A primitive example would be:

test Add-1.1 {test the tcl expr command} {
	expr { 2 + 2 }
} {4}

If for some reason 2+2 did not equal 4, then when the test was run, a message would be printed. A simple test of the java.lang.String constructor might look like:

test SimpleTest-1.1 {Test Foo} {
	set a [java::new {String String} "A string"]
	$a toString
} {A string

An actual test from Ptolemy II looks like:

test NamedObj-2.1 {Create a NamedObj, set the name, change it} {
    set n [java::new  ptolemy.kernel.util.NamedObj]
    set result1 [$n getName]
    $n setName "A Named Obj"
    set result2 [$n getName]
    $n setName "A different Name"
    set result3 [$n getName]
    $n setName {}
    set result4 [$n getName]
    list $result1 $result2 $result3 $result4
} {{} {A Named Obj} {A different Name} {}}

Floating Point

Tests that return floating point results can present problems, especially when moving the tests to another platform. Often, the floating point representation is slightly different, so one platform might return 2.00000001, and another might return 1.9999999.

Unfortunately, Unix diff will indicate that such a change is an error, because the characters are different.

A better approach is to use an epsilon method to compare two numbers and return whether the results are close enough or not.