BIE regression test using DejaGNU

Changing or adding new code to a program can easily introduce errors. Regression testing allows one to confirm, or at least increase confidence, that no new errors or bugs have been introduced into the program with changes.

For BIE, there are four main types of regression tests that are important:

• CLI built in commands and expression syntax: This set of tests will check cli responds as expected to interaction. This requires running the built in cli commands, using a selection of expressions that tests CLI's expression syntax fairly comprehensively, and in each case checking that CLI's reaction is appropriate.

• Simulation results. We would like to confirm that simulation results are consistent through code changes. However, consistency can be difficult to ascertain due to the nature of the calculations performed by BIE as the exact values of the numbers in the output can vary slightly. At the moment, there are no tests of this kind.

• Functionality of classes and their methods. Most classes and methods operate deterministically and so their performance can be verified more easily. Functionality can be checked using either a CLI script or by writing a separate test harness in C++. The CLI script approach is more desirable because there is no requirement to build an executable, but not all classes and methods are supported by CLI and so this is not always possible.

• Tests for other tools and utilities. We also should provide testsuites for other tools or scripts that are part of the BIE distribution. An example of this is the galext tool, which extracts information about classes for use by cli.

DejaGNU

The DejaGnu software package provides a framework for regression tests, and this is the basis for our regression testing system. Test scripts are written in Expect, which is Tcl with a few additional features. See the DejaGnu page for download details and documentation.

Running existing tests

To run all regression tests, simply type make check in the root directory. Where an executable has to be built or rebuilt, make will do this before running the tests using this executable.

It is possible to run tests for one specific tool without testing all the other tools using the runtest command. For example, to run only cli tests:

1. Change into cli directory.
2. Run the command runtest --tool cli --srcdir testsuite

Adding new tests using the existing framework

• Adding a cli script test:
  1. Write a cli script. This script should produce enough output to confirm that everything is working exactly as expected.
  2. Run the script, and check the output.
  3. Copy the script and expected output to BIE/cli/testsuite/cli.test.
  4. Rename the script to SCRIPTPREFIX.testscript (ie. ".testscript" extension)
  5. Rename the expected output file to SCRIPTPREFIX.expected (ie ".expected" extension)
  6. The new test will be detected automatically.

• Adding an interactive cli test: Adding an interactive test for cli involves using the send and expect commands built into Expect, which send input and analyse output respectively. The basic form is as follows:

    set test testname  
    send "command string\n"

    expect {
        -re ".*prompt>"     { pass $test }
        -re "Exception.*"   { fail $test }
        timeout             { fail "(timeout) $test" }
    }
    

  There are several examples in the file cli_commands.exp, which is in the cli testsuite directory (cli/testsuite/cli.test).

• Adding a test to the testing directory:
  1. Write the C++ driver for the code you wish to test. This test should produce enough output to confirm that everything is working exactly as expected.
  2. Run the program, and check the output.
  3. Create a file containing the expected output to BIE/testing/testsuite/alltests.test.
  4. Add the following to BIE/testing/testsuite/alltests.test/testingdriver.exp:

       runcomparetest TESTNAME 1 "testsuite/alltests.test" EXECUTABLE_PATH ARGS
       

  5. Rename the file containing expected to TESTNAME.expected and move it to the BIE/testing/testsuite/alltests.test/ directory.
  6. The new test will run the next time you type make check.

  It is possible to indicate that a test is expected to fail. This is useful if a test exposes a problem but it is not a priority, for example. By incorporating the test you provide a reminder of the problem, but by indicating failure is expected people can run the tests and feel OK and the failure. To specify a test is expected to fail, make the second argument to runcomparetest a 0.

    runcomparetest <testname> 0 "testsuite/alltests.test" EXECUTABLE_PATH ARGS
    

• Adding a new simulation test Simulation tests not yet supported.

• Adding a galext test:
  1. Write one or more header files containing cli stylized comments. that will test the condition you are interested in checking. Choose a short name describing the test, and rename the header files so that they all have this name as a prefix.
  2. Run galext on this set of files, followed by gal_string_gen, gal_method_gen, and gal_constr_gen. Check that all the files generated by the scripts are correct.
  3. Save all the automatically generated files into the testsuite/galext.test directory, and rename then so that they have the form PREFIX_ACTUALNAME. For example, for the file "methods.out" and prefix "akmd", the file containing the expected contents would be "akmd_methods.out".
  4. Add a line to run the test in galext.exp. This will have the form:

       rungalexttest prefix "header1.h header2.h"
       

  It may be easier to modify an existing test and update the expected files for that test.