CAUTION: This page is in the process of being migrated from an older page. Until this process is completed, information found here may be incorrect.

Introduction

SciPy's testing structure relies on the NumPy testing system, which is based on the unit testing framework offered by unittest.py.

Our goal is that every module and package in SciPy includes a thorough set of unit tests. These tests should exercise the functionality of a routine as well as its robustness to erroneous or unexpected input arguments. The best time to write the tests is when the module itself is being written. Whenever a new bug is found in a routine, a unit test should be written to test for the error so that it can't creep back in unnoticed after future code changes.

Running SciPy's test suite

To run tests on the scipy package, use the following:

>>> import scipy
>>> scipy.test()

This runs through the set of tests suite for SciPy. If you are only interested in testing a subset of SciPy, for example, the integrate module, use the following:

>>> scipy.integrate.test()

Levels and verbosity

The test method may takes two arguments; the first specifies the level of testing and the second the verbosity:

>>> scipy.test(level=1, verbosity=2)

The test level can be varied from 1 to 10. Increasing the verbosity provides more detailed messages about what tests are being executed.

level:
  None           --- do nothing, return None
  < 0            --- scan for tests of level=abs(level),
                     don't run them, return TestSuite-list
  > 0            --- scan for tests of level, run them,
                     return TestRunner

verbosity:
  >= 0           --- show information messages
  > 1            --- show warnings on missing tests

Writing your own tests

Much of SciPy is legacy code that was written without unit tests. As such, much of the functionality remains untested. However, more unit tests are being written all the time (and we encourage you to contribute). If you are writing a package that you'd like to become part of SciPy, please write the tests as you develop the package.

Setting up a module for testing

Every module or package requires two things for its testing to be included in the SciPy test suite.

1. A test module -- Every directory in the SciPy directory structure has a tests/ sub-directory. The test module for dir1/module_xxx.py is named dir1/tests/test_module_xxx.py.
2. Two test routines -- Every module should have the functions test() and test_suite() defined. The first of these is used when running tests from the command line. The second returns a unittest.TestSuite object that can be used by other programs to test SciPy. This is beneficial, for instance, in generating web pages that show how healthly the current version of SciPy is.

Template for test_module_xxx.py

The following code will give you a basic idea of how to add unit tests to your module in SciPy. Suppose you have a module scipy/stats/foo.py that has a function sum(a) that should accept a 1-D array or a scalar value. Create a test module called scipy/stats/tests/test_foo.py. This test file should include a class that tests sum() and a couple of other functions, test_suite() and test(). The test class has test methods that test various aspects of sum(). Within these test methods, assert() is used to test whether some case is true. If the assert fails, the test fails. Again, see here for more detailed info on defining test classes. The test_suite() function combines all the test classes that live in this module together to form a test suite that can be used for testing. The test() function actually runs the test suite. This file should look something like this:

...

Template for test() and test_suite() in module_xxx.py

Once the test_foo.py is written, its possible to run the tests by going to the tests/ directory and typing:

python test_foo.py

Or if {{{tests/}} was put in the python path, you could run the tests interactively in the interpreter like this:

>>> import test_foo.py
>>> test_foo.test()

Usually however, adding the tests/ directory to the python path isn't desireable. It would be nice to invoke the test straight from the module foo.py. To this end, test() and test_suite() methods are added to the foo.py module also. They use a couple of helper functions that live in scipy/common/scipy_test.py to handle the messy work. Place the following two definitions at the end of your module:

""" foo.py
"""

def sum(a):
   ...

def test():
    from scipy.common.scipy_test import module_test
    module_test(__name__,__file__)

def test_suite():
    from scipy.common.scipy_test import module_test_suite
    return module_test_suite(__name__,__file__)

Now you can do the following to test your module:

>>> import scipy
>>> scipy.common.foo.test()

Also, when invoking the entire SciPy test suite, your tests will be found and run:

>>> import scipy
>>> scipy.test() 
# your tests are included and run automatically!

Testing packages is a little more work, but not bad. See scipy/common/__init__.py.

numpy.testing

This package holds a few helper routines. Several were discussed in the previous section. A couple of others are something of a verbose assert() function that give feedback as to what error occured. One of them, assert_array_equal(), is useful for comparing values in two arrays. The library of assert functions should grow to test for other general cases.

Testing Wishes

• Speed testing: It'd be nice to have some speed testing facilities so that we could keep track of whether new code hurts or helps our speed. It would provide some simple benchmarking facilities also. To do this correctly, we'd need something like start_timer() and stop_timer() calls that could be placed in the check_xxx() methods. This would allow the set up and error checking code at the beginning and end of check methods to be ignored for timing purposes. If these functions weren't used in the check method, then the entire check method would be timed.
• Regression testing and database: If each test could be stored in a database, including timing information, that would allow us to see how speed is changing over time (using a reference machine).
• Web output of test results: Haven't looked much at the TestRunner stuff, but I imagine it is possible to grab info from the test suite and write it out to an HTML file. We should do this on several architectures every evening on the SciPy site.