Changes between Version 2 and Version 3 of TestingGuidelines

Timestamp:

12/23/06 06:07:13 (6 years ago)

Author:

jarrod.millman

Comment:

--

TestingGuidelines

@@ +1 @@
+'''CAUTION:  This page is in the process of being migrated from an older page.  Until this process is completed, information found here may be incorrect.'''
 = Testing Guidelines =
 SciPy's testing structure is based on the unit testing framework offered by unittest.py. scipy_test.py also provides a few helper functions.
@@ -30 +31 @@
 }}}
 
+== Setting up your module for testing ==
+Every module or package requires two things for its testing to be included in the SciPy test suite.
+ 1. A test module -- Every directory in the SciPy directory structure has a {{{tests/}}} sub-directory. The test module for {{{dir1/module_xxx.py}}} is named {{{dir1/tests/test_module_xxx.py}}}.
+ 1. Two test routines -- Every module should have the functions {{{test()}}} and {{{test_suite()}}} defined. The first of these is used when running tests from the command line. The second returns a {{{unittest.TestSuite}}} object that can be used by other programs to test SciPy. This is beneficial, for instance, in generating web pages that show how healthly the current version of SciPy is.
+
+== Template for {{{test_module_xxx.py}}} ==
+More complete documentation for how to use the unittest framework is found here. The following code will give you a basic idea of how to add unit tests to your module in SciPy. Suppose you have a module {{{scipy/stats/foo.py}}} that has a function sum(a) that should accept a 1-D array or a scalar value. Create a test module called {{{scipy/stats/tests/test_foo.py}}}. This test file should include a class that tests {{{sum()}}} and a couple of other functions, {{{test_suite()}}} and {{{test()}}}. The test class has test methods that test various aspects of {{{sum()}}}. Within these test methods, {{{assert()}}} is used to test whether some case is true. If the assert fails, the test fails. Again, see here for more detailed info on defining test classes. The {{{test_suite()}}} function combines all the test classes that live in this module together to form a test suite that can be used for testing. The {{{test()}}} function actually runs the test suite. This file should look something like this:
+{{{
+...
+}}}
+
+== Template for {{{test()}}} and {{{test_suite()}}} in {{{module_xxx.py}}} ==
+
+Once the {{{test_foo.py}}} is written, its possible to run the tests by going to the {{{tests/}}} directory and typing:
+{{{
+python test_foo.py
+}}}
+Or if {{{tests/}} was put in the python path, you could run the tests interactively in the interpreter like this:
+{{{
+>>> import test_foo.py
+>>> test_foo.test()
+}}}
+Usually however, adding the {{{tests/}}} directory to the python path isn't desireable. It would be nice to invoke the test straight from the module {{{foo.py}}}. To this end, {{{test()}}} and {{{test_suite()}}} methods are added to the {{{foo.py}}} module also. They use a couple of helper functions that live in {{{scipy/common/scipy_test.py}}} to handle the messy work. Place the following two definitions at the end of your module:
+{{{
+""" foo.py
+"""
+
+def sum(a):
+   ...
+
+def test():
+    from scipy.common.scipy_test import module_test
+    module_test(__name__,__file__)
+
+def test_suite():
+    from scipy.common.scipy_test import module_test_suite
+    return module_test_suite(__name__,__file__)
+}}}
+Now you can do the following to test your module:
+{{{
+>>> import scipy
+>>> scipy.common.foo.test()
+}}}
+
+Also, when invoking the entire SciPy test suite, your tests will be found and run:
+{{{
+>>> import scipy
+>>> scipy.test() 
+# your tests are included and run automatically!
+}}}
+Testing packages is a little more work, but not bad. See {{{scipy/common/__init__.py}}}.
+
+== {{{scipy_test.py}}} ==
+
+This module holds a few helper routines. Several were discussed in the previous section. A couple of others are something of a verbose assert() function that give feedback as to what error occured. One of them, assert_array_equal(), is useful for comparing values in two arrays. The library of assert functions should grow to test for other general cases.
+
+== Testing Wishes ==
+
+ * '''Speed testing:''' It'd be nice to have some speed testing facilities so that we could keep track of whether new code hurts or helps our speed. It would provide some simple benchmarking facilities also. To do this correctly, we'd need something like {{{start_timer()}}} and {{{stop_timer()}}} calls that could be placed in the {{{check_xxx()}}} methods. This would allow the set up and error checking code at the beginning and end of check methods to be ignored for timing purposes. If these functions weren't used in the check method, then the entire check method would be timed.
+ * '''Regression testing and database:''' If each test could be stored in a database, including timing information, that would allow us to see how speed is changing over time (using a reference machine).
+ * '''Web output of test results:''' Haven't looked much at the {{{TestRunner}}} stuff, but I imagine it is possible to grab info from the test suite and write it out to an HTML file. We should do this on several architectures every evening on the scipy.org site.