Context Navigation

Changes between Version 4 and Version 5 of TestingGuidelines

Timestamp:: 12/23/06 06:28:55 (6 years ago)
Author:: jarrod.millman
Comment:: --

Legend:

: Unmodified
: Added
: Removed
: Modified

TestingGuidelines

v4	v5
2	2	= Testing Guidelines =
3	3	[[PageOutline]]
4		SciPy's testing structure relies on the [http://projects.scipy.org/scipy/numpy/browser/trunk/numpy/testing NumPy testing system], which is based on the unit testing framework offered by [http://docs.python.org/lib/module-unittest.html unittest.py].
	4	!SciPy's testing structure relies on the [http://projects.scipy.org/scipy/numpy/browser/trunk/numpy/testing NumPy testing system], which is based on the unit testing framework offered by [http://docs.python.org/lib/module-unittest.html unittest.py].
5	5
6		Our goal is that every module and package in SciPy includes a thorough set of unit tests. These tests should exercise the functionality of a routine as well as its robustness to erroneous or unexpected input arguments. The best time to write the tests is when the module itself is being written. Whenever a new bug is found in a routine, a unit test should be written to test for the error so that it can't creep back in unnoticed after future code changes.
	6	Our goal is that every module and package in !SciPy includes a thorough set of unit tests. These tests should exercise the functionality of a routine as well as its robustness to erroneous or unexpected input arguments. The best time to write the tests is when the module itself is being written. Whenever a new bug is found in a routine, a unit test should be written to test for the error so that it can't creep back in unnoticed after future code changes.
7	7
8		Much of SciPy is legacy code that was written without unit tests. As such, much of the functionality remains untested. However, more unit tests are being written all the time (and we encourage you to contribute). If you are writing a package that you'd like to become part of SciPy, please write the tests as you develop the package.
	8	Much of !SciPy is legacy code that was written without unit tests. As such, much of the functionality remains untested. However, more unit tests are being written all the time (and we encourage you to contribute). If you are writing a package that you'd like to become part of !SciPy, please write the tests as you develop the package.
9	9
10		== Running SciPy's test suite ==
11		To run tests on the ~~scipy~~ package, use the following:
	10	== Running !SciPy's test suite ==
	11	To run tests on the {{{scipy}}} package, use the following:
12	12	{{{
13	13	>>> import scipy
14	14	>>> scipy.test()
15	15	}}}
16		This runs through the set of tests suite for ~~SciPy. If you are only interested in testing a subset of~~ SciPy, for example, the {{{integrate}}} module, use the following:
	16	This runs through the set of tests suite for !SciPy. If you are only interested in testing a subset of !SciPy, for example, the {{{integrate}}} module, use the following:
17	17	{{{
18	18	>>> scipy.integrate.test()
…	…
33	33
34	34	== Setting up your module for testing ==
35		Every module or package requires two things for its testing to be included in the SciPy test suite.
36		1. A test module -- Every directory in the SciPy directory structure has a {{{tests/}}} sub-directory. The test module for {{{dir1/module_xxx.py}}} is named {{{dir1/tests/test_module_xxx.py}}}.
37		1. Two test routines -- Every module should have the functions {{{test()}}} and {{{test_suite()}}} defined. The first of these is used when running tests from the command line. The second returns a {{{unittest.TestSuite}}} object that can be used by other programs to test SciPy. This is beneficial, for instance, in generating web pages that show how healthly the current version of SciPy is.
	35	Every module or package requires two things for its testing to be included in the !SciPy test suite.
	36	1. A test module -- Every directory in the !SciPy directory structure has a {{{tests/}}} sub-directory. The test module for {{{dir1/module_xxx.py}}} is named {{{dir1/tests/test_module_xxx.py}}}.
	37	1. Two test routines -- Every module should have the functions {{{test()}}} and {{{test_suite()}}} defined. The first of these is used when running tests from the command line. The second returns a {{{unittest.TestSuite}}} object that can be used by other programs to test !SciPy. This is beneficial, for instance, in generating web pages that show how healthly the current version of SciPy is.
38	38
39	39	== Template for {{{test_module_xxx.py}}} ==
40		~~More complete documentation for how to use the unittest framework is found here. The following code will give you a basic idea of how to add unit tests to your module in~~ SciPy. Suppose you have a module {{{scipy/stats/foo.py}}} that has a function sum(a) that should accept a 1-D array or a scalar value. Create a test module called {{{scipy/stats/tests/test_foo.py}}}. This test file should include a class that tests {{{sum()}}} and a couple of other functions, {{{test_suite()}}} and {{{test()}}}. The test class has test methods that test various aspects of {{{sum()}}}. Within these test methods, {{{assert()}}} is used to test whether some case is true. If the assert fails, the test fails. Again, see here for more detailed info on defining test classes. The {{{test_suite()}}} function combines all the test classes that live in this module together to form a test suite that can be used for testing. The {{{test()}}} function actually runs the test suite. This file should look something like this:
	40	The following code will give you a basic idea of how to add unit tests to your module in !SciPy. Suppose you have a module {{{scipy/stats/foo.py}}} that has a function sum(a) that should accept a 1-D array or a scalar value. Create a test module called {{{scipy/stats/tests/test_foo.py}}}. This test file should include a class that tests {{{sum()}}} and a couple of other functions, {{{test_suite()}}} and {{{test()}}}. The test class has test methods that test various aspects of {{{sum()}}}. Within these test methods, {{{assert()}}} is used to test whether some case is true. If the assert fails, the test fails. Again, see here for more detailed info on defining test classes. The {{{test_suite()}}} function combines all the test classes that live in this module together to form a test suite that can be used for testing. The {{{test()}}} function actually runs the test suite. This file should look something like this:
41	41	{{{
42	42	...
…	…
76	76	}}}
77	77
78		Also, when invoking the entire SciPy test suite, your tests will be found and run:
	78	Also, when invoking the entire !SciPy test suite, your tests will be found and run:
79	79	{{{
80	80	>>> import scipy
…	…
84	84	Testing packages is a little more work, but not bad. See {{{scipy/common/__init__.py}}}.
85	85
86		== {{{~~scipy_test.py~~}}} ==
	86	== {{{numpy.testing}}} ==
87	87
88		This module holds a few helper routines. Several were discussed in the previous section. A couple of others are something of a verbose assert() function that give feedback as to what error occured. One of them, assert_array_equal(), is useful for comparing values in two arrays. The library of assert functions should grow to test for other general cases.
	88	This package holds a few helper routines. Several were discussed in the previous section. A couple of others are something of a verbose {{{assert()}}} function that give feedback as to what error occured. One of them, {{{assert_array_equal()}}}, is useful for comparing values in two arrays. The library of assert functions should grow to test for other general cases.
89	89
90	90	== Testing Wishes ==
…	…
92	92	* '''Speed testing:''' It'd be nice to have some speed testing facilities so that we could keep track of whether new code hurts or helps our speed. It would provide some simple benchmarking facilities also. To do this correctly, we'd need something like {{{start_timer()}}} and {{{stop_timer()}}} calls that could be placed in the {{{check_xxx()}}} methods. This would allow the set up and error checking code at the beginning and end of check methods to be ignored for timing purposes. If these functions weren't used in the check method, then the entire check method would be timed.
93	93	* '''Regression testing and database:''' If each test could be stored in a database, including timing information, that would allow us to see how speed is changing over time (using a reference machine).
94		* '''Web output of test results:''' Haven't looked much at the {{{TestRunner}}} stuff, but I imagine it is possible to grab info from the test suite and write it out to an HTML file. We should do this on several architectures every evening on the ~~scipy.org~~ site.
	94	* '''Web output of test results:''' Haven't looked much at the {{{TestRunner}}} stuff, but I imagine it is possible to grab info from the test suite and write it out to an HTML file. We should do this on several architectures every evening on the [www.scipy.org SciPy] site.