Testing Modules in Chainer

Introduction

Thanks to many contributors, Chainer has been developed greatly. But as we review the PRs, we noticed there are limited guidelines on how to test the implementation. So those who tries to send PR must make test cases by imitating the existing test cases or read the code directly. In this article, I will briefly review the testing modules in Chainer and CuPy. As Chainer's testing modules are subset of CuPy's ones, I will focus on CuPy.

CuPy's (resp. Chainer's) testing tools are located in cupy.testing (resp. chainer.testing) and consists of the following modules (* indicates that Chainer also has this module in chainer.testing) :

• array
• attr (*)
• condition (*)
• helper
• hypothesis
• parameterized (*)

Note that this article is based on v1.5.0.3

array module

cupy.testing.array module implements NumPy-like assertion functions. These functions accept both NumPy and CuPy ndarrays. The implemented assertions are as follows:

• assert_arrays_almost_equal_nulp
• assert_array_max_ulp
• assert_array_equal
• assert_array_list_equal
• assert_array_less

These assertions are used in NumPy-CuPy consistency check decorators explained in helper module. As we use this decorators more often, we do not expect testers use these assertions directly (of course we can choose to use them).

attr module

cupy.testing.attr contains several decorators that enable/disable test cases and test fixtures¹.

• @attr.gpu specifies that the test uses single GPU.
• @attr.multi_gpu(N) specifies this test requires N GPUs. @attr.gpu is equivalent to @attr.multi_gpu(1)
• @attr.cudnn specifies this test uses cuDNN module.

For more details, please see the testing guideline.

hypothesis module

cupy.testing.hypothesis module contains hypothesis testing tools, to test statistical behaviors. For now, it has simple goodness-of-fit test with Peason's Chi-squared test only. We use it for testing random generator of ints like cupy.randint or cupy.random_integers.

parameterized module

cupy.testing.parameterized module offers the standard way of parameterized tests. Basic usage is as follows.

usage_of_parameterized_module.py

@testing.parameterize(
    {'height': 150, 'weight': 45},
    {'height': 180, 'weight': 80})
class BMITest(unittest.TestCase):
    def test_bmi(self):
        self.assertLessEqual(
             calculate_bmi(self.height, self.weight), 25.0)

This test calculates BMI(Body Mass Index) based on the height and weight and checks if it is less than the threshold. height and weight are parameters in this test. We can access them as attributes of the test case.

parameterized decorator automatically generates the test case for each set of parameters. The naming convention of generated test cases is <original class name>_param_<n> where <n> is the index number of the set parameters. In this example, BMITest_param_0 and BMITest_param_1 are generated. Note that original test case (BMITest in the example) is not executed.

We have an utility that makes the product set of parameter set. For example, testing.product({‘a': [1, 2]}, {‘b’: [3, 4]}) is equivalent to [{‘a’: 1, ‘b’: 3}, {‘a’: 1, ‘b’: 4}, {‘a’: 2, ‘b’: 3}, {‘a’: 2, ‘b’: 4}]

condition module

Decorators in cupy.testing.condition module customize the condition test fixtures are regarded as "success". For now, we have decorators for running test fixtures multiple times.

• @attr.retry(N) tries the decorated fixture N times and considers success if at least one of the trial is successful.
• @attr.repeat(N) tries the decorated fixture N times and considers success if all trials are successful.

Decorators abort the trials if we can judge the final result before we do execute remaining trials. If the decorated test fixture is considered failed, it shows the number of failed and successful trials and error message of first failed trial.

helper module

cupy.testing.helper consists of some utility decorators for test cases and fixtures. Currently there are two types of decorators, namely, NumPy-CuPy consistency check and parameterized dtype test²

NumPy-CuPy consistency check

cupy.ndarray is designed so that most of its API (method name and arguments) is identical to corresponding ones in numpy.ndarray. NumPy-CuPy consistency check decorators offer easy way to check the consistency of these APIs.

Take testing.numpy_cupy_allclose decorator for example. The typical usage is as follows (we modified slightly from the original code):

usage_of_numpy_cupy_consistency_decorator.py

@testing.gpu
class TestFoo(unittest.TestCase):

    @testing.numpy_cupy_allclose()
    def test_mean_all(self, xp):
        a = testing.shaped_arange((2, 3), xp)
        return a.mean()

This test fixture checks numpy.mean() and cupy.mean() should be the same result. xp is an additional argument inserted by the decorator. It takes either numpy or cupy. Decorated function is required to return the same value³ even if xp is numpy or cupy. We can change the argument name from xp by name argument.

Parameterized dtype test

This kind of decorators makes decorated test fixture parameterized with respect to dtype. Of course, we can do the parameterized test with respect to dtype with @testing.parameterized decorator. But parameterized dtype test decorators offer easier way. Let's look at the example in CuPy test code.

tests/cupy_tests/io_tests/test_npz.py

@testing.gpu
class TestNpz(unittest.TestCase):
...
    @testing.for_all_dtypes()
    def test_pickle(self, dtype):
        a = testing.shaped_arange((2, 3, 4), dtype=dtype)
        s = six.moves.cPickle.dumps(a)
        b = six.moves.cPickle.loads(s)
        testing.assert_array_equal(a, b)

This test fixture checks if cPickle successfully reconstructs cupy.ndarray for various dtypes. dtype is an argument inserted by the decorator as with the NumPy-CuPy consistency check decorator's case. We can change the argument name by name argument of the decorator.

Here is the correspondence table of decorators and dtypes to be checked.

||bool|float[16, 32, 64]|int[8, 16, 32, 64]|uint[8, 16, 32, 64]|
|-----|:-----:|:-----:|:-----:|:-----:|:-----:|
|for_all_dtypes|○|○|○|○|
|for_float_dtypes||○|||
|for_signed_dtypes|||○||
|for_unsigned_dtypes||||○|
|for_int_dtypes|○||○|○|

numpy.bool_ and numpy.float16 are optional. If no_float16 (resp. no_bool) option set True, numpy.float16 (resp. numpy.bool_ ) is disabled.

combinatorial dtype test

Some test fixtures require parameterization with respect to the product of dtypes. Decorators named as for_***_dtypes_combination offer this functionality.

usage_of_combinatorial_dtype_test.py

@testing.gpu
class TestFoo(unittest.TestCase):
    @testing.for_all_dtypes_combination(dtypes=['a_type', 'b_type'], full=True)
    def test_foo(self, dtype):
         a = cupy.arange(10, dtype=a_type)
         b = cupy.arange(20, dtype=b_type)
         # (some assertions with a and b)

Let N be the number of dtypes and M be the number of values each dtype can take. This decorator exexutes N**M tests if full option is set True. In some case, this can be costly, so we only check selected M tests only if full is False. If full argument is not specified, the default behavior depends on the environment variable CUPY_TEST_FULL_COMBINATION.

We can (and in most case do) use both of NumPy-CuPy consistency check decorator and Parameterized dtype test decorator simultaneously. Here is the example of unittests for cupy.mean

tests/cupy_tests/statics_tests/test_meanvar.py

    @testing.for_all_dtypes()
    @testing.numpy_cupy_allclose()
    def test_mean_all(self, xp, dtype):
        a = testing.shaped_arange((2, 3), xp, dtype)
        return a.mean()

Note that for implementation reason, Parameterized dtype decorators must be preceded by NumPy-CuPy consistency check decorators.

Conclusion

As we have seen in this article, Chainer and CuPy have various kind of utility that make the test cases easier. We briefly review the function and basic usage of them. I hope this article helps to promote the user to contribute to Chainer.

After we finish writing the article, I noticed that I forget to write about chainer.gradient_check, which is one of the most important testing tool in Chainer. I will write it in another article (or I hope someone write about it).

This article resolves issue #714 :)

1. In this article, test case denotes the subclass of unittest.TestCase, while test fixture the method of the test case. ↩

2. These names are not official. ↩

3. Of course, the type of array module(numpy/cupy) is different the decorator checks cupy.asnumpy(cupy_result) is equivalent to numpy's result. ↩