Testing
=======

There are two directories in the PyTrilinos package that provide
python scripts for testing PyTrilinos, ``example`` and ``test``.
Generally speaking, unit tests go in the ``test`` directory and
descriptive or demonstrative example scripts go in ``example``.

Naming Conventions
------------------

Unit tests scripts shall begin with ``test``, followed by the package
name and an underscore, followed by a short description of the test,
typically the class name being tested.  For example::

    testTeuchos_ParameterList
    testEpetra_MultiVector

are the base names of the unit tests for ``Teuchos.ParameterList`` and
``Epetra.MultiVector``, respectively.  In certain situations, the
underscore and test description can be omitted.

Example scripts shall begin with ``ex``, followed by the name of the
primary package being demonstrated, followed by an underscore and a
short description of the test.  For example::

  exEpetra_Comm
  exAztecOO_Operator

are the base names for example scripts demonstrating the Epetra.Comm
class and an AztecOO solver that uses an Epetra.Operator,
respectively.  In certain situations, the underscore and test
description can be omitted.

Build System
------------

Running ``make`` in either the top, ``example`` or ``test`` build
directories copies the test scripts from the source directory to the
build directory while performing certain text substitutions.  Similar
to the configuration file naming convention, the source files have the
suffix ``.py.in`` and the build files have the suffix ``.py``.  The
reason for this is that CMake variable values can be substituted
during the copy procedure.  For example, the first line of each test
script source file is now::

  #! ${Python3_EXECUTABLE}

which means that the python executable that is recognized (and
compiled against) by the CMake build system will also be the python
executable invoked by the PyTrilinos test scripts.  Note that this
substitution is available to the test developers for any variable that
is defined in the ``CMakeCache.txt`` file found in the top build
directory.

The ``CMakeLists.txt`` files in the ``test`` and ``example``
directories control which scripts get copied to the build directory.
Each test script, or group of related test scripts, should be
protected with ``IF()`` statements, depending on which PyTrilinos
modules need to be present for the script to run.

Running All Tests
-----------------

To run all of the enabled tests, first make sure PyTrilinos and all
tests and examples are up-to-date::

    $ cd BUILD/packages/PyTrilinos
    $ make

Then you can use the CMake ``ctest`` program to run all of the tests::

    $ ctest -W 10

The results of all of the tests can be found in the ``Testing``
directory, present in the directory from which ``ctest`` was run.

Test Script Conventions
-----------------------

All test scripts shall use either the ``optparse`` module (deprecated,
but used historically in PyTrilinos), or the ``argparse`` module to
parse command line options and support the following options::

  -t, --testharness     test local build modules; prevent loading
                        system-installed modules
  -v VERBOSITY, --verbosity=VERBOSITY
                        set the verbosity level [default 2]

The ``-t`` option is to force use of the locally-built PyTrilinos,
preventing the importing of any installed PyTrilinos modules.  The
verbosity option is used in all ``test`` scripts and optionally in any
``example`` scripts where it makes sense.

Tests scripts use the ``from PyTrilinosImport`` function in the
``testutil`` module, local to both ``test`` and ``example``
directories, to control where the PyTrilinos modules are imported
from.  The user controls this import location from the command line:
``-t`` or ``--testharness`` indicates that the build directory should
be used; otherwise an import from standard locations will be
attempted.

This policy is enabled by code in each test/example script like the
following::

    parser = OptionParser()
    parser.add_option("-t", "--testharness", action="store_true",
                      dest="testharness", default=False,
                      help="test local build modules; prevent loading system-installed modules")
    options,args = parser.parse_args()
    from testutil import fromPyTrilinosImport
    Teuchos = fromPyTrilinosImport('Teuchos', options.testharness)
    Epetra  = fromPyTrilinosImport('Epetra' , options.testharness)

If the user specifies ``-t`` or ``--testharness`` then
``options.testharness`` will be True, else it will be False.  When
``fromPyTrilinosImport()`` is called, the ``options.testharness``
argument will determine where the import is read from.

Test scripts shall run in both serial or parallel.  You may use
either::

    comm = Teuchos.DefaultComm.getComm()

or::

    comm = Epetra.PyComm()

to obtain an appropriate communicator object for the test scripts.  By
convention, set the variable ``iAmRoot`` to either ``True`` or
``False`` depending on whether the communicator's rank is 0.

The test script shall output ``End Result: TEST PASSED`` if the test
passes correctly.  This helps the Trilinos test harness determine
which tests pass and which do not, especially in parallel.

Unit Tests
----------

Unit tests are based on the ``unittest`` python library module.  Test
case classes inherit from ``unittest.TestCase``.  Individual tests are
implemented as methods of these classes that begin with ``test``.  See
the python documentation (http://www.python.org) for details.

Each unit test script can have one or more ``TestCase`` classes.  In
``main()``, each test case class should be added to a
``unittest.TestSuite`` object named ``suite``.

Unit tests shall print, from processor 0, a header with a message such
as "Testing Epetra.Object" with a line of asterisks above and below
the message::

    *********************
    Testing Epetra.Object
    *********************

The suite of tests should be run with::

    verbosity = options.verbosity * int(iAmRoot)
    result = unittest.TextTestRunner(verbosity=verbosity).run(suite)

and the success should be determined and output via (for the case of
an Epetra communicator)::

    errsPlusFails = comm.SumAll(len(result.errors) + len(result.failures))
    if errsPlusFails == 0 and iAmRoot: print "End Result: TEST PASSED"
    sys.exit(errsPlusFails)

If there are no errors and no failures on any processors, then the
test will pass.

Example Scripts
---------------

Example scripts are more free-form and should be written for
readability, to make for clear demonstrations of PyTrilinos usage.
However, it is encouraged that example script output be consistent
with unit test output whenever possible.
