Please note:The SCons wiki is in read-only mode due to ongoing spam/DoS issues. Also, new account creation is currently disabled. We are looking into alternative wiki hosts.

Want to have "scons test" run your unit tests?

Here are two suggestions:

See http://spacepants.org/blog/scons-unit-test for another suggestion.

To have the process of adding unit test nicely encapsulated into an scons Tool, see the section below - "Unit Test integration with an scons Tool".

http://snappaction.blogspot.com/2007/02/scons-unit-testing-with-cxxtest-in.html shows a way to make adding UnitTests very simple by using CxxTest and automatically finding unit tests in a test directory.

Alias

   1 # Build one or more test runners.
   2 program = env.Program('test', 'TestMain.cpp')
   3 # Depend on the runner to ensure that it's built before running it.
   4 test_alias = Alias('test', [program], program[0].path)
   5 # Simply required.  Without it, 'test' is never considered out of date.
   6 AlwaysBuild(test_alias)

Check out PhonyTargets for another way of defining a 'test' target.

Note that program[0].path might give issues when running on OS'es that do not explicitly search for executables in the current directory (Unix-like OS'es where you explicitly need to add '.' as a search path). In that case, you can use the following:

   1 # Build one or more test runners.
   2 program = env.Program('test', 'TestMain.cpp')
   3 # Depend on the runner to ensure that it's built before running it - Note: using abspath.
   4 test_alias = Alias('test', [program], program[0].abspath)
   5 # Simply required.  Without it, 'test' is never considered out of date.
   6 AlwaysBuild(test_alias)

Command

Another idea is inspired on the boost build V2 system, that will create a file stamp if the unittest has run succesful. If it ran succesfull (exit code 0) and there is nothing changed, there is no need to run the unit test again.

   1 def runUnitTest(env,target,source):
   2    import subprocess
   3    app = str(source[0].abspath)
   4    if not subprocess.call(app):
   5       open(str(target[0]),'w').write("PASSED\n")
   6 program = env.Program('test', 'TestMain.cpp')
   7 Command("test.passed",'test',runUnitTest)

Note by Dov Grobgeld 2005-12-18

I modified the method mentioned above in order to be able to use it in a SConscript file without needing to defining runUnitTest in each SConscript file. Here is what I did:

In the SConstruct file:

   1 def builder_unit_test(target, source, env):
   2     app = str(source[0].abspath)
   3     if os.spawnl(os.P_WAIT, app, app)==0:
   4         open(str(target[0]),'w').write("PASSED\n")
   5     else:
   6         return 1
   7 # Create a builder for tests
   8 bld = Builder(action = builder_unit_test)
   9 env.Append(BUILDERS = {'Test' :  bld})

The test may then be declared in each of the library SConscript files by doing:

   1 Import('env')
   2 # Build the library
   3 :
   4 # Test the library
   5 test_lib = env.Program('test_library',
   6                                          ['test_library.cpp'])
   7 env.Test("test.passed", test_lib)
   8 env.Alias("test", "test.passed")

Now I am going to build a builder for automatically running valgrind/purify as well.

Note by Matt Doar 2007-01-24

I modified Dov's work to support comparing the results of running a test to a file containing the expected results. I also added an option to regenerate the expected results file. This is working nicely for us in practice.

In the SConstruct file, first add the boilerplate for the new regenerate option:

   1 # Add some command line options to SCons to support different build types.
   2 # Example of using an option: scons regenerate=1 ...
   3 command_line_options = Options()
   4 command_line_options.AddOptions(
   5     ('regenerate', 'Set to 1 to regenerate the expected results of unit tests', 0),
   6     )
   7 # The default build environment, used for all programs
   8 env = Environment(
   9     options = command_line_options,
  10     )
  11 # Generate the "scons --help" text for the options
  12 Help(command_line_options.GenerateHelpText(env))
  13 # Used in UnitTest
  14 env['REGENERATE'] = 0
  15 if str(ARGUMENTS.get('regenerate', 0)) == '1':
  16     env['REGENERATE'] = 1

Now add the UnitTest builder to the environment:

   1 import os
   2 def run(cmd, env):
   3     """Run a Unix command and return the exit code."""
   4     res = os.system(cmd)
   5     if (os.WIFEXITED(res)):
   6         code = os.WEXITSTATUS(res)
   7         return code
   8     # Assumes that if a process doesn't call exit, it was successful
   9     return 0
  10 def unit_test_emitter(target, source, env):
  11     base, ext = os.path.splitext(source[1].abspath)
  12     source.append(base + '.expected')
  13     return (target, source)
  14 def UnitTest(target, source, env):
  15     '''Run some app with an inputfile and compare the output with a .expected file
  16     containing the expected results.'''
  17     app = str(source[0].abspath)
  18     inputfile = str(source[1].abspath)
  19     base, ext = os.path.splitext(inputfile)
  20     expected = base + '.expected'
  21     # Output can come on both stdout and stderr
  22     cmd = app + ' ' + inputfile + ' 2>&1 | diff ' + expected + ' -'
  23     if env['REGENERATE'] == 1:
  24         print "Regenerating expected results file: " + expected
  25         cmd = app + ' ' + inputfile + ' &> ' + expected
  26     res = run(cmd, env)
  27     # If the test passed, create the target file so the test won't be run again
  28     if res == 0:
  29         open(str(target[0]),'w').write("PASSED\n")
  30     return res
  31 # Create a builder for running unit tests
  32 bld = Builder(action = Action(UnitTest, varlist = ['REGENERATE']), emitter = unit_test_emitter)
  33 env.Append(BUILDERS = {'UnitTest' :  bld})
  34 # NOTE: Only apply changes to env above here
  35 Export('env')

Using the new Builder in an SConscript file:

   1 Import('env')
   2 # removed the code to build myapp ...
   3 # Note: this test will look for a file named inputfile1.expected so you may have
   4 # to touch that file to bootstrap the creation of the test.
   5 mytest1 = env.UnitTest("tests/test1.passed", [myapp, 'inputfile1.txt'])
   6 Alias("mytest1", mytest1)

First generate the expected results file with "scons regenerate=1 mytest1". Then run the unit test with "scons mytest1".

Unit Test integration with an scons Tool

Section added by Chris Foster, 23-7-2007

I wanted to integrate unit testing into the aqsis scons build system, in a way which made it as easy as possible to add tests from our Sconstruct files. I ended up writing an scons Tool (see listing below) to encapsulate adding the appropriate things to an environment, building on Dov's work above.

The nice thing about this is that you can very cleanly create add a test environment which includes the tool, and add any libraries which you need to link with to that test building environment. Here's what a section of a main SConstruct file might look like, when using boost.test for testing:

   1 # make an initial construction environment
   2 env = Environment()
   3 Export('env')
   4 # Set up the test environment.  We copy the environment so that we can add the
   5 # extra libraries needed without messing up the environment for production
   6 # builds.
   7 #
   8 # Here we use boost.test as the unit testing framework.
   9 testEnv = env.Copy()
  10 testEnv.Tool('unittest',
  11                 toolpath=['build_tools'],
  12                 UTEST_MAIN_SRC=File('build_tools/boostautotestmain.cpp'),
  13                 LIBS=['boost_unit_test_framework']
  14         )
  15 Export('testEnv')
  16 # grab stuff from sub-directories.
  17 env.SConscript(dirs = ['onelib'])

In some sub-directory, onelib, you can then add tests quite easily, as follows:

   1 #-------------------------------------------------------------------------------
   2 # Unit tests
   3 Import('testEnv')
   4 testEnv = testEnv.Copy()
   5 testEnv.AppendUnique(LIBPATH=[env.Dir('.')], LIBS=['one'])
   6 testEnv.PrependENVPath('LD_LIBRARY_PATH', env.Dir('.').abspath)
   7 # We can add single file unit tests very easily.
   8 testEnv.addUnitTest('two_test.cpp')
   9 # also, multiple files can be compiled into a single test suite.
  10 libone_test_sources = Split('''
  11         one_test.cpp
  12         two_test.cpp
  13         ''')
  14 testEnv.addUnitTest('libone_test_all', libone_test_sources)
  15 # all the tests added above are automatically added to the 'test' alias.

Because the tool automatically adds Aliases, it's easy to run a particular test,

$ scons two_test

or the whole set of tests:

$ scons test

Here's the code for the tool:

   1 import os
   2 def unitTestAction(target, source, env):
   3         '''
   4         Action for a 'UnitTest' builder object.
   5         Runs the supplied executable, reporting failure to scons via the test exit
   6         status.
   7         When the test succeeds, the file target.passed is created to indicate that
   8         the test was successful and doesn't need running again unless dependencies
   9         change.
  10         '''
  11         app = str(source[0].abspath)
  12         if os.spawnle(os.P_WAIT, app, env['ENV'])==0:
  13                 open(str(target[0]),'w').write("PASSED\n")
  14         else:
  15                 return 1
  16 def unitTestActionString(target, source, env):
  17         '''
  18         Return output string which will be seen when running unit tests.
  19         '''
  20         return 'Running tests in ' + str(source[0])
  21 def addUnitTest(env, target=None, source=None, *args, **kwargs):
  22         '''
  23         Add a unit test
  24         Parameters:
  25                 target - If the target parameter is present, it is the name of the test
  26                                 executable
  27                 source - list of source files to create the test executable.
  28                 any additional parameters are passed along directly to env.Program().
  29         Returns:
  30                 The scons node for the unit test.
  31         Any additional files listed in the env['UTEST_MAIN_SRC'] build variable are
  32         also included in the source list.
  33         All tests added with addUnitTest can be run with the test alias:
  34                 "scons test"
  35         Any test can be run in isolation from other tests, using the name of the
  36         test executable provided in the target parameter:
  37                 "scons target"
  38         '''
  39         if source is None:
  40                 source = target
  41                 target = None
  42         source = [source, env['UTEST_MAIN_SRC']]
  43         program = env.Program(target, source, *args, **kwargs)
  44         utest = env.UnitTest(program)
  45         # add alias to run all unit tests.
  46         env.Alias('test', utest)
  47         # make an alias to run the test in isolation from the rest of the tests.
  48         env.Alias(str(program[0]), utest)
  49         return utest
  50 #-------------------------------------------------------------------------------
  51 # Functions used to initialize the unit test tool.
  52 def generate(env, UTEST_MAIN_SRC=[], LIBS=[]):
  53         env['BUILDERS']['UnitTest'] = env.Builder(
  54                         action = env.Action(unitTestAction, unitTestActionString),
  55                         suffix='.passed')
  56         env['UTEST_MAIN_SRC'] = UTEST_MAIN_SRC
  57         env.AppendUnique(LIBS=LIBS)
  58         # The following is a bit of a nasty hack to add a wrapper function for the
  59         # UnitTest builder, see http://www.scons.org/wiki/WrapperFunctions
  60         from SCons.Script.SConscript import SConsEnvironment
  61         SConsEnvironment.addUnitTest = addUnitTest
  62 def exists(env):
  63         return 1

scons check with CxxTest

While you can still use the code here, I have since created a CxxTest builder. see here: CxxTestBuilder

I struggled with CxxTest and scons for a while, and this is the closest thing to 'make check' I have been able to come. It's quite close, I believe, and I tried to minimize the amount of code it took.

I started with suggestions from here: http://spacepants.org/blog/scons-unit-test, and modified the general idea somewhat for it to work with the CxxTest c++ test framework http://cxxtest.sourceforge.net/.

Since I am quite new to scons, I wasn't able to figure out how exactly to put my extensions into a separate file to be sourced by scons automatically, and I hope someone can supply that knowledge.

This also does not support all CxxTest functionality. I only built in what I required, but the result is neat and simple.

Without further ado, here is the code from my SConstruct:

   1 from SCons.Script.SConscript import SConsEnvironment
   2 env = Environment()
   3 # required for the cxxbuilder.
   4 # If you use the normal header files, just use .h here.
   5 env['TEST_SUFFIX'] = '.t.h'
   6 # ----------------------------------
   7 # cxx test builder
   8 # ----------------------------------
   9 CxxTestCpp_bld = Builder(
  10     action = "./cxxtestgen.py --error-printer -o $TARGET $SOURCE",
  11     suffix = ".cpp",
  12     src_suffix = '$TEST_SUFFIX'
  13     )
  14 env['BUILDERS']['CxxTestCpp'] = CxxTestCpp_bld
  15 # ----------------------------------
  16 # UnitTest function - a wrapper around
  17 # the Program call that adds the result
  18 # of the build to the tests-to-run target.
  19 # ----------------------------------
  20 def UnitTest(environ, target, source = [], **kwargs):
  21     test = environ.Program(target, source = source, **kwargs)
  22     environ.AlwaysBuild('check')
  23     environ.Alias('check', test, test[0].abspath)
  24     return test
  25 SConsEnvironment.UnitTest = UnitTest
  26 # ----------------------------------
  27 # A wrapper that supplies the multipart
  28 # build functionality CxxTest requires.
  29 # ----------------------------------
  30 def CxxTest(environ, target, source = [], **kwargs):
  31     if (source == []):
  32         source = Split(target + environ['TEST_SUFFIX'])
  33     sources = Split(source)
  34     sources[0] = environ.CxxTestCpp(sources[0])
  35     return environ.UnitTest(target, source = sources, **kwargs)
  36 SConsEnvironment.CxxTest = CxxTest

Usage

The function is modelled to be called as the Program() call is:

env.CxxTest('target_name') will build the test from the source target_name + env['TEST_SUFFIX'],

env.CxxTest('target_name', source = 'test_src.t.h') will build the test from test_src.t.h source,

env.CxxTest('target_name, source = ['test_src.t.h', other_srcs] builds the test .cpp from source[0] and passes other sources to the Program call verbatim.

You may also add additional arguments to the function. In that case, they will be passed to the actual Program builder call unmodified. Convenient for passing different CPPPATHs and the sort.

Anyway, this is the way I call it:

   1 # #/src/test/SConscript
   2 Import('env')
   3 env['CPPPATH'] = '#' # CxxTest headers are in #/cxxtest/
   4 env.CxxTest('test_quaternion', source = 'Quaternion.t.h')
   5 env.CxxTest('test_utility', ['utility.t.h', '../utility.cpp'])

I run the tests by typing 'scons check'.

The tests do not compile by scons . (which is identical to the behaviour of make check)

If the tests are out of date, they compile - scons dependency tracking works.

Any suggestions, improvements and/or criticism are welcome. As I said, I am new to scons.

Cheers, GasperAzman

-- Comment on Gasper's code by Matt Doar:

Just what I wanted, and nicely done, thank you. However, I think that the last line in the CxxTest function should be

return environ.UnitTest(target, source = sources, **kwargs)

instead of

return env.UnitTest(target, source = sources, **kwargs)

to make sure that the correct env is propagated to the Program.

-- Thanks Matt, good spot. It worked in my code due to moonphase. I corrected the code above. Thanks, GasperAzman

SConstructs: unit and functional testing

I'm developing quite a complex build process. To have "scons test" isn't an issue for me. Instead, I'm concenrating on checking established internal dependencies, such as:

Here are blog entries (probably should be copied here instead of linking):

And here are some hints on functional (integration) testing (running scons and checking that the result is as expected):

UnitTests (last edited 2008-10-27 06:44:58 by AnomalousUnderdog)