SCons Testing Methodology
Contents
SCons uses extensive automated tests to try to ensure quality. The primary goal is that users should be able to upgrade from version to version without any surprise changes in behavior.
In general, no change goes into SCons unless it has one or more new or modified tests that demonstrably exercise the bug being fixed or the feature being added. There are exceptions to this guideline, but they should be just that, exceptions. When in doubt, make sure it's tested.
Test Organization
There are two types of SCons tests:
End-to-End Tests
End-to-end tests of SCons are all Python scripts (*.py) underneath the test/ subdirectory. They use the test infrastructure modules in the QMTest subdirectory.
Unit Tests
Unit tests for individual SCons modules live underneath the src/engine/ subdirectory and are the same base name as the module with Tests.py appended--for example, the unit tests for the Builder.py module are in the BuilderTests.py script.
Contrasting End-to-End and Unit Tests
In general, anything that we've put into an end-to-end test script should be considered a hardened part of the interface (that is, it's something that a user might do) and should not be broken. Unit tests are now considered more malleable, more for testing internal interfaces that can change so long as we don't break users' SConscript files. (This wasn't always the case, and there's a lot of meaty code in many of the unit test scripts that does, in fact, capture external interface behavior. In general, we should try to move those things to end-to-end scripts as we find them.)
It's more difficult to debug end-to-end tests. You can actually go straight into the Python debugger on the unit test scripts by using the runtest.py --pdb option, but the end-to-end tests treat an SCons invocation as a "black box" and just look for external effects. Simple print statements within the SCons code itself often don't help debug end-to-end because they end up in SCons output that gets compared against expected output and cause a test failure. Probably the most effective technique is to use the internal SCons.Debug.Trace() function, which prints output to /dev/tty on Linux/UNIX systems and con on Windows systems, so you can see what's going on.
Running Tests
Tests are run from the top-level directory by the runtest.py script. If the QMTest package is installed on your system, the runtest.py script will use it. If not, it will fall back to executing tests directly. There is a --noqmtest option that will force direct execution of tests even if QMTest is installed.
Help is available through the -h option:
$ python runtest.py -h
To run all the tests, use the -a option:
$ python runtest.py -a
You may specifically list one or more tests to be run:
$ python runtest.py src/engine/SCons/BuilderTests.py $ python runtest.py test/option-j.py test/Program.py
You also use the -f option to execute just the tests listed in a specified text file:
$ cat testlist.txt test/option-j.py test/Program.py $ python runtest.py -f testlist.txt
One test must be listed per line, and any lines that begin with '#' will be ignored (the intend being to allow you, for example, to comment out tests that are currently passing and then uncomment all of the tests in the file for a final validation run).
If more than one test is run, the runtest.py script prints a summary of how many tests passed, failed, or yielded no result, and lists any unsuccessful tests.
The above invocations all test directly the files underneath the src/ subdirectory, and do not require that a packaging build be performed first. The runtest.py script supports additional options to run tests against unpacked packages in the build/test-*/ subdirectories.
"Hello, world!" SCons Test Script
To illustrate how the end-to-end test scripts work, let's walk through a simple "Hello, world!" example:
1 import TestSCons
2
3 test = TestSCons.TestSCons()
4
5 test.write('SConstruct', """\
6 Program('hello.c')
7 """)
8
9 test.write('hello.c', """\
10 int
11 main(int argc, char *argv[])
12 {
13 printf("Hello, world!\\n");
14 exit (0);
15 }
16 """)
17
18 test.run()
19
20 test.run(program='./hello', stdout="Hello, world!\n")
21
22 test.pass_test()
import TestSCons: Imports the main infrastructure for writing SCons tests. This is normally the only part of the infrastructure that needs importing. Sometimes other Python modules are necessary or helpful, and get imported before this line.
test = TestSCons.TestSCons(): This initializes an object for testing. A fair amount happens under the covers when the object is created, including:
- A temporary directory is created for all the in-line files that will get created.
- The temporary directory'ys removal is arranged for when the test is finished.
We os.chdir() to the temporary directory.
test.write('SConstruct', ...: This line creates an SConstruct file in the temporary directory, to be used as input to the scons run(s) that we're testing. Note the use of the Python triple-quote syntax for the contents of the SConstruct file. Because input files for tests are all created from in-line data like this, the tests can sometimes get a little confusing to read, because some of the Python code is found
test.write('hello.c', ...: This lines creates an hello.c file in the temporary directory. Note that we have to escape the \\n in the "Hello, world!\\n" string so that it ends up as a single backslash in the hello.c file on disk.
test.run(): This actually runs SCons. Like the object initialization, things happen under the covers:
- The exit status is verified; the test exits with a failure if the exit status is not zero.
- The error output is examined, and the test exits with a failure if there is any
test.run(program='./hello', stdout="Hello, world!\n"): This shows use of the TestSCons.run() method to execute a program other than scons, in this case the hello program we just presumably built. The stdout= keyword argument also tells the TestSCons.run() method to if if the program output does not match the expected string "Hello, world!\n". Like the previous test.run() line, it will also fail the test if the exit status is non-zero, or there is any error output.
test.pass_test(): This is always the last line in a test script. It prints PASSED on the screen and makes sure we exit with a 0 status to indicate the test passed. As a side effect of destroy the test object, the created temporary directory will be removed.
Test Infrastructure
The test API is in QMTest/TestSCons.py. TestSCons is a subclass of TestCommon, which is a subclass of TestCmd; all those python files are in QMTest. Start in QMTest/TestCmd.py for the base API definitions, like how to create files (test.write()) and run commands (test.run()).
The match functions work like this:
TestSCons.match_re:: match each line with a RE
- Splits the lines into a list (unless they already are)
- splits the REs at newlines (unless already a list) and puts ^..$ around each
- then each RE must match each line. This means there must be as many REs as lines.
TestSCons.match_re_dotall:: match all the lines against a single RE
- Joins the lines with newline (unless already a string)
- joins the REs with newline (unless it's a string) and puts ^..$ around the whole thing
- then whole thing must match with python re.DOTALL.
Use them in a test like this:
test.run(..., match=TestSCons.match_re, ...)
or:
test.must_match(..., match=TestSCons.match_re, ...)
Avoiding Tests based on Tool existence
Here's an easy sample:
See QMTest/TestSCons.py for the detect_tool method. It calls the tool's generate() method, and then looks for the given prog (tool name by default) in env['ENV']['PATH'].
See also DevelopingTests for more info on writing and debugging SCons tests.
The Test Methods Themselves
(This content is also provided in a page by itself to make it easier to search.)
These are the classes in the test infrastructure and their methods. Each class described below is located in a module by the same name:
from TestCmd import TestCmd
If a class is derived from another class, the superclass is given in parenthesis.
xxx At the moment, only limited documentation is present; basically, this is just something I can search when I need a particular function. In time, descriptions will be added for each method. There may be methods missing that need to be added and there may be internal routines present that need to be removed. TLC is needed until this becomes actual documentation. This is intended to be "quick-look" documentation and detailed documentation (including the vaunted examples) would be linked from the description.
TestCmd
Access to test directory. Create pathnames within the test directory. Write (and read) test files. Run a command and capture stdout and stderr. Miscellaneous operating system and file system operations. Succeed, fail, and skip test.
Initialize and tear down.
- TestCmd.__init__(self, description = None, program = None, interpreter = None, workdir = None, subdir = None, verbose = None, match = None, combine = 0, universal_newlines = 1)
- TestCmd.description_set(self, description)
- TestCmd.program_set(self, program)
- String or list of strings. The command to be tested and its arguments.
- TestCmd.interpreter_set(self, interpreter)
- String or list of strings. If the command to be tested is a script, the interpreter and any arguments.
- TestCmd.workdir_set(self, path)
Creates a temporary working directory. if path is None or '' a unique name is created.
- cnt = TestCmd.subdir(self, *subdirs)
- Create subdirectories under the current working directory. Errors are ignored; the return value is the number of subdirectories actually created.
- TestCmd.verbose_set(self, verbose)
- TestCmd.preserve(self, *conditions)
If the test terminates due to one of the conditions, preserve the temporary test directory. The conditions can be any or all of 'pass_test', 'fail_test', or 'no_result'; no conditions preserves the directory for all cases.
- TestCmd.cleanup(self, condition = None)
Skip, fail, succeed. One of these must be executed to terminate the test run.
- TestCmd.no_result(self, condition = 1, function = None, skip = 0)
Reports NO RESULT from the test if the condition is true. If the test is failing, runs function with no arguments to clean up. The first skip stack frames are not displayed; set it to a large number to suppress the display completely.
- TestCmd.fail_test(self, condition = 1, function = None, skip = 0)
Fails the test if the condition is true. If the test is failing, runs function with no arguments to clean up. The first skip stack frames are not displayed; set it to a large number to suppress the display completely.
- TestCmd.pass_test(self, condition = 1, function = None)
Passes the test if the condition is true. If the test is succeeding, runs function with no arguments.
Workspace. The canonicalize() and workpath() methods appear to be equivalent; I don't know why both exist.
- TestCmd.canonicalize(self, path)
Returns the full path within the working directory. The path argument may either be a string or a list of strings representing path elements.
- TestCmd.workpath(self, *args)
- Returns the full path within the working directory. The arguments are strings of path elements.
- TestCmd.write(self, file, content, mode = 'wb')
Write content (a string) to the workspace file.
- TestCmd.read(self, file, mode = 'rb')
- Return the content of the workspace file.
Run command.
- cmd_list = TestCmd.command_args(self, program = None, interpreter = None, arguments = None)
Constructs a command list. If program or interpreter are not specified, the corresponding default is used.
- popen = TestCmd.start(self, program = None, interpreter = None, arguments = None, universal_newlines = None, **kw)
- TestCmd.finish(self, popen, **kw)
- TestCmd.run(self, program = None, interpreter = None, arguments = None, chdir = None, stdin = None, universal_newlines = None)
Test results.
- content = TestCmd.stdout(self, run = None)
Return the standard output from the specified run. If run is None, the latest run is returned.
- content = TestCmd.stderr(self, run = None)
Return the standard error from the specified run. If run is None, the latest run is returned.
- bool = TestCmd.match(self, lines, matches)
Either match_exact or match_re depending on the match argument when the class was initialized.
- bool = TestCmd.match_exact(self, lines, matches)
Returns true if lines and matches are the same and false otherwise. Both lines and matches can be either a string or a list of strings.
- bool = TestCmd.match_re(self, lines, res)
Returns true if lines matches the regular expressions in res. Both lines and res can be either a string or a list of strings.
- bool = TestCmd.match_re_dotall(self, lines, res)
Same as match_re except that the regular expressions are compiled with re.DOTALL.
- module.diff_re(a, b, fromfile='', tofile='', fromfiledate='', tofiledate='', n=3, lineterm='\n')
A quick-and-dirty line-by-line diff. Both a and b must be lists of strings; lines in a are compared with regular expressions in b. The keyword arguments are all ignored.
Miscellaneous.
- path = TestCmd.where_is(self, file, path=None, pathext=None)
- path = TestCmd.tempdir(self, path=None)
Creates a temporary directory and arranges to remove it when the test completes. A path of None causes a unique directory to be created. Note that the path is not canonicalized.
- TestCmd.chmod(self, path, mode)
- TestCmd.symlink(self, target, link)
- TestCmd.touch(self, path, mtime=None)
- TestCmd.unlink(self, file)
- TestCmd.sleep(self, seconds = default_sleep_seconds)
- TestCmd.rmdir(self, dir)
- TestCmd.readable(self, top, read=1)
- TestCmd.writable(self, top, write=1)
- TestCmd.executable(self, top, execute=1)
TestCommon(TestCmd)
Extension of TestCmd.
Module variables.
- module.python_executable
- module.exe_suffix
- module.obj_suffix
- module.shobj_prefix
- module.shobj_suffix
- module.lib_prefix
- module.lib_suffix
- module.dll_prefix
- module.dll_suffix
Uncategorized.
- TestCommon.__init__(self, **kw)
- TestCommon.banner(self, s, width=None)
- TestCommon.diff(self, a, b, name, *args, **kw)
- TestCommon.skip_test(self, message="Skipping test.\n")
- Print the message. If TESTCOMMON_PASS_SKIPS is set in the (shell) environment, pass the test, otherwise say no result.
File status.
- TestCommon.must_exist(self, *files)
- Fail test if file(s) do not exist.
- TestCommon.must_not_exist(self, *files)
- Fail test if file(s) exist.
- TestCommon.must_be_writable(self, *files)
- Fail test if file(s) do not exist or cannot be written.
- TestCommon.must_not_be_writable(self, *files)
- Fail test if file(s) do not exist or can be written.
File contents.
- TestCommon.must_match(self, file, expect, mode = 'rb')
Ensure that the file contents exactly match the expect parameter.
- TestCommon.must_contain(self, file, required, mode = 'rb')
Ensure that the file contains the required text somewhere within it.
- TestCommon.must_contain_all_lines(self, output, lines, title=None, find=None)
Ensure that all of the lines (a list of strings) appear in output. The title appears above any failure output. The find function returns true if the lie (second arg) is in the output (first arg).
- TestCommon.must_contain_any_line(self, output, lines, title=None, find=None)
Ensure that output contains at least one of the lines specified. The `ti tle appears above any failure output. The find` function returns true if the lie (second arg) is in the output (first arg).
- TestCommon.must_contain_lines(self, lines, output, title=None)
- Deprecated.
- TestCommon.must_not_contain_any_line(self, output, lines, title=None, find=None)
- TestCommon.must_not_contain_lines(self, lines, output, title=None)
Overrides.
- popen = TestCommon.start(self, program = None, interpreter = None, arguments = None, universal_newlines = None, **kw)
- TestCommon.finish(self, popen, stdout = None, stderr = '', status = 0, **kw)
- TestCommon.run(self, options = None, arguments = None, stdout = None, stderr = '', status = 0, **kw)
TestSCons(TestCommon)
Specialized routines with SCons-specific knowledge for running SConstructs.
Module variables and functions.
- module.default_version
- module.copyright_years
- module.SConsVersion
- module.machine
- module.python
- = python_executable
- module._python_
- = '"' + python_executable + '"'
- module._exe
- = exe_suffix
- module._obj
- = obj_suffix
- module._shobj
- = shobj_suffix
- module.shobj_
- = shobj_prefix
- module._lib
- = lib_suffix
- module.lib_
- = lib_prefix
- module._dll
- = dll_suffix
- module.dll_
- = dll_prefix
- module.re_escape(str)
- module.python_version_string()
- module.python_minor_version_string()
- module.unsupported_python_version(version=sys.version_info)
- module.deprecated_python_version(version=sys.version_info)
- module.deprecated_python_expr
Uncategorized.
- TestSCons.scons_version
- = SConsVersion
- TestSCons.get_python_version(self)
Deprecated, use module.python_minor_version_string().
- TestSCons.__init__(self, **kw)
- TestSCons.Environment(self, ENV=None, *args, **kw)
- TestSCons.detect(self, var, prog=None, ENV=None, norm=None)
- TestSCons.detect_tool(self, tool, prog=None, ENV=None)
- TestSCons.where_is(self, prog, path=None)
- TestSCons.wrap_stdout(self, build_str = "", read_str = "", error = 0, cleaning = 0)
- TestSCons.up_to_date(self, options = None, arguments = None, read_str = "", **kw)
- TestSCons.not_up_to_date(self, options = None, arguments = None, **kw)
- msg = TestSCons.diff_substr(self, expect, actual, prelen=20, postlen=40)
Return a string displaying the first mismatched character between two strings (expect and actual) that are known to be different.
- TestSCons.python_file_line(self, file, line)
- TestSCons.normalize_pdf(self, s)
- paths = TestSCons.paths(self, patterns)
Expand a list of patterns into a list of matching pathnames.
- TestSCons.wait_for(self, fname, timeout=10.0, popen=None)
Wait up to timeout seconds for fname to exist. If popen is given, terminate it by closing the standard input.
- TestSCons.get_alt_cpp_suffix(self)
- Return altername C++ suffix based on whether the filesystem is case sensitive.
- TestSCons.checkLogAndStdout(self, checks, results, cached, logfile, sconf_dir, sconstruct, doCheckLog=1, doCheckStdout=1)
Java.
- TestSCons.java_ENV(self, version=None)
- TestSCons.java_where_includes(self,version=None)
- TestSCons.java_where_java_home(self,version=None)
- TestSCons.java_where_jar(self, version=None)
- TestSCons.java_where_java(self, version=None)
- TestSCons.java_where_javac(self, version=None)
- TestSCons.java_where_javah(self, version=None)
- TestSCons.java_where_rmic(self, version=None)
Qt.
- TestSCons.Qt_dummy_installation(self, dir='qt')
- TestSCons.Qt_create_SConstruct(self, place)
SWIG.
- TestSCons.get_platform_python_info(self)
Skips test if there is no 'python' in the path. Returns pathname of Python executable, pathname of Python include directory (for CPPPATH), pathname of Python library directory (for LIBPATH), and the library name (for LIBS).
TestSConsign(TestSCons)
- TestSConsign.__init__(self, *args, **kw)
- TestSConsign.script_path(self, script)
- TestSConsign.set_sconsign(self, sconsign)
- TestSConsign.run_sconsign(self, *args, **kw)
TestSConsMSVS(TestSCons)
- TestSConsMSVS.msvs_versions(self)
- TestSConsMSVS.vcproj_sys_path(self, fname)
- TestSConsMSVS.msvs_substitute(self, input, msvs_ver, subdir=None, sconscript=None, python=None, project_guid=None)
- TestSConsMSVS.get_msvs_executable(self, version)
TestRuntest(TestCommon)
- TestRuntest.__init__(self, **kw)
- TestRuntest.write_fake_scons_source_tree(self)
- TestRuntest.write_failing_test(self, name)
- TestRuntest.write_no_result_test(self, name)
- TestRuntest.write_passing_test(self, name)
TestSCons_time(TestCommon)
- TestSCons_time.__init__(self, **kw)
- TestSCons_time.archive_split(self, path)
- TestSCons_time.fake_logfile(self, logfile_name, index=0)
- TestSCons_time.profile_data(self, profile_name, python_name, call, body)
- TestSCons_time.tempdir_re(self, *args)
- TestSCons_time.write_fake_aegis_py(self, name)
- TestSCons_time.write_fake_scons_py(self)
- TestSCons_time.write_fake_svn_py(self, name)
- TestSCons_time.write_sample_directory(self, archive, dir, files)
- TestSCons_time.write_sample_tarfile(self, archive, dir, files)
- TestSCons_time.write_sample_zipfile(self, archive, dir, files)
- TestSCons_time.write_sample_project(self, archive, dir=None)