Development Process

Testing Criteria

The underlying criteria for testing changes to the SCons code are taken from Aegis:

• Every change must have one or more new or modified tests checked in along with the code.

• The new code being checked in must pass all of the new and/or modified tests.

• The old, already checked-in code in must fail all of the new and/or modified tests.

• The new code being checked in must pass all unmodified, already checked-in tests.

In practice, these restrictions can be overridden as necessary­for example, when changing comments or documentation.

The criterion that surprises many people is having the old code fail the tests in the change. This makes sure that the new tests or modified tests really do exercise the bug fix or feature being added by the change.

Together, these criteria ensure that every newly checked-in version SCons conforms to defined behavior, as defined by the tests. Whenever a bug is found, its fix is checked in with a new or modified test that guarantees the bug will not recur in the future. We have already built up a regression test base of almost 90 tests that cover the vast majority of SCons' functionality.

Testing Infrastructure

Testing standards are no good if they're too much of a burden for developers, who will at best work around or ignore the testing requirements, or at worst stop contributing code and go join a project that's more fun. To this end, good testing infrastructure that makes it easy to write tests is crucial.

SCons development uses two development methodologies, one for the individual modules in the build engine, and the other for end-to-end tests of the SCons script.

For the build engine modules, we use PyUnit. Every change to a build engine module must have a change to its corresponding unit tests, which live side-by-side in a separate file that imports module. As we build up a large body of unit tests, this ensures that the build engine will perform correctly whenever someone uses it in some application other than the SCons script itself.

For end-to-end script tests, we have developed two modules to make writing tests easy. The first, TestCmd.py, is a generic module for testing commands or scripts (in any language, not just Python). The second module, TestScons.py, is a subclass of the generic TestCmd.py module. TestScons.py takes care of initialization and displaying error conditions specific to testing SCons.

In practice, simple tests only need to initialize a test object, use the object to write some input files, run SCons, and then check whatever criteria determine whether the test passed or failed. A complete test of the Program method, for example, looks like this:

      test = TestSCons.TestSCons()

      test.write('SConstruct',
      """env = Environment()
      env.Program(target = 'foo', source = 'foo.c')
      """)

      test.write('foo.c',
      """
      int
      main(int argc, char *argv[])
      {
          argv[argc++] = "-"; /* dummy use of args */
          printf("foo.c successfully compiled\\n");
          exit (0);
      }
      """)

      test.run(arguments = 'foo') # runs SCons

      test.run(program = test.workpath('foo'))

      test.fail_test(test.stdout() != "foo.c successfully compiled\n")

      test.pass_test()