Please note:The SCons wiki is in read-only mode due to ongoing spam/DoS issues. Also, new account creation is currently disabled. We are looking into alternative wiki hosts.
Differences between revisions 13 and 14
Revision 13 as of 2013-08-28 20:43:58
Size: 6548
Comment: Spam
Revision 14 as of 2014-07-18 02:03:34
Size: 6670
Editor: AlexBurton
Comment: Added link to NonDeterministicDependencies
Deletions are marked like this. Additions are marked like this.
Line 145: Line 145:

If you have subsequent build steps that are dependent on results similar to the above see NonDeterministicDependencies

Using a Source Generator to Add Targets Dynamically

Here's what we do to add targets dynamically, from a list which isn't known in advance.

The scenario in our case is you have a source generator 'srcgen' which is built by scons from sources in the usual way. It puts out a bunch of other sources whose names can't be predicted in advance. In our case it also puts out a src-list.text file. Call the generated sources srcA, srcB, etc.

Now you want to (say) compile each of those srcX into binX and include those in your final product.

In our SConscript we do this:

   1 srcgen = env.Program('srcgen', ...)
   2 srclist = env.Command('src-list.txt', srcgen, 
   3                       '$SOURCE $SRCGEN_ARGS > $TARGET')
   4 dummy = env.ScanSrcs('#dummy-target', srclist, SCANSRCS_FUNC=add_target)
   5 env.AlwaysBuild(dummy)
   6 Alias('TopLevelAlias', dummy)

Now when scons goes to build TopLevelAlias, the dependencies are set so it builds srcgen, runs it to produce the srclist (and the other sources), and then runs ScanSrcs on the srclist. The interesting thing is that the ScanSrcs function will be called during the build phase after its dependencies are up-to-date, rather than while the SConscript is being read. It will call Program() or SharedLibrary() or whatever builders we want, which will update the dependency graph while scons is building. Sounds scary, but it works.

We made our ScanSrcs parameterized so it could be reused. Note that it's practically all plain python, no SCons calls at all except adding the Builder and checking for construction variables.

   1     def scansrcs(target, source, env):
   2         """Scans through the list in the file 'source', calling
   3         env['SCANSRCS_FUNC'](env,line) on each line of that file.
   4         Calls env['SCANSRCS_PREFUNC'](env,source) once before scanning. 
   5         """
   6         if SCons.Util.is_List(source):
   7             source=source[0]
   8         if 'SCANSRCS_FUNC' not in env:
   9             raise Error, "You must define SCANSRCS_FUNC as a scons env function."
  10         # Call the pre-func
  11         if 'SCANSRCS_PREFUNC' in env:
  12             env['SCANSRCS_PREFUNC'](env, source.path)
  13 
  14         try:
  15             f=open(source.path, 'r')
  16         except:
  17             print "scansrcs: Can't open source list file '%s' in %s" % \
  18                   (source.path,os.getcwd())
  19             raise
  20         # Scan through the lines
  21         for line in f:
  22             src = line.strip()
  23             # print "Found " + src
  24             try:
  25                 env['SCANSRCS_FUNC'](env, src)
  26             except:
  27                 print "SCANSRCS func raised exception:"
  28                 raise
  29         f.close()
  30     # This is a funky builder, because it never creates its target.
  31     # Should always be called with a fake target name.
  32     env.Append(BUILDERS = {'ScanSrcs' : Builder(action=scansrcs)})

And finally, here's the function add_target, called from ScanSrcs, that does the real work:

   1     def add_target(env, source):
   2         """Add scons commands to build a new target from a scanned src generator list file."""
   3         # Build the new targets in the build dir:
   4         target = '#'+join(env['BUILDDIR'], basename(re.sub(r'\.c$', env.get('PROGSUFFIX',''), source)))
   5         tgt = env.Program(target, source)
   6         env.Depends(tgt, '#dummy-target')
   7         p = env.Install(env['INSTALLDIR'], tgt)
   8         Alias('TopLevelAlias', p)

Note that it adds new Nodes for the program target and the installed version of it, and also adds that to the TopLevelAlias. This way as the scanner runs scons knows that the new target doesn't exist or is out of date, so it marks the TopLevelAlias out of date; then scons comes around again to look at TopLevelAlias and discovers it now needs to build 'binA', which depends on 'srcA' and so it builds it and installs it as we want it to.

The above has been cut down from our real version, but I hope it shows how it can be done, and that it's not really that hard.


Here is a version of the code above that you can copy & paste, ready to edit. I added some imports and altered some small stuff that was too related to the code's origins. Now this example works as soon as you change the file names to match those in your project.

   1 import types
   2 import os.path
   3 import re
   4 
   5 env = Environment(BUILDDIR = "bin",INSTALLDIR = "installdir")
   6 
   7 def scansrcs(target, source, env):
   8         """Scans through the list in the file 'source', calling
   9         env['SCANSRCS_FUNC'](env,line) on each line of that file.
  10         Calls env['SCANSRCS_PREFUNC'](env,source) once before scanning.
  11         """
  12         if type(source) is list:
  13             source=source[0]
  14         if 'SCANSRCS_FUNC' not in env:
  15             raise Error, "You must define SCANSRCS_FUNC as a scons env function."
  16         # Call the pre-func
  17         if 'SCANSRCS_PREFUNC' in env:
  18             env['SCANSRCS_PREFUNC'](env, source.path)
  19 
  20         try:
  21             f=open(source.path, 'r')
  22         except:
  23             print "scansrcs: Can't open source list file '%s' in %s" % \
  24                   (source.path,os.getcwd())
  25             raise
  26         # Scan through the lines
  27         for line in f:
  28             src = line.strip()
  29             # print "Found " + src
  30             try:
  31                 env['SCANSRCS_FUNC'](env, src)
  32             except:
  33                 print "SCANSRCS func raised exception:"
  34                 raise
  35         f.close()
  36 
  37 
  38 # This is a funky builder, because it never creates its target.
  39 # Should always be called with a fake target name.
  40 env.Append(BUILDERS = {'ScanSrcs' : Builder(action=scansrcs)})
  41 
  42 
  43 def add_target(env, source):
  44         """Add scons commands to build a new target from a scanned src generator list file."""
  45         # Build the new targets in the build dir:
  46         target = '#' + os.path.join(env['BUILDDIR'], os.path.basename(re.sub(r'\.c$',\
  47                                     env.get('PROGSUFFIX',''),source)))
  48         tgt = env.Program(target, source)
  49         env.Depends(tgt, '#dummy-target')
  50         p = env.Install(env['INSTALLDIR'],tgt)
  51         Alias('TopLevelAlias', p)
  52 
  53 
  54 srcgen = env.Program('srcgen', 'srcgen.c')
  55 srclist = env.Command('src-list.txt', srcgen,
  56                       './$SOURCE $SRCGEN_ARGS > $TARGET')
  57 dummy = env.ScanSrcs('#dummy-target', srclist, SCANSRCS_FUNC=add_target)
  58 env.AlwaysBuild(dummy)
  59 Alias('TopLevelAlias', dummy)

If you have subsequent build steps that are dependent on results similar to the above see NonDeterministicDependencies

DynamicSourceGenerator (last edited 2014-07-18 02:03:34 by AlexBurton)