Description of Problem

Many a time it is necessary to create a ZIP file or tarball for distribution of sources, binaries, and documentation. The ZIP file or tarball also must be properly rooted.

Unfortunately, SCons does not have a built-in capability to do this easily, but instead exposes an interface which makes this relatively easy.

One could use standard Python tools (e.g., shutil.copy()) to copy the necessary files and directories and then Zip then using the builders that come with SCons, but not only could you create something that isn't properly rooted but you also lose SCon's dependency tracking, cleaning, etc.

See also DistTarBuilder for another solution to this problem (with ability to handle file extension lists and excluded directory lists, but without the dependency tracking stuff)

Solution

The solution is two-step:

The builder to collect the files and directories is called "Accumulate", and the code for which is as follows.

   1 ##
   2 ## AccumulatorAction.py
   3 ##
   4 
   5 import myShutil # for better copytree()
   6 import os
   7 import shutil
   8 
   9 def accumulatorFunction(target, source, env):
  10   """Function called when builder is called"""
  11   destDir = str(target[0])
  12   if not os.path.exists(destDir):
  13       os.makedirs(destDir)
  14   for s in source:
  15       s = str(s)
  16       if os.path.isdir(s):
  17           myShutil.copytree(s, destDir, symlinks = False)
  18       else:
  19           shutil.copy2(s, destDir) 

The above code requires a better copytree() because shutil.copytree() is very limiting. What is needed is the equivalent of 'cp -R', which the following module provides.

   1 ##
   2 ## myShutil.py
   3 ##
   4 
   5 import os.path
   6 import shutil
   7     
   8 def copytree(src, dest, symlinks=False):
   9     """My own copyTree which does not fail if the directory exists.
  10     
  11     Recursively copy a directory tree using copy2().
  12 
  13     If the optional symlinks flag is true, symbolic links in the
  14     source tree result in symbolic links in the destination tree; if
  15     it is false, the contents of the files pointed to by symbolic
  16     links are copied.
  17     
  18     Behavior is meant to be identical to GNU 'cp -R'.    
  19     """
  20     def copyItems(src, dest, symlinks=False):
  21         """Function that does all the work.
  22         
  23         It is necessary to handle the two 'cp' cases:
  24         - destination does exist
  25         - destination does not exist
  26         
  27         See 'cp -R' documentation for more details
  28         """
  29         for item in os.listdir(src):
  30            srcPath = os.path.join(src, item)
  31            if os.path.isdir(srcPath):
  32                srcBasename = os.path.basename(srcPath)
  33                destDirPath = os.path.join(dest, srcBasename)
  34                if not os.path.exists(destDirPath):
  35                    os.makedirs(destDirPath)
  36                copyItems(srcPath, destDirPath)
  37            elif os.path.islink(item) and symlinks:
  38                linkto = os.readlink(item)
  39                os.symlink(linkto, dest)
  40            else:
  41                shutil.copy2(srcPath, dest)
  42                
  43     # case 'cp -R src/ dest/' where dest/ already exists
  44     if os.path.exists(dest):
  45        destPath = os.path.join(dest, os.path.basename(src))
  46        if not os.path.exists(destPath):
  47            os.makedirs(destPath)
  48     # case 'cp -R src/ dest/' where dest/ does not exist
  49     else:
  50        os.makedirs(dest)
  51        destPath = dest
  52     # actually copy the files
  53     copyItems(src, destPath)  

Using env.Zip() is not very useful for creating properly rooted ZIP files, but the following builder is:

   1 ##
   2 ## Zipper.py
   3 ##
   4 import distutils.archive_util 
   5 
   6 def zipperFunction(target, source, env):
   7         """Function to use as an action which creates a ZIP file from the arguments"""
   8         targetName = str(target[0])
   9         sourceDir = str(source[0])
  10         distutils.archive_util.make_archive(targetName, 'zip', sourceDir)

To actually create the builders, use the following example:

   1 ##
   2 ## SConstruct
   3 ##
   4 env = Environment()
   5 
   6 # add builder to accumulate files
   7 accuBuilder = env.Builder(action=AccumulatorAction.accumulatorFunction,
   8     source_factory=SCons.Node.FS.default_fs.Entry,
   9     target_factory=SCons.Node.FS.default_fs.Entry,
  10     multi=1)
  11 env['BUILDERS']['Accumulate'] = accuBuilder
  12 
  13 # add builder to zip files
  14 zipBuilder = env.Builder(action=Zipper.zipperFunction,
  15    source_factory=SCons.Node.FS.default_fs.Entry,
  16    target_factory=SCons.Node.FS.default_fs.Entry,
  17    multi=0)
  18 env['BUILDERS']['Zipper'] = zipBuilder

And you are done! Quite the hassle, but I think the result is worthwhile (IMHO). See the following code example for yourself:

   1 ##
   2 ## SConstruct
   3 ##
   4 env.Accumulate('distDir/src', 'main.cpp')
   5 env.Accumulate('distDir/images', 'something.png')
   6 
   7 Export('env')
   8 SConscript('inner/SConscript')
   9 
  10 env.Zipper('package'   , 'distDir')

   1 ##
   2 ## inner/SConscript
   3 ##
   4 Import('env')
   5 
   6 env.Accumulate('#/distDir', 'include')

Scons output will be as follows:

scons: Reading SConscript files ...
scons: done reading SConscript files.
scons: Building targets ...
accumulatorFunction(["distDir/images"], ["something.png"])
accumulatorFunction(["distDir/src"], ["main.cpp"])
accumulatorFunction(["distDir"], ["inner/include"])
zipperFunction(["package"], ["distDir"])
scons: done building targets.

And your ZIP file will be properly rooted:

Archive:  package.zip
  Length     Date   Time    Name
 --------    ----   ----    ----
       59  05-09-05 08:15   src/main.cpp
        7  05-09-05 08:15   images/something.png
        0  05-09-05 08:15   include/app/app.h
        0  05-09-05 08:15   include/app/db/db.h
        0  05-09-05 08:15   include/more/sun.h
        0  05-09-05 08:15   include/more/build/moon.h
        0  05-09-05 08:15   include/tool/something.h
 --------                   -------
       66                   7 files

TODO

All this can be improved by:

AccumulateBuilder (last edited 2008-03-12 02:47:11 by localhost)