Please note:The SCons wiki is now restored from the attack in March 2013. All old passwords have been invalidated. Please reset your password if you have an account. If you note missing pages, please report them to webmaster@scons.org. Also, new account creation is currently disabled due to an ongoing spam flood (2013/08/27).
Differences between revisions 15 and 16
Revision 15 as of 2013-06-08 09:46:16
Size: 33560
Editor: PhilippKraus
Comment: add url node
Revision 16 as of 2013-06-08 10:25:12
Size: 34057
Editor: PhilippKraus
Comment: class explanation
Deletions are marked like this. Additions are marked like this.
Line 117: Line 117:
The {{{generate}}} function initializes the builder, so the emitter and action function are set. The {{{single_source}}} option is set to true, because the builder creates only one file. Important options are the {{{target_factory}}} and the {{{source_factory}}}, because this builder should create a file, so the {{{target_factory}}} must be set to a file, but the {{{source_factory}}} gets an URL input, so a normal string (Python) value must be used. The {{{generate}}} function initializes the builder, so the emitter and action function are set. The {{{single_source}}} option is set to true, because the builder creates only one file. Important options are the {{{target_factory}}} and the {{{source_factory}}}, because this builder should create a file, so the {{{target_factory}}} must be set to a file, but the {{{source_factory}}} gets an URL input, so a normal string (Python) value must be used. The target should be build only if the data behind the source is changed, an URL is a static value, so we need a check on the server. In this case we need to define our own node ({{{URLNode}}}). This node derivate all data from the {{{SCons.Node.Python.Value}}}, be we need to overload the {{{get_csig}}}, because with this method SCons creates the information if a target is up-to-date. I have copied the content from the {{{SCons.Node.Python.Value}}} and append only the URL header information.

Download- & Unpack-Builder

The idea for these builders is borned by the problem, that a project uses different libraries and some libraries must be installed with different compiler- and linker options. The libraries must be also updated during the project lifetime. The task of the builders should be:

  1. download the source code of a library (eg tar.gz / tar.bz2)
  2. unpack this file
  3. build the source code with SCons to a shared / static library
  4. install the header files and library

process.png

The picture shows the build process of a shared library, so the latest version of a library should be read from the project webpage, the download URL should be pushed to the builder, that gets the file, this file is pushed to a builder, which extracts the file and pushs the file content, which is needed by the shared library builder, to SCons SharedLibrary call, which is build finally the library. The update process is worked in an equal way, because only the URL changes.

Download Builder

The Download-Builder should only download a file from an URL input. Python supports the URLLib2 extension. The filename of the downloaded file can be created by the users target name or should be created by the server, that sends the file. This information can be handle by Python's URLParse extension.

   1 import urllib2, urlparse
   2 import SCons.Builder, SCons.Node, SCons.Errors
   3 
   4 
   5 # define an own node, for checking the data behind the URL,
   6 # we must download only than, if the data is changed, the
   7 # node derivates from the Python.Value node
   8 class URLNode(SCons.Node.Python.Value) :
   9 
  10     # overload the get_csig (copy the source from the
  11     # Python.Value node and append the data of the URL header
  12     def get_csig(self, calc=None): 
  13         try: 
  14             return self.ninfo.csig 
  15         except AttributeError: 
  16             pass 
  17         
  18         # read URL header information
  19         try :
  20             response = urllib2.urlopen( str(self.value) ).info()
  21         except Exception, e :
  22             raise SCons.Errors.StopError( e )
  23             
  24         contents = ""
  25         # append the data from the URL header if exists
  26         # otherwise the returning data is equal to the Python.Value node
  27         if "Last-Modified" in response :
  28             contents = contents + response["Last-Modified"]
  29         if "Content-Length" in response :
  30             contents = contents + response["Content-Length"]
  31         if not contents :
  32             contents = self.get_contents() 
  33         self.get_ninfo().csig = contents 
  34         return contents 
  35 
  36 
  37 # creates the downloading output message
  38 # @param s original message
  39 # @param target target name
  40 # @param source source name
  41 # @param env environment object
  42 def __message( s, target, source, env ) :
  43     print "downloading [%s] to [%s] ..." % (source[0], target[0])
  44 
  45 
  46 
  47 # the download function, which reads the data from the URL
  48 # and writes it down to the file
  49 # @param target target file on the local drive
  50 # @param source URL
  51 # @param env environment object
  52 def __action( target, source, env ) :
  53     try :
  54         stream = urllib2.urlopen( str(source[0]) )
  55         file   = open( str(target[0]), "wb" )
  56         file.write(stream.read())
  57         file.close()
  58         stream.close()
  59     except Exception, e :
  60         raise SCons.Errors.StopError( e )
  61 
  62 
  63 # defines the emitter of the builder
  64 # @param target target file on the local drive
  65 # @param source URL
  66 # @param env environment object
  67 def __emitter( target, source, env ) :
  68     # we need a temporary file, because the dependency graph
  69     # of Scons need a physical existing file - so we prepare it
  70     target[0].prepare()
  71 
  72     if not env.get("URLDOWNLOAD_USEURLFILENAME", False) :
  73         return target, source
  74 
  75     try :
  76         url = urlparse.urlparse( str(source[0]) )
  77     except Exception, e :
  78         raise SCons.Errors.StopError( e )
  79 
  80     return url.path.split("/")[-1], source
  81 
  82 
  83 
  84 # generate function, that adds the builder to the environment,
  85 # the value "DOWNLOAD_USEFILENAME" replaces the target name with
  86 # the filename of the URL
  87 # @param env environment object
  88 def generate( env ) :
  89     env["BUILDERS"]["URLDownload"] = SCons.Builder.Builder( action = __action,  emitter = __emitter,  target_factory = SCons.Node.FS.File,  source_factory = URLNode,  single_source = True,  PRINT_CMD_LINE_FUNC = __message )
  90     env.Replace(URLDOWNLOAD_USEURLFILENAME =  True )
  91 
  92 
  93 # existing function of the builder
  94 # @param env environment object
  95 # @return true
  96 def exists(env) :
  97     return 1

The __action is the builder function, that downloads the file. The function uses the URLLib2 object and writes the data to a file stream with the target name. The emitter function __emitter defines the emitter function, so this function translates the URL filename into the target filename. Within the builder this option can be enabled / disabled by setting the flag URLDOWNLOAD_USEURLFILENAME with a boolean value. The URLParse extension can read the file information and returns it. The emitter is run before the builder creates the file (downloads the data to a file), so the emitter must be "prepare" the target first.

The generate function initializes the builder, so the emitter and action function are set. The single_source option is set to true, because the builder creates only one file. Important options are the target_factory and the source_factory, because this builder should create a file, so the target_factory must be set to a file, but the source_factory gets an URL input, so a normal string (Python) value must be used. The target should be build only if the data behind the source is changed, an URL is a static value, so we need a check on the server. In this case we need to define our own node (URLNode). This node derivate all data from the SCons.Node.Python.Value, be we need to overload the get_csig, because with this method SCons creates the information if a target is up-to-date. I have copied the content from the SCons.Node.Python.Value and append only the URL header information.

The builder can be used with (the URL can be any URL type which is supported by Pythons URLLIB2)

   1 env.URLDownload( "<filename>", "<download url>" )

Unpack Builder

The next step is an Unpack-Builder, that can unpack a tar.gz or tar.bz2 file. Unix derivatives uses GZip, BZip2 and Tar for extracting these filetypes, which are often part of the distribution. On MS Windows 7-Zip can handle these files, so the builder uses depend on the system the correct toolset. Each tool can return another format of the archive file & directory list, so the builder must understand the format for create the correct target list, in this case the builder supports a callable Python structure that splits the text output of the tool into a target file list. Because of this circumstances the call of the extracting tool must catch the output. The emitter of the builder should create a file list with individual split of the output and the builder should run the extract command.

   1 import subprocess, os
   2 import SCons.Errors, SCons.Warnings, SCons.Util
   3 
   4 
   5 
   6 # enables Scons warning for this builder
   7 class UnpackWarning(SCons.Warnings.Warning) :
   8     pass
   9 
  10 SCons.Warnings.enableWarningClass(UnpackWarning)
  11 
  12 
  13 
  14 # extractor function for Tar output
  15 # @param env environment object
  16 # @param count number of returning lines
  17 # @param no number of the output line
  18 # @param i line content
  19 def __fileextractor_nix_tar( env, count, no, i ) :
  20     return i.split()[-1]
  21 
  22 # extractor function for GZip output,
  23 # ignore the first line
  24 # @param env environment object
  25 # @param count number of returning lines
  26 # @param no number of the output line
  27 # @param i line content
  28 def __fileextractor_nix_gzip( env, count, no, i ) :
  29     if no == 0 :
  30         return None
  31     return i.split()[-1]
  32 
  33 # extractor function for Unzip output,
  34 # ignore the first & last two lines
  35 # @param env environment object
  36 # @param count number of returning lines
  37 # @param no number of the output line
  38 # @param i line content
  39 def __fileextractor_nix_unzip( env, count, no, i ) :
  40     if no < 3 or no >= count - 2 :
  41         return None
  42     return i.split()[-1]
  43 
  44 # extractor function for 7-Zip
  45 # @param env environment object
  46 # @param count number of returning lines
  47 # @param no number of the output line
  48 # @param i line content
  49 def __fileextractor_win_7zip( env, count, no, i ) :
  50     item = i.split()
  51     if no > 8 and no < count - 2 :
  52         return item[-1]
  53     return None
  54 
  55 
  56 
  57 
  58 # returns the extractor item for handling the source file
  59 # @param source input source file
  60 # @param env environment object
  61 # @return extractor entry or None on non existing
  62 def __getExtractor( source, env ) :
  63     # we check each unpacker and get the correct list command first, run the command and
  64     # replace the target filelist with the list values, we sorte the extractors by their priority
  65     for unpackername, extractor in sorted(env["UNPACK"]["EXTRACTOR"].iteritems(), key = lambda (k,v) : (v["PRIORITY"],k)):
  66 
  67         if not SCons.Util.is_String(extractor["RUN"]) :
  68             raise SCons.Errors.StopError("list command of the unpack builder for [%s] archives is not a string" % (unpackername))
  69         if not len(extractor["RUN"]) :
  70             raise SCons.Errors.StopError("run command of the unpack builder for [%s] archives is not set - can not extract files" % (unpackername))
  71 
  72 
  73         if not SCons.Util.is_String(extractor["LISTFLAGS"]) :
  74             raise SCons.Errors.StopError("list flags of the unpack builder for [%s] archives is not a string" % (unpackername))
  75         if not SCons.Util.is_String(extractor["LISTCMD"]) :
  76             raise SCons.Errors.StopError("list command of the unpack builder for [%s] archives is not a string" % (unpackername))
  77 
  78         if not SCons.Util.is_String(extractor["EXTRACTFLAGS"]) :
  79             raise SCons.Errors.StopError("extract flags of the unpack builder for [%s] archives is not a string" % (unpackername))
  80         if not SCons.Util.is_String(extractor["EXTRACTCMD"]) :
  81             raise SCons.Errors.StopError("extract command of the unpack builder for [%s] archives is not a string" % (unpackername))
  82 
  83 
  84         # check the source file suffix and if the first is found, run the list command
  85         if not SCons.Util.is_List(extractor["SUFFIX"]) :
  86             raise SCons.Errors.StopError("suffix list of the unpack builder for [%s] archives is not a list" % (unpackername))
  87 
  88         for suffix in extractor["SUFFIX"] :
  89             if str(source[0]).lower()[-len(suffix):] == suffix.lower() :
  90                 return extractor
  91 
  92     return None
  93 
  94 
  95 # creates the extracter output message
  96 # @param s original message
  97 # @param target target name
  98 # @param source source name
  99 # @param env environment object
 100 def __message( s, target, source, env ) :
 101     print "extract [%s] ..." % (source[0])
 102 
 103 
 104 # action function for extracting of the data
 105 # @param target target packed file
 106 # @param source extracted files
 107 # @param env environment object
 108 def __action( target, source, env ) :
 109     extractor = __getExtractor(source, env)
 110     if not extractor :
 111         raise SCons.Errors.StopError( "can not find any extractor value for the source file [%s]" % (source[0]) )
 112 
 113 
 114     # if the extract command is empty, we create an error
 115     if len(extractor["EXTRACTCMD"]) == 0 :
 116         raise SCons.Errors.StopError( "the extractor command for the source file [%s] is empty" % (source[0]) )
 117 
 118     # build it now (we need the shell, because some programs need it)
 119     handle = None
 120     cmd    = env.subst(extractor["EXTRACTCMD"], source=source, target=target)
 121 
 122     if env["UNPACK"]["VIWEXTRACTOUTPUT"] :
 123         handle  = subprocess.Popen( cmd, shell=True )
 124     else :
 125         devnull = open(os.devnull, "wb")
 126         handle  = subprocess.Popen( cmd, shell=True, stdout=devnull )
 127 
 128     if handle.wait() <> 0 :
 129         raise SCons.Errors.BuildError( "error running extractor [%s] on the source [%s]" % (cmd, source[0])  )
 130 
 131 
 132 # emitter function for getting the files
 133 # within the archive
 134 # @param target target packed file
 135 # @param source extracted files
 136 # @param env environment object
 137 def __emitter( target, source, env ) :
 138     extractor = __getExtractor(source, env)
 139     if not extractor :
 140         raise SCons.Errors.StopError( "can not find any extractor value for the source file [%s]" % (source[0]) )
 141 
 142     # we do a little trick, because in some cases we do not have got a physical
 143     # file (eg we download a packed archive), so we don't get a list or knows
 144     # the targets. On physical files we can do this with the LISTCMD, but on
 145     # non-physical files we hope the user knows the target files, so we inject
 146     # this knowledge into the return target.
 147     if env.has_key("UNPACKLIST") :
 148         if not SCons.Util.is_List(env["UNPACKLIST"]) and not SCons.Util.is_String(env["UNPACKLIST"]) :
 149             raise SCons.Errors.StopError( "manual target list of [%s] must be a string or list" % (source[0]) )
 150         if not env["UNPACKLIST"] :
 151             raise SCons.Errors.StopError( "manual target list of [%s] need not be empty" % (source[0]) )
 152         return env["UNPACKLIST"], source
 153 
 154 
 155     # we check if the source file exists, because we would like to read the data
 156     if not source[0].exists() :
 157         raise SCons.Errors.StopError( "source file [%s] must be exist" % (source[0]) )
 158 
 159     # create the list command and run it in a subprocess and pipes the output to a variable,
 160     # we need the shell for reading data from the stdout
 161     cmd    = env.subst(extractor["LISTCMD"], source=source, target=target)
 162     handle = subprocess.Popen( cmd, shell=True, stdout=subprocess.PIPE )
 163     target = handle.stdout.readlines()
 164     handle.communicate()
 165     if handle.returncode <> 0 :
 166         raise SCons.Errors.StopError("error on running list command [%s] of the source file [%s]" % (cmd, source[0]) )
 167 
 168     # if the returning output exists and the listseperator is a callable structure
 169     # we run it for each line of the output and if the return of the callable is
 170     # a string we push it back to the target list
 171     try :
 172         if callable(extractor["LISTEXTRACTOR"]) :
 173             target = filter( lambda s : SCons.Util.is_String(s), [ extractor["LISTEXTRACTOR"]( env, len(target), no, i) for no, i in enumerate(target) ] )
 174     except Exception, e :
 175         raise SCons.Errors.StopError( "%s" % (e) )
 176 
 177     # the line removes duplicated names - we need this line, otherwise a cyclic dependency error will occured,
 178     # because the list process can create redundant data (an archive file can not store redundant content in a filepath)
 179     target = [i.strip() for i in list(set(target))]
 180     if not target :
 181         SCons.Warnings.warn(UnpackWarning, "emitter file list on target [%s] is empty, please check your extractor list function [%s]" % (source[0], cmd) )
 182 
 183     # we append the extractdir to each target if is not absolut
 184     if env["UNPACK"]["EXTRACTDIR"] <> "." :
 185         target = [i if os.path.isabs(i) else os.path.join(env["UNPACK"]["EXTRACTDIR"], i) for i in target]
 186 
 187     return target, source
 188 
 189 
 190 
 191 # generate function, that adds the builder to the environment
 192 # @param env environment object
 193 def generate( env ) :
 194     # setup environment variable
 195     toolset = {
 196         "STOPONEMPTYFILE"  : True,
 197         "VIWEXTRACTOUTPUT" : False,
 198         "EXTRACTDIR"       : ".",
 199         "EXTRACTOR" : {
 200             "TARGZ" : {
 201                 "PRIORITY"       : 0,
 202                 "SUFFIX"         : [".tar.gz", ".tgz", ".tar.gzip"],
 203                 "EXTRACTSUFFIX"  : "",
 204                 "EXTRACTFLAGS"   : "",
 205                 "EXTRACTCMD"     : "${UNPACK['EXTRACTOR']['TARGZ']['RUN']} ${UNPACK['EXTRACTOR']['TARGZ']['EXTRACTFLAGS']} $SOURCE ${UNPACK['EXTRACTOR']['TARGZ']['EXTRACTSUFFIX']}",
 206                 "RUN"            : "",
 207                 "LISTCMD"        : "${UNPACK['EXTRACTOR']['TARGZ']['RUN']} ${UNPACK['EXTRACTOR']['TARGZ']['LISTFLAGS']} $SOURCE ${UNPACK['EXTRACTOR']['TARGZ']['LISTSUFFIX']}",
 208                 "LISTSUFFIX"     : "",
 209                 "LISTFLAGS"      : "",
 210                 "LISTEXTRACTOR"  : None
 211             },
 212 
 213             "TARBZ" : {
 214                 "PRIORITY"       : 0,
 215                 "SUFFIX"         : [".tar.bz", ".tbz", ".tar.bz2", ".tar.bzip2", ".tar.bzip"],
 216                 "EXTRACTSUFFIX"  : "",
 217                 "EXTRACTFLAGS"   : "",
 218                 "EXTRACTCMD"     : "${UNPACK['EXTRACTOR']['TARBZ']['RUN']} ${UNPACK['EXTRACTOR']['TARBZ']['EXTRACTFLAGS']} $SOURCE ${UNPACK['EXTRACTOR']['TARBZ']['EXTRACTSUFFIX']}",
 219                 "RUN"            : "",
 220                 "LISTCMD"        : "${UNPACK['EXTRACTOR']['TARBZ']['RUN']} ${UNPACK['EXTRACTOR']['TARBZ']['LISTFLAGS']} $SOURCE ${UNPACK['EXTRACTOR']['TARBZ']['LISTSUFFIX']}",
 221                 "LISTSUFFIX"     : "",
 222                 "LISTFLAGS"      : "",
 223                 "LISTEXTRACTOR"  : None
 224             },
 225 
 226             "BZIP" : {
 227                 "PRIORITY"       : 1,
 228                 "SUFFIX"         : [".bz", "bzip", ".bz2", ".bzip2"],
 229                 "EXTRACTSUFFIX"  : "",
 230                 "EXTRACTFLAGS"   : "",
 231                 "EXTRACTCMD"     : "${UNPACK['EXTRACTOR']['BZIP']['RUN']} ${UNPACK['EXTRACTOR']['BZIP']['EXTRACTFLAGS']} $SOURCE ${UNPACK['EXTRACTOR']['BZIP']['EXTRACTSUFFIX']}",
 232                 "RUN"            : "",
 233                 "LISTCMD"        : "${UNPACK['EXTRACTOR']['BZIP']['RUN']} ${UNPACK['EXTRACTOR']['BZIP']['LISTFLAGS']} $SOURCE ${UNPACK['EXTRACTOR']['BZIP']['LISTSUFFIX']}",
 234                 "LISTSUFFIX"     : "",
 235                 "LISTFLAGS"      : "",
 236                 "LISTEXTRACTOR"  : None
 237             },
 238 
 239             "GZIP" : {
 240                 "PRIORITY"       : 1,
 241                 "SUFFIX"         : [".gz", ".gzip"],
 242                 "EXTRACTSUFFIX"  : "",
 243                 "EXTRACTFLAGS"   : "",
 244                 "EXTRACTCMD"     : "${UNPACK['EXTRACTOR']['GZIP']['RUN']} ${UNPACK['EXTRACTOR']['GZIP']['EXTRACTFLAGS']} $SOURCE ${UNPACK['EXTRACTOR']['GZIP']['EXTRACTSUFFIX']}",
 245                 "RUN"            : "",
 246                 "LISTCMD"        : "${UNPACK['EXTRACTOR']['GZIP']['RUN']} ${UNPACK['EXTRACTOR']['GZIP']['LISTFLAGS']} $SOURCE ${UNPACK['EXTRACTOR']['GZIP']['LISTSUFFIX']}",
 247                 "LISTSUFFIX"     : "",
 248                 "LISTFLAGS"      : "",
 249                 "LISTEXTRACTOR"  : None
 250             },
 251 
 252             "TAR" : {
 253                 "PRIORITY"       : 1,
 254                 "SUFFIX"         : [".tar"],
 255                 "EXTRACTSUFFIX"  : "",
 256                 "EXTRACTFLAGS"   : "",
 257                 "EXTRACTCMD"     : "${UNPACK['EXTRACTOR']['TAR']['RUN']} ${UNPACK['EXTRACTOR']['TAR']['EXTRACTFLAGS']} $SOURCE ${UNPACK['EXTRACTOR']['TAR']['EXTRACTSUFFIX']}",
 258                 "RUN"            : "",
 259                 "LISTCMD"        : "${UNPACK['EXTRACTOR']['TAR']['RUN']} ${UNPACK['EXTRACTOR']['TAR']['LISTFLAGS']} $SOURCE ${UNPACK['EXTRACTOR']['TAR']['LISTSUFFIX']}",
 260                 "LISTSUFFIX"     : "",
 261                 "LISTFLAGS"      : "",
 262                 "LISTEXTRACTOR"  : None
 263             },
 264 
 265             "ZIP" : {
 266                 "PRIORITY"       : 1,
 267                 "SUFFIX"         : [".zip"],
 268                 "EXTRACTSUFFIX"  : "",
 269                 "EXTRACTFLAGS"   : "",
 270                 "EXTRACTCMD"     : "${UNPACK['EXTRACTOR']['ZIP']['RUN']} ${UNPACK['EXTRACTOR']['ZIP']['EXTRACTFLAGS']} $SOURCE ${UNPACK['EXTRACTOR']['ZIP']['EXTRACTSUFFIX']}",
 271                 "RUN"            : "",
 272                 "LISTCMD"        : "${UNPACK['EXTRACTOR']['ZIP']['RUN']} ${UNPACK['EXTRACTOR']['ZIP']['LISTFLAGS']} $SOURCE ${UNPACK['EXTRACTOR']['ZIP']['LISTSUFFIX']}",
 273                 "LISTSUFFIX"     : "",
 274                 "LISTFLAGS"      : "",
 275                 "LISTEXTRACTOR"  : None
 276             }
 277         }
 278     }
 279 
 280     # read tools for Windows system
 281     if env["PLATFORM"] <> "darwin" and "win" in env["PLATFORM"] :
 282 
 283         if env.WhereIs("7z") :
 284             toolset["EXTRACTOR"]["TARGZ"]["RUN"]           = "7z"
 285             toolset["EXTRACTOR"]["TARGZ"]["LISTEXTRACTOR"] = __fileextractor_win_7zip
 286             toolset["EXTRACTOR"]["TARGZ"]["LISTFLAGS"]     = "x"
 287             toolset["EXTRACTOR"]["TARGZ"]["LISTSUFFIX"]    = "-so -y | ${UNPACK['EXTRACTOR']['TARGZ']['RUN']} l -sii -ttar -y -so"
 288             toolset["EXTRACTOR"]["TARGZ"]["EXTRACTFLAGS"]  = "x"
 289             toolset["EXTRACTOR"]["TARGZ"]["EXTRACTSUFFIX"] = "-so -y | ${UNPACK['EXTRACTOR']['TARGZ']['RUN']} x -sii -ttar -y -oc:${UNPACK['EXTRACTDIR']}"
 290 
 291             toolset["EXTRACTOR"]["TARBZ"]["RUN"]           = "7z"
 292             toolset["EXTRACTOR"]["TARBZ"]["LISTEXTRACTOR"] = __fileextractor_win_7zip
 293             toolset["EXTRACTOR"]["TARBZ"]["LISTFLAGS"]     = "x"
 294             toolset["EXTRACTOR"]["TARBZ"]["LISTSUFFIX"]    = "-so -y | ${UNPACK['EXTRACTOR']['TARGZ']['RUN']} l -sii -ttar -y -so"
 295             toolset["EXTRACTOR"]["TARBZ"]["EXTRACTFLAGS"]  = "x"
 296             toolset["EXTRACTOR"]["TARBZ"]["EXTRACTSUFFIX"] = "-so -y | ${UNPACK['EXTRACTOR']['TARGZ']['RUN']} x -sii -ttar -y -oc:${UNPACK['EXTRACTDIR']}"
 297 
 298             toolset["EXTRACTOR"]["BZIP"]["RUN"]            = "7z"
 299             toolset["EXTRACTOR"]["BZIP"]["LISTEXTRACTOR"]  = __fileextractor_win_7zip
 300             toolset["EXTRACTOR"]["BZIP"]["LISTFLAGS"]      = "l"
 301             toolset["EXTRACTOR"]["BZIP"]["LISTSUFFIX"]     = "-y -so"
 302             toolset["EXTRACTOR"]["BZIP"]["EXTRACTFLAGS"]   = "x"
 303             toolset["EXTRACTOR"]["BZIP"]["EXTRACTSUFFIX"]  = "-y -oc:${UNPACK['EXTRACTDIR']}"
 304 
 305             toolset["EXTRACTOR"]["GZIP"]["RUN"]            = "7z"
 306             toolset["EXTRACTOR"]["GZIP"]["LISTEXTRACTOR"]  = __fileextractor_win_7zip
 307             toolset["EXTRACTOR"]["GZIP"]["LISTFLAGS"]      = "l"
 308             toolset["EXTRACTOR"]["GZIP"]["LISTSUFFIX"]     = "-y -so"
 309             toolset["EXTRACTOR"]["GZIP"]["EXTRACTFLAGS"]   = "x"
 310             toolset["EXTRACTOR"]["GZIP"]["EXTRACTSUFFIX"]  = "-y -oc:${UNPACK['EXTRACTDIR']}"
 311 
 312             toolset["EXTRACTOR"]["ZIP"]["RUN"]             = "7z"
 313             toolset["EXTRACTOR"]["ZIP"]["LISTEXTRACTOR"]   = __fileextractor_win_7zip
 314             toolset["EXTRACTOR"]["ZIP"]["LISTFLAGS"]       = "l"
 315             toolset["EXTRACTOR"]["ZIP"]["LISTSUFFIX"]      = "-y -so"
 316             toolset["EXTRACTOR"]["ZIP"]["EXTRACTFLAGS"]    = "x"
 317             toolset["EXTRACTOR"]["ZIP"]["EXTRACTSUFFIX"]   = "-y -oc:${UNPACK['EXTRACTDIR']}"
 318 
 319             toolset["EXTRACTOR"]["TAR"]["RUN"]             = "7z"
 320             toolset["EXTRACTOR"]["TAR"]["LISTEXTRACTOR"]   = __fileextractor_win_7zip
 321             toolset["EXTRACTOR"]["TAR"]["LISTFLAGS"]       = "l"
 322             toolset["EXTRACTOR"]["TAR"]["LISTSUFFIX"]      = "-y -ttar -so"
 323             toolset["EXTRACTOR"]["TAR"]["EXTRACTFLAGS"]    = "x"
 324             toolset["EXTRACTOR"]["TAR"]["EXTRACTSUFFIX"]   = "-y -ttar -oc:${UNPACK['EXTRACTDIR']}"
 325 
 326         # here can add some other Windows tools, that can handle the archive files
 327         # but I don't know which ones can handle all file types
 328 
 329 
 330 
 331     # read the tools on *nix systems and sets the default parameters
 332     elif env["PLATFORM"] in ["darwin", "linux", "posix"] :
 333 
 334         if env.WhereIs("unzip") :
 335             toolset["EXTRACTOR"]["ZIP"]["RUN"]             = "unzip"
 336             toolset["EXTRACTOR"]["ZIP"]["LISTEXTRACTOR"]   = __fileextractor_nix_unzip
 337             toolset["EXTRACTOR"]["ZIP"]["LISTFLAGS"]       = "-l"
 338             toolset["EXTRACTOR"]["ZIP"]["EXTRACTFLAGS"]    = "-oqq"
 339 
 340         if env.WhereIs("tar") :
 341             toolset["EXTRACTOR"]["TAR"]["RUN"]             = "tar"
 342             toolset["EXTRACTOR"]["TAR"]["LISTEXTRACTOR"]   = __fileextractor_nix_tar
 343             toolset["EXTRACTOR"]["TAR"]["LISTFLAGS"]       = "tvf"
 344             toolset["EXTRACTOR"]["TAR"]["EXTRACTFLAGS"]    = "xf"
 345             toolset["EXTRACTOR"]["TAR"]["EXTRACTSUFFIX"]   = "-C ${UNPACK['EXTRACTDIR']}"
 346 
 347             toolset["EXTRACTOR"]["TARGZ"]["RUN"]           = "tar"
 348             toolset["EXTRACTOR"]["TARGZ"]["LISTEXTRACTOR"] = __fileextractor_nix_tar
 349             toolset["EXTRACTOR"]["TARGZ"]["EXTRACTFLAGS"]  = "xfz"
 350             toolset["EXTRACTOR"]["TARGZ"]["LISTFLAGS"]     = "tvfz"
 351             toolset["EXTRACTOR"]["TARGZ"]["EXTRACTSUFFIX"] = "-C ${UNPACK['EXTRACTDIR']}"
 352 
 353             toolset["EXTRACTOR"]["TARBZ"]["RUN"]           = "tar"
 354             toolset["EXTRACTOR"]["TARBZ"]["LISTEXTRACTOR"] = __fileextractor_nix_tar
 355             toolset["EXTRACTOR"]["TARBZ"]["EXTRACTFLAGS"]  = "xfj"
 356             toolset["EXTRACTOR"]["TARBZ"]["LISTFLAGS"]     = "tvfj"
 357             toolset["EXTRACTOR"]["TARBZ"]["EXTRACTSUFFIX"] = "-C ${UNPACK['EXTRACTDIR']}"
 358 
 359         if env.WhereIs("bzip2") :
 360             toolset["EXTRACTOR"]["BZIP"]["RUN"]            = "bzip2"
 361             toolset["EXTRACTOR"]["BZIP"]["EXTRACTFLAGS"]   = "-df"
 362 
 363         if env.WhereIs("gzip") :
 364             toolset["EXTRACTOR"]["GZIP"]["RUN"]            = "gzip"
 365             toolset["EXTRACTOR"]["GZIP"]["LISTEXTRACTOR"]  = __fileextractor_nix_gzip
 366             toolset["EXTRACTOR"]["GZIP"]["LISTFLAGS"]      = "-l"
 367             toolset["EXTRACTOR"]["GZIP"]["EXTRACTFLAGS"]   = "-df"
 368 
 369     else :
 370         raise SCons.Errors.StopError("Unpack tool detection on this platform [%s] unkown" % (env["PLATFORM"]))
 371 
 372     # the target_factory must be a "Entry", because the target list can be files and dirs, so we can not specified the targetfactory explicite
 373     env.Replace(UNPACK = toolset)
 374     env["BUILDERS"]["Unpack"] = SCons.Builder.Builder( action = __action,  emitter = __emitter,  target_factory = SCons.Node.FS.Entry,  source_factory = SCons.Node.FS.File,  single_source = True,  PRINT_CMD_LINE_FUNC = __message )
 375 
 376 
 377 # existing function of the builder
 378 # @param env environment object
 379 # @return true
 380 def exists(env) :
 381     return 1

The builder need an own warning, so a class is created and the warning is enabled. After that for each toolset the "splitting functions" are defined, that can understand the output of the tool, the function gets four parameter (environment, number of returning output lines, current line number, line content). One of these functions is called on each returning output line and the function must return the directory- or filename (on None the line is ignored). The __getExtractor function returns the parameter of an extractor command. All toolkits are stored in a Python dict, so this function returns one item of this dict and checks the parameter. This list uses a priority, which sets an order to the items, because file suffixes like tar.gz can be used by Tar and by GZip. The order defines that tar.gz is used by Tar first, because Tar can handle these files and pipe the data to GZip. GZip can extract only the gz-part, but we need two runs over the archive in this case to get the archive content (one for gz and next for tar). The __action function is the builder call, which runs the extract command, but here an own subprocess is used, because the output of the tool should suppress and the shell option must be enabled, because some toolkits need a shell.

The emitter __emitter must read the archive data, so it must create a file- and/or dictory list of the content, so for each extractor must be set a list command and an optional splitting function. The Unpack builder should be used in combination with the Download builder. In this process the emitter can create a problem, because the emitter of the Download builder does not create a correct archive, which can be read by the emitter of the Unpack builder, so there must be an "injection hack" with "user knowledge". The user knows which files are needed by the build process, so the Unpack builder has got a flag UNPACKLIST, which can be used for injection the emitter, so the filelist can be pushed into the emitter and is returned during the build process. With this injection the Download- and Unpack builder can connect in order, so the full build process works correct. The emitter reads equal to the builder the extractor commands, if an unpacklist is set, the emitter returns this list, if not the emitter uses the list command of the toolkit and runs the command in a subprocess, get its output and push the output line-by-line to the splitting function, uniquifies the resulted file- & directorylist and returns the list.

The generate function initializes the builder depend on the environment.

The builder can be used with

   1 # use without injection
   2 env.Unpack( "<target-name>", "<archive file>" )
   3 
   4 # use with injection
   5 env.Unpack( "<target-name>", "<archive file>", UNPACKLIST=[<list of files in the archive>] )

Conclusion

In the example the working process is shown. The example builds LUA by downloading the source package, extracting and building. LUA has got a Makefile in the source directory, but on MSVC or MinGW the build file must be created manually. With this solution the library can be build with SCons only.

   1 import urllib2, re, os
   2 import SCons.Errors
   3 
   4 
   5 # function that extract the URL from LUA's webpage
   6 def LUA_DownloadURL() :
   7     # read download path of the LUA library (latest version)
   8     f = urllib2.urlopen("http://www.lua.org/download.html")
   9     html = f.read()
  10     f.close()
  11 
  12     found = re.search("<a href=\"ftp/lua-(.*)\.tar\.gz\">", html, re.IGNORECASE)
  13     if found == None :
  14         raise SCons.Errors.StopError("LUA Download URL not found")
  15 
  16     downloadurl = found.group(0).replace("\"", "").replace("<", "").replace(">", "")
  17     downloadurl = re.sub(r'(?i)a href=', "", downloadurl)
  18 
  19     return "http://www.lua.org/" + downloadurl
  20 
  21 
  22 # create environment
  23 env        = Environment( tools = ["default", "URLDownload", "Unpack"], ENV = os.environ )
  24 
  25 # downloads the LUA source package
  26 dw         = env.URLDownload( "lua-download", LUA_DownloadURL() )
  27 
  28 # create the extract directory and call the unpack builder with an injection list
  29 extractdir = str(dw).replace("'", "").replace("[", "").replace("]", "").replace(".tar.gz", "")
  30 extract    = env.Unpack("lua-extract",  dw, UNPACKLIST=[os.path.join(extractdir, "src", i) for i in [ "lapi.c", "lcode.c", "lctype.c", "ldebug.c", "ldo.c", "ldump.c", "lfunc.c", "lgc.c", "llex.c", "lmem.c", "lobject.c", "lopcodes.c", "lparser.c", "lstate.c", "lstring.c", "ltable.c", "ltm.c", "lundump.c", "lvm.c", "lzio.c", "lauxlib.c", "lbaselib.c", "lbitlib.c", "lcorolib.c", "ldblib.c", "liolib.c", "lmathlib.c", "loslib.c", "lstrlib.c", "ltablib.c", "loadlib.c", "linit.c"]])
  31 
  32 # define compiler and linker options depend on the toolkit (here Linux, MSVC (Win32) and OSX),
  33 # but the platform detection should be better (eg with a script parameter)
  34 env.AppendUnique(CPPDEFINES  = ["LUA_COMPAT_ALL", "NDEBUG"])
  35 
  36 if env["PLATFORM"] == "darwin" :
  37     env.AppendUnique(CDEFINES    = ["LUA_USE_MACOSX"])
  38     env.AppendUnique(CFLAGS      = ["-O2"])
  39 
  40 elif env["PLATFORM"] == "posix" :
  41     env.AppendUnique(CPPDEFINES = ["LUA_USE_POSIX"])
  42     env.AppendUnique(CPPFLAGS   = ["-O2"])
  43 
  44 elif env["PLATFORM"] == "win32" :
  45     env.AppendUnique(CPPDEFINES = ["LUA_BUILD_AS_DLL"])
  46     env.AppendUnique(CPPFLAGS   = ["/O2", "/GR", "/EHsc", "/nologo", "/OPT:REF", "/OPT:ICF", "/LTCG"])
  47 
  48 env.SharedLibrary(target="lua", source=extract)

For each library a "download URL" function is used, which extracts with regular expressions the download URL from the project homepage. This function is called by the Download builder. The Download builder returns a filename, which is also used for the directoryname of the extracted data. The injection list of the Unpack builder can be created by the filenames and Python defaults path-join calls. After that the normale SCons C build process with SharedLibrary is started.

In my build processes I use a target / alias library which downloads, extracts and compiles all libraries that are needed by the project. The compiled libraries and their headers are stored in a subdirectory of the project library/<name of the library>/<version of the library>. With this process the update to a new library version is very easy, because it is full automated. For large libraries eg Qt or Boost this automation is very helpfull. The Boost installation process uses by default the bJam / b2 compiler, which can be build by a bootstrap process, so for my Boost building process I call env.Command after the unpack call, so the bJam / b2 is build first and after that, the command runs the build process of the library. For other libraries (eg LaPack or HDF) I call CMake from the command call, so SCons wraps only the library default build process. During the linking process of my project the SConstruct scripts scans the library installation directory and gets always the library with the latest / newest version (see Distutils Version), so the project upgrade to a new library version is very fast.

The source code of both builder can be downloaded here: Unpack-Builder / Download-Builder

DownloadUnpack (last edited 2013-06-08 10:25:12 by PhilippKraus)