This page presents material that is no longer needed but is being kept for historical purposes. As of version 0.98, SCons has a Glob function built-in.
This page shows you how to glob (search files in dir) based on scons Nodes rather than files. This way you get all the files that will be built as well as the ones that already exist in the filesystem.
As of this writing, none of these functions recurse into subdirs, but that's not hard either. Just check each node returned by all_children() and if it's a Dir node, recurse. Here's a trivial one:
1 def print_all_nodes(dirnode, level=0): 2 """Print all the scons nodes that are children of this node, recursively.""" 3 if type(dirnode)==type(''): 4 dirnode=Dir(dirnode) 5 dt = type(Dir('.')) 6 for f in dirnode.all_children(): 7 if type(f) == dt: 8 print "%s%s: .............."%(level * ' ', str(f)) 9 print_dir(f, level+2) 10 print "%s%s"%(level * ' ', str(f))
- Here is another version of Glob, similar to the one below, except it allows multiple excludes and includes, as well as a directory specification. It also deals directly with SCons Nodes.
NOTE: this is the best version available. The others below are mostly of historical interest.
1 import fnmatch 2 import os 3 def Glob( includes = Split( '*' ), excludes = None, dir = '.'): 4 """Similar to glob.glob, except globs SCons nodes, and thus sees 5 generated files and files from build directories. Basically, it sees 6 anything SCons knows about. A key subtlety is that since this function 7 operates on generated nodes as well as source nodes on the filesystem, 8 it needs to be called after builders that generate files you want to 9 include. 10 It will return both Dir entries and File entries 11 """ 12 def fn_filter(node): 13 fn = os.path.basename(str(node)) 14 match = False 15 for include in includes: 16 if fnmatch.fnmatchcase( fn, include ): 17 match = True 18 break 19 if match and excludes is not None: 20 for exclude in excludes: 21 if fnmatch.fnmatchcase( fn, exclude ): 22 match = False 23 break 24 return match 25 def filter_nodes(where): 26 children = filter(fn_filter, where.all_children(scan=0)) 27 nodes =  28 for f in children: 29 nodes.append(gen_node(f)) 30 return nodes 31 def gen_node(n): 32 """Checks first to see if the node is a file or a dir, then 33 creates the appropriate node. [code seems redundant, if the node 34 is a node, then shouldn't it just be left as is? 35 """ 36 if type(n) in (type(''), type(u'')): 37 path = n 38 else: 39 path = n.abspath 40 if os.path.isdir(path): 41 return Dir(n) 42 else: 43 return File(n) 44 here = Dir(dir) 45 nodes = filter_nodes(here) 46 node_srcs = [n.srcnode() for n in nodes] 47 src = here.srcnode() 48 if src is not here: 49 for s in filter_nodes(src): 50 if s not in node_srcs: 51 # Probably need to check if this node is a directory 52 nodes.append(gen_node(os.path.join(dir,os.path.basename(str(s))))) 53 return nodes
You can either include this function directly in your SConscript, or place in an external python script to include in any SConscript it's needed. SCons CVS has a better way to do it, but 0.96.1 requires the use of SConscript( 'external_script.py' ) and Import( ) / Export( )
I don't take much credit for this. Most credit belongs to John Meinel from the SCons mailinglist.
If you want to glob for generated files or for source files in a BuildDir, copy the following function into your project and call Glob instead of glob.glob.
1 def Glob(match): 2 """Similar to glob.glob, except globs SCons nodes, and thus sees 3 generated files and files from build directories. Basically, it sees 4 anything SCons knows about. A key subtlety is that since this function 5 operates on generated nodes as well as source nodes on the filesystem, 6 it needs to be called after builders that generate files you want to 7 include.""" 8 def fn_filter(node): 9 fn = str(node) 10 return fnmatch.fnmatch(os.path.basename(fn), match) 11 here = Dir('.') 12 children = here.all_children() 13 nodes = map(File, filter(fn_filter, children)) 14 node_srcs = [n.srcnode() for n in nodes] 15 src = here.srcnode() 16 if src is not here: 17 src_children = map(File, filter(fn_filter, src.all_children())) 18 for s in src_children: 19 if s not in node_srcs: 20 nodes.append(File(os.path.basename(str(s)))) 21 return nodes
Eventually this should go into SCons proper with test cases and documentation.
Note: Glob currently does not support usages such as Glob('src/*.cpp'). Eventually it should split the match string into directories and use the fnmatch module to glob on subdirectories as well.
(Notes by firstname.lastname@example.org ) Below is the grab function used in my building script:
Here's another version that has includes and excludes:
1 def Matches(file, includes, excludes): 2 match = False 3 for pattern in includes: 4 if fnmatch.fnmatchcase(file, pattern): 5 match = True 6 break 7 if match and excludes is not None: 8 for pattern in excludes: 9 if fnmatch.fnmatchcase(file, pattern): 10 match = False 11 break 12 return match 13 def AddDir(src_dir, includes, excludes=None): 14 files =  15 for file in os.listdir(Dir(src_dir).srcnode().abspath): 16 fqpath = src_dir + '/' + file 17 if os.path.isdir(fqpath): continue #don't include subdirs 18 if Matches(file, includes, excludes): 19 files.append(file) 20 return files 21 used like so: 22 AddDir("c:/mystuff", "*.cpp", excludes=['test*.cpp', '.svn'])
And yet another version: SGlob('*.cpp'), SGlob('Test/*.cpp')
Note: This Version is kind a dirty, but it searches the sourcedirectory, which makes it usable until there is an offical 'the-SCons-way' Glob.
I have written my own version of the SCons Glob when I couldn't get the above to work properly. I can't guarantee it under all conditions but it seems to work for me and handles a few edge cases I have discovered. Brad Phelan's SCons Magic Glob I'll update the linked page as I further test the code.
Q: Just a question: Why isn't glob.glob from the stdlib good?
A: I haven't proven this, but here's a guess. The latest version of this function claims to scan SCons build nodes. If you used glob.glob then you'd get a list of files that existed before the build. If you use the most up-to-date glob above, you'll probably get a list of files that SCons expects to build in the directory, as well as the files that currently exist. That might be important if you're untarring source files, or checking them out from source control, during the build process.
A': Yes, that's exactly right. Another example: if you have a source-generator, you need to glob against all the files the source-generator will generate, not just the files that exist when the build starts.
The glob.glob(mask) results depend on the current directory while SConscripts are supposed to be independent of it. [There is a call SConscriptChdir(enable) that can disable descending into SConscripts' directories at the time of their harvesting]. Changing to Dir(".").abspath to run glob.glob() could be, in my opinion, dangerous in parallel builds if the latter are implemented as threads. --IL
The glob.glob(mask) will produce a list of file paths with inconsistent path separators. That is, if mask is "dir1/dir2/*.cpp", the result in the win32 build of Python will have paths such as "dir1/dir2\\file.cpp". --IL