This page presents material that is no longer needed but is being kept for historical purposes. As from version 0.98 SCons has a Glob function built in.
This page shows you how to glob (search files in dir) based on scons Nodes rather than files. This way you get all the files that will be built as well as the ones that already exist in the filesystem.
As of this writing, none of these functions recurse into subdirs, but that's not hard either. Just check each node returned by all_children() and if it's a Dir node, recurse. Here's a trivial one:
1 def print_all_nodes(dirnode, level=0):
2 """Print all the scons nodes that are children of this node, recursively."""
3 if type(dirnode)==type(''):
4 dirnode=Dir(dirnode)
5 dt = type(Dir('.'))
6 for f in dirnode.all_children():
7 if type(f) == dt:
8 print "%s%s: .............."%(level * ' ', str(f))
9 print_dir(f, level+2)
10 print "%s%s"%(level * ' ', str(f))
- Here is another version of Glob, similar to the one below, except it allows multiple excludes and includes, as well as a directory specification. It also deals directly with SCons Nodes.
NOTE: this is the best version available. The others below are mostly of historical interest.
1 import fnmatch
2 import os
3 def Glob( includes = Split( '*' ), excludes = None, dir = '.'):
4 """Similar to glob.glob, except globs SCons nodes, and thus sees
5 generated files and files from build directories. Basically, it sees
6 anything SCons knows about. A key subtlety is that since this function
7 operates on generated nodes as well as source nodes on the filesystem,
8 it needs to be called after builders that generate files you want to
9 include.
10 It will return both Dir entries and File entries
11 """
12 def fn_filter(node):
13 fn = os.path.basename(str(node))
14 match = 0
15 for include in includes:
16 if fnmatch.fnmatchcase( fn, include ):
17 match = 1
18 break
19 if match == 1 and not excludes is None:
20 for exclude in excludes:
21 if fnmatch.fnmatchcase( fn, exclude ):
22 match = 0
23 break
24 return match
25 def filter_nodes(where):
26 children = filter(fn_filter, where.all_children(scan=0))
27 nodes = []
28 for f in children:
29 nodes.append(gen_node(f))
30 return nodes
31 def gen_node(n):
32 """Checks first to see if the node is a file or a dir, then
33 creates the appropriate node. [code seems redundant, if the node
34 is a node, then shouldn't it just be left as is?
35 """
36 if type(n) in (type(''), type(u'')):
37 path = n
38 else:
39 path = n.abspath
40 if os.path.isdir(path):
41 return Dir(n)
42 else:
43 return File(n)
44 here = Dir(dir)
45 nodes = filter_nodes(here)
46 node_srcs = [n.srcnode() for n in nodes]
47 src = here.srcnode()
48 if src is not here:
49 for s in filter_nodes(src):
50 if s not in node_srcs:
51 # Probably need to check if this node is a directory
52 nodes.append(gen_node(os.path.join(dir,os.path.basename(str(s)))))
53 return nodes
You can either include this function directly in your SConscript, or place in an external python script to include in any SConscript it's needed. SCons CVS has a better way to do it, but 0.96.1 requires the use of SConscript( 'external_script.py' ) and Import( ) / Export( )
I don't take much credit for this. Most credit belongs to John Meinel from the SCons mailinglist.
If you want to glob for generated files or for source files in a BuildDir, copy the following function into your project and call Glob instead of glob.glob.
1 def Glob(match):
2 """Similar to glob.glob, except globs SCons nodes, and thus sees
3 generated files and files from build directories. Basically, it sees
4 anything SCons knows about. A key subtlety is that since this function
5 operates on generated nodes as well as source nodes on the filesystem,
6 it needs to be called after builders that generate files you want to
7 include."""
8 def fn_filter(node):
9 fn = str(node)
10 return fnmatch.fnmatch(os.path.basename(fn), match)
11 here = Dir('.')
12 children = here.all_children()
13 nodes = map(File, filter(fn_filter, children))
14 node_srcs = [n.srcnode() for n in nodes]
15 src = here.srcnode()
16 if src is not here:
17 src_children = map(File, filter(fn_filter, src.all_children()))
18 for s in src_children:
19 if s not in node_srcs:
20 nodes.append(File(os.path.basename(str(s))))
21 return nodes
Eventually this should go into SCons proper with test cases and documentation.
Note: Glob currently does not support usages such as Glob('src/*.cpp'). Eventually it should split the match string into directories and use the fnmatch module to glob on subdirectories as well.
(Notes by chenlee@ustc.edu ) Below is the grab function used in my building script:
1 def Glob( pattern = '*.*', dir = '.' ):
2 import os, fnmatch
3 files = []
4 for file in os.listdir( Dir(dir).srcnode().abspath ):
5 if fnmatch.fnmatch(file, pattern) :
6 files.append( os.path.join( dir, file ) )
7 return files
Here's another version that has includes and excludes:
1 def Matches(file, includes, excludes):
2 match = 0
3 for pattern in includes:
4 if fnmatch.fnmatchcase(file, pattern):
5 match = 1
6 break
7 if match == 1 and not excludes is None:
8 for pattern in excludes:
9 if fnmatch.fnmatchcase(file, pattern):
10 match = 0
11 break
12 return match
13 def AddDir(src_dir, includes, excludes=None):
14 files = []
15 for file in os.listdir(Dir(src_dir).srcnode().abspath):
16 fqpath = src_dir + '/' + file
17 if (os.path.isdir(fqpath)): continue #don't include subdirs
18 if Matches(file, includes, excludes):
19 files.append(file)
20 return files
21 used like so:
22 AddDir("c:/mystuff", "*.cpp", excludes=['test*.cpp', '.svn'])
And yet another version: SGlob('*.cpp'), SGlob('Test/*.cpp')
1 def SGlob(pattern):
2 path = string.replace(GetBuildPath('SConscript'), 'SConscript', '')
3 result = []
4 for i in glob.glob(path + pattern):
5 result.append(string.replace(i, path, ''))
6 return result
Note: This Version is kind a dirty, but it searches the sourcedirectory, which makes it usable until there is an offical 'the-SCons-way' Glob.
I have written my own version of the SCons Glob when I couldn't get the above to work properly. I can't guarantee it under all conditions but it seems to work for me and handles a few edge cases I have discovered. Brad Phelan's SCons Magic Glob I'll update the linked page as I further test the code.
Q: Just a question: Why isn't glob.glob from the stdlib good?
A: I haven't proven this, but here's a guess. The latest version of this function claims to scan SCons build nodes. If you used glob.glob then you'd get a list of files that existed before the build. If you use the most up-to-date glob above, you'll probably get a list of files that SCons expects to build in the directory, as well as the files that currently exist. That might be important if you're untarring source files, or checking them out from source control, during the build process.
A': Yes, that's exactly right. Another example: if you have a source-generator, you need to glob against all the files the source-generator will generate, not just the files that exist when the build starts.The glob.glob(mask) results depend on the current directory while SConscripts are supposed to be independent of it. [There is a call SConscriptChdir(enable) that can disable descending into SConscripts' directories at the time of their harvesting]. Changing to Dir(".").abspath to run glob.glob() could be, in my opinion, dangerous in parallel builds if the latter are implemented as threads. --IL
The glob.glob(mask) will produce a list of file paths with inconsistent path separators. That is, if mask is "dir1/dir2/*.cpp", the result in the win32 build of Python will have paths such as "dir1/dir2\\file.cpp". --IL
