Please note:The SCons wiki is now restored from the attack in March 2013. All old passwords have been invalidated. Please reset your password if you have an account. If you note missing pages, please report them to webmaster@scons.org. Also, new account creation is currently disabled due to an ongoing spam flood (2013/08/27).

Doxygen Builder

The Doxygen builder will generate doxygen docs, given a Doxyfile. It will scan the Doxyfile and determine what directories will be created and what sources are used to generate the docs. This frees you up from writing special code to manage clean up and regeneration of the docs.

N.B. It seems there was a bug in scons versions before 0.97.0d20070918 which prevented dependencies from working for this builder. See this email to the mailing list for details.

Russel Winder has started a Bazaar branch on Launchpad to try and acrete all the work found on and from this page into a single good tool. You may want to use this version rather than trying to replicate the various changes needed to reconcile all the variations reported on this page. If you find any errors or improvements please contribute them back via a pull request rather than posting code to this page. Thanks.

The above now appears to have moved to a Mercurial branch on Bitbucket: https://bitbucket.org/russel/scons_doxygen.

Please use the Bazaar Mercurial branch above and not any of the codes below, which are left here just to preserve the historical record.

Usage

Check out the Mercurial repository into site_scons/site_tools/doxygen. Then, in your SConstruct file:

   1 # scons buildfile
   2 
   3 # the doxygen package file needs to be in toolpath
   4 env = Environment(tools = ["default", "doxygen"])
   5 env.Doxygen("Doxyfile")

Historical instructions

The original instructions for use are given below.

Save the following script as file 'doxygen.py' and put its directory in the 'toolpath' list as shown in "Usage" below.

   1 # vim: set et sw=3 tw=0 fo=awqorc ft=python:
   2 #
   3 # Astxx, the Asterisk C++ API and Utility Library.
   4 # Copyright (C) 2005, 2006  Matthew A. Nicholson
   5 # Copyright (C) 2006  Tim Blechmann
   6 #
   7 # This library is free software; you can redistribute it and/or
   8 # modify it under the terms of the GNU Lesser General Public
   9 # License version 2.1 as published by the Free Software Foundation.
  10 #
  11 # This library is distributed in the hope that it will be useful,
  12 # but WITHOUT ANY WARRANTY; without even the implied warranty of
  13 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
  14 # Lesser General Public License for more details.
  15 #
  16 # You should have received a copy of the GNU Lesser General Public
  17 # License along with this library; if not, write to the Free Software
  18 # Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
  19 
  20 import os
  21 import os.path
  22 import glob
  23 from fnmatch import fnmatch
  24 
  25 def DoxyfileParse(file_contents):
  26    """
  27    Parse a Doxygen source file and return a dictionary of all the values.
  28    Values will be strings and lists of strings.
  29    """
  30    data = {}
  31 
  32    import shlex
  33    lex = shlex.shlex(instream = file_contents, posix = True)
  34    lex.wordchars += "*+./-:"
  35    lex.whitespace = lex.whitespace.replace("\n", "")
  36    lex.escape = ""
  37 
  38    lineno = lex.lineno
  39    token = lex.get_token()
  40    key = token   # the first token should be a key
  41    last_token = ""
  42    key_token = False
  43    next_key = False
  44    new_data = True
  45 
  46    def append_data(data, key, new_data, token):
  47       if new_data or len(data[key]) == 0:
  48          data[key].append(token)
  49       else:
  50          data[key][-1] += token
  51 
  52    while token:
  53       if token in ['\n']:
  54          if last_token not in ['\\']:
  55             key_token = True
  56       elif token in ['\\']:
  57          pass
  58       elif key_token:
  59          key = token
  60          key_token = False
  61       else:
  62          if token == "+=":
  63             if key not in data:
  64                data[key] = []
  65          elif token == "=":
  66             data[key] = []
  67          else:
  68             append_data( data, key, new_data, token )
  69             new_data = True
  70 
  71       last_token = token
  72       token = lex.get_token()
  73 
  74       if last_token == '\\' and token != '\n':
  75          new_data = False
  76          append_data( data, key, new_data, '\\' )
  77 
  78    # compress lists of len 1 into single strings
  79    for k, v in data.items():
  80       if len(v) == 0:
  81          data.pop(k)
  82 
  83       # items in the following list will be kept as lists and not converted to strings
  84       if k in ["INPUT", "FILE_PATTERNS", "EXCLUDE_PATTERNS"]:
  85          continue
  86 
  87       if len(v) == 1:
  88          data[k] = v[0]
  89 
  90    return data
  91 
  92 def DoxySourceScan(node, env, path):
  93    """
  94    Doxygen Doxyfile source scanner.  This should scan the Doxygen file and add
  95    any files used to generate docs to the list of source files.
  96    """
  97    default_file_patterns = [
  98       '*.c', '*.cc', '*.cxx', '*.cpp', '*.c++', '*.java', '*.ii', '*.ixx',
  99       '*.ipp', '*.i++', '*.inl', '*.h', '*.hh ', '*.hxx', '*.hpp', '*.h++',
 100       '*.idl', '*.odl', '*.cs', '*.php', '*.php3', '*.inc', '*.m', '*.mm',
 101       '*.py',
 102    ]
 103 
 104    default_exclude_patterns = [
 105       '*~',
 106    ]
 107 
 108    sources = []
 109 
 110    data = DoxyfileParse(node.get_contents())
 111 
 112    recursive = data.get("RECURSIVE") == "YES"
 113 
 114    file_patterns = data.get("FILE_PATTERNS", default_file_patterns)
 115    exclude_patterns = data.get("EXCLUDE_PATTERNS", default_exclude_patterns)
 116 
 117    for node in data.get("INPUT", []):
 118       if os.path.isfile(node):
 119          sources.append(node)
 120       elif os.path.isdir(node):
 121          if recursive:
 122             for root, dirs, files in os.walk(node):
 123                for f in files:
 124                   filename = os.path.join(root, f)
 125 
 126                   pattern_check = any(fnmatch(filename, y) for y in file_patterns)
 127                   exclude_check = any(fnmatch(filename, y) for y in exclude_patterns)
 128 
 129                   if pattern_check and not exclude_check:
 130                      sources.append(filename)
 131          else:
 132             for pattern in file_patterns:
 133                sources.extend(glob.glob("/".join([node, pattern])))
 134 
 135    sources = [env.File(path) for path in sources]
 136    return sources
 137 
 138 
 139 def DoxySourceScanCheck(node, env):
 140    """Check if we should scan this file"""
 141    return os.path.isfile(node.path)
 142 
 143 def DoxyEmitter(source, target, env):
 144    """Doxygen Doxyfile emitter"""
 145    # possible output formats and their default values and output locations
 146    output_formats = {
 147       "HTML": ("YES", "html"),
 148       "LATEX": ("YES", "latex"),
 149       "RTF": ("NO", "rtf"),
 150       "MAN": ("NO", "man"),
 151       "XML": ("NO", "xml"),
 152    }
 153 
 154    data = DoxyfileParse(source[0].get_contents())
 155 
 156    targets = []
 157    out_dir = data.get("OUTPUT_DIRECTORY", ".")
 158 
 159    # add our output locations
 160    for k, v in output_formats.items():
 161       if data.get("GENERATE_" + k, v[0]) == "YES":
 162          targets.append(env.Dir( os.path.join(out_dir, data.get(k + "_OUTPUT", v[1]))) )
 163 
 164    # don't clobber targets
 165    for node in targets:
 166       env.Precious(node)
 167 
 168    # set up cleaning stuff
 169    for node in targets:
 170       env.Clean(node, node)
 171 
 172    return (targets, source)
 173 
 174 def generate(env):
 175    """
 176    Add builders and construction variables for the
 177    Doxygen tool.  This is currently for Doxygen 1.4.6.
 178    """
 179    doxyfile_scanner = env.Scanner(
 180       DoxySourceScan,
 181       "DoxySourceScan",
 182       scan_check = DoxySourceScanCheck,
 183    )
 184 
 185    import SCons.Builder
 186    doxyfile_builder = SCons.Builder.Builder(
 187       action = "cd ${SOURCE.dir}  &&  ${DOXYGEN} ${SOURCE.file}",
 188       emitter = DoxyEmitter,
 189       target_factory = env.fs.Entry,
 190       single_source = True,
 191       source_scanner =  doxyfile_scanner,
 192    )
 193 
 194    env.Append(BUILDERS = {
 195       'Doxygen': doxyfile_builder,
 196    })
 197 
 198    env.AppendUnique(
 199       DOXYGEN = 'doxygen',
 200    )
 201 
 202 def exists(env):
 203    """
 204    Make sure doxygen exists.
 205    """
 206    return env.Detect("doxygen")

Note added by Robert Lupton, rhl@astro.princeton.edu

I had to make two changes to make this work.

1. I had to double the $ in the Action:

2. As written, the Builder runs from the top level directory TOP when it scans the doxyfile, but runs doxygen from the source directory. This means that it you set INPUT to e.g. "..", the scanner will set the dependencies to refer to all files found by searching TOP/.. --- which isn't what you want!

Here's a fix (around line 122):

   #
   # We're running in the top-level directory, but the doxygen
   # configuration file is in the same directory as node; this means
   # that relative pathnames in node must be adjusted before they can
   # go onto the sources list
   #
   conf_dir = os.path.dirname(str(node))

   for node in data.get("INPUT", []):
      if not os.path.isabs(node):
         node = os.path.join(conf_dir, node)

Note added by SK

The code above originally had the following initialization of the action = argument when creating the Builder:

      action = env.Action("cd ${SOURCE.dir}  &&  ${DOXYGEN} ${SOURCE.file}"),

The env.Action() call explicitly asks for the string to be evaluated at call time, when the action is created, which is why Robert found it necessary to double the $ characters. (It probably did work in earlier versions, but variable substitution in construction environment methods has been "cleaned up" in some recent versions, and this may have been a casualty.)

Since there's nothing special about the action being created (no strfunction, for example), it's much simpler to just pass the command-line string to the Builder and let SCons create the Action object.

      action = "cd ${SOURCE.dir}  &&  ${DOXYGEN} ${SOURCE.file}",

I updated the code above so that people who cut and paste without reading all the way to the bottom of the page shouldn't have this problem.

Additional update 6 March 2007: There was also a left-over env.Builder that had to be changed to the raw form of the call to avoid variable expansion earlier than we want. Code above changed.

Note added by Dirk Reiners, dirk@louisiana.edu

I added two (at least for me ;)) important features of doxygen: variable substituion and hierarchical doxygen files.

Variable substituion allows doxygen to reference variables from the scons environment using $(VARNAME). This is very useful for things like version numbers or for only having certain parts (as defined by scons) included in the documentation without having to mess with doxygen files.

Hierarchical doxygen files just interpret the @INCLUDE key as an include.

I also had trouble with files that started with a key, I fixed that.

The changes are a little longish for putting them in the text, so I attached the changed file doxygen.py_dr_070226. Note that I'm a python newbie, so there are probably more elegant ways to do some of the things I did. Feel free to change them.

Hope it helps.

Note added by anonymous

Replace the line token = env[token[2:-1]] by token = env[token[2:token.find(")")]] to suppress wrong warnings when using environment variables in Doxyfile as path. (Like in "$(MY_LIBRARY)/include")

Note added by Christoph Boehme, cxb632@bham.ac.uk

Robert Lupton noted that you have to change the source paths if you keep your Doxyfile in a subdirectory and use relative paths. I found that I had to do the same for the target path in the Doxyfile. Therefore, I added the following lines after line 160:

   1    if not os.path.isabs(out_dir):
   2       conf_dir = os.path.dirname(str(source[0]))
   3       out_dir = os.path.join(conf_dir, out_dir)

This is essentially the same code as Robert Lupton's.

Adding tagfile to targets and html templates to sources

The following code adds the tagfile to the target list. I added it in line 166:

   1    # add the tag file if neccessary:
   2    tagfile = data.get("GENERATE_TAGFILE", "")
   3    if tagfile != "":
   4       if not os.path.isabs(tagfile):
   5          conf_dir = os.path.dirname(str(source[0]))
   6          tagfile = os.path.join(conf_dir, tagfile)
   7       targets.append(env.File(tagfile));

To add the html templates from the Doxyfile to the list of sources, you need to apply Robert Lupton's change and add the following snippet in line 137:

   1    # Add additional files to the list ouf source files:
   2    def append_additional_source(option):
   3       file = data.get(option, "")
   4       if file != "":
   5          if not os.path.isabs(file):
   6             file = os.path.join(conf_dir, file)
   7          if os.path.isfile(file):
   8             sources.append(file)
   9 
  10    append_additional_source("HTML_STYLESHEET")
  11    append_additional_source("HTML_HEADER")
  12    append_additional_source("HTML_FOOTER")

You can easily add dependencies on other output file templates by adding additional calls to append_additional_source().

Addendum 18 July 2007: I added some code to add tagfiles to the list of sources. Since the tagfiles-option allows for equal-signs in the value, I had to change the parsing code a bit. The new code is found in file doxygen.py . This file also includes the other changes I have made.

Note added by Reinderien

I believe that the line

"MAN": ("YES", "man"),

should read

"MAN": ("NO", "man"),

I was getting unnecessary doxygen runs, and scons --debug-explain showed that doxygen.py thinks the man target is on by default when it isn't.

DoxygenBuilder (last edited 2013-08-12 11:09:39 by RichardVanDerHoff)