Very long command lines on Windows

by PhilMartin

Short version

We found some problems with MingW on windows during the link phase. The command line length could exceed 10000 characters, and this was causing problems.

Just go to the bottom of the page for the code we used to fix it.

Long version (for those interested in the reasons)

When we changed the build process for Aztec to use SCons, we encountered a few problems along the way. Our project contained a number of SharedLibraries and a number of Program's as well, and wanted to build them on Windows, using both Microsoft Visual C and MingW.

MingW had some problems during the link phase of the build process. It turned out our link lines were in excess of 10000 characters, and this was causing some major grief with python calling spawn().

After some investigation, it turns out this is not a limitation of Windows, but a limitation of the C Runtime that MSVC uses (which Python was compiled with). This meant that it was at least fixable. Thanks to the flexible design of SCons, it was not only fixable, but it was very easy to do.

All we need to do was to change the env['SPAWN'] to use our own custom spawn function. We should replace PIPED_SPAWN as well, but that never got called in our build process, so we never got around to it. The fix uses the Win32 API directly, so we avoid the command line length problem entirely.

Here is the code that we placed in our Scons files, and it solves the linking problem with MingW quite well. The interesting part is that it does not handle the standard Windows commands, like del or mkdir. Special consideration has to be made for these.

Here is the final section of code to acheive what we wanted.

Python Code

if env['PLATFORM'] == 'win32':
    import win32file
    import win32event
    import win32process
    import win32security

    def my_spawn(sh, escape, cmd, args, spawnenv):
        for var in spawnenv:
            spawnenv[var] = spawnenv[var].encode('ascii', 'replace')

        sAttrs = win32security.SECURITY_ATTRIBUTES()
        StartupInfo = win32process.STARTUPINFO()
        newargs = ' '.join(map(escape, args[1:]))
        cmdline = cmd + " " + newargs

        # check for any special operating system commands
        if cmd == 'del':
            for arg in args[1:]:
                win32file.DeleteFile(arg)
            exit_code = 0
        else:
            # otherwise execute the command.
            hProcess, hThread, dwPid, dwTid = win32process.CreateProcess(None, cmdline, None, None, 1, 0, spawnenv, None, StartupInfo)
            win32event.WaitForSingleObject(hProcess, win32event.INFINITE)
            exit_code = win32process.GetExitCodeProcess(hProcess)
            win32file.CloseHandle(hProcess);
            win32file.CloseHandle(hThread);
        return exit_code

    env['SPAWN'] = my_spawn


Acknowledgements

Many thanks to Tobias Sargeant for working out most of this, especially the Python side of things.


Remarks

by AdrianNeagu

Some Other Tricks

We had the problem with "command line too long" error message only on Windows 2000 (XP builds would work okay). This issue is "addressed" here: http://support.microsoft.com/default.aspx?scid=kb;en-us;830473

The solution we used was the discovery that ARM toolchain can also parse command line options from file (via file) and add custom builder for building libraries.

by Amit Chakradeo

Another way

You can also use the subprocess module (since Python 2.4).

class ourSpawn:
    def ourspawn(self, sh, escape, cmd, args, env):
        newargs = ' '.join(args[1:])
        cmdline = cmd + " " + newargs
        startupinfo = subprocess.STARTUPINFO()
        startupinfo.dwFlags |= subprocess.STARTF_USESHOWWINDOW
        proc = subprocess.Popen(cmdline, stdin=subprocess.PIPE, stdout=subprocess.PIPE,
            stderr=subprocess.PIPE, startupinfo=startupinfo, shell = False, env = env)
        data, err = proc.communicate()
        rv = proc.wait()
        if rv:
            print "====="
            print err
            print "====="
        return rv

def SetupSpawn( env ):
    if sys.platform == 'win32':
        buf = ourSpawn()
        buf.ourenv = env
        env['SPAWN'] = buf.ourspawn

Call SetupSpawn(env) when you need this. Thanks to xuru on #scons we can now build and link Blender without the need to split large libraries into smaller parts.

by Nathan Letwory (jesterKing)

This works great, except for when there is redirection. Is there a way to improve on this so that it can handle redirection or even if it could go back to the old way if there is redirection?

LongCmdLinesOnWin32 (last edited 2011-06-21 18:17:46 by JoeyMukherjee)