[Scons-dev] Excessive exec calls at startup

[Ivan Nedrehagen]

Just an idea, as creating loads of new environments aren't unusual.

Perhaps caching the version like this would be impossible, but how about  
caching results based on executable paths and arguments?

Regards,
Ivan Nedrehagen

On Mon, 23 Nov 2015 00:26:00 +0100, Bill Deegan  
<bill at baddogconsulting.com> wrote:

> Johan,
>
> SCons is functioning correctly.
> It cannot cache the version (at least not as simply as you have  
> implemented it) because each ENV can specify a different PATH , CC, CXX,  
> or other >variables to specify different tools.
>
> Why are you creating a new Environment for each of your libraries?
>
> -Bill
>
> On Sun, Nov 22, 2015 at 3:15 PM, Johan Holmberg <johan556 at gmail.com>  
> wrote:
>> Hi!
>>
>> While building a project with SCons with 144 different Environment()  
>> objects (one for each of my libraries), I observed that a lot of  
>> external >>processes were created even before SCons did build anything.  
>> I use MacOS, so I turned to DTrace to snoop on new processes created:
>>
>>  $ sudo newproc.d > newproc.log
>>
>> Then I ran "scons non_existing_file" in another terminal, to see what  
>> SCons did at startup. According to "newproc.d" there are 3486 exec  
>> calls (new >>processes). By looking in detail in the log from  
>> "newproc.d", I see that "gcc --version" is called 290 times, and "g++  
>> --version" another 290 times >>(each *two* times per Environment). And  
>> each of these 580 processes seem to use 6 exec calls internally, like:
>>
>>    gcc --version
>>    /Applications/Xcode.app/Contents/Developer/usr/bin/gcc --version
>>    sh -c /usr/bin/xcode-select -print-path 2> /dev/null
>>    /usr/bin/xcode-select -print-path
>>    /usr/bin/xcrun clang --version
>>    /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang  
>> --version
>>
>> I modified the code in "src/engine/SCons/Tool/gcc.py" to cache calls to  
>> "detect_version" where the same gcc/g++ is asked about its version  
>> >>several times. With this change the number of exec calls went from  
>> 3486 to 18. And the startup time went down from 18 seconds to 8  
>> seconds. >>(MacOS Yosemite on an iMac from 2011).
>> I'm not very familiar with the SCons source code, so my patch is  
>> probably not entirely correct. But I hope something similar could be  
>> added to >>SCons to avoid wasting time at startup.
>>
>> Regards,
>> /Johan Holmberg
>>
>> --- a/src/engine/SCons/Tool/gcc.py	Fri Nov 20 13:54:09 2015 -0800
>> +++ b/src/engine/SCons/Tool/gcc.py	Sat Nov 14 17:09:59 2015 +0100
>> @@ -63,11 +63,22 @@
>>     # is executable, and is a GNU compiler (or accepts '--version' at  
>> least)
>>     return detect_version(env, env.Detect(env.get('CC', compilers)))
>>+_detected_version_cache = dict()
>> +
>> def detect_version(env, cc):
>>     """Return the version of the GNU compiler, or None if it is not a  
>> GNU compiler."""
>>     cc = env.subst(cc)
>>     if not cc:
>>         return None
>> +    cc_path = env.WhereIs(cc)
>> +    try:
>> +        return _detected_version_cache[cc_path]
>> +    except KeyError:
>> +        version = _detect_version_1(env, cc)
>> +        _detected_version_cache[cc_path] = version
>> +        return version
>> +
>> +def _detect_version_1(env, cc):
>>     version = None
>>     #pipe = SCons.Action._subproc(env, SCons.Util.CLVar(cc) +  
>> ['-dumpversion'],
>>     pipe = SCons.Action._subproc(env, SCons.Util.CLVar(cc) +  
>> ['--version'],
>>
>>
>> _______________________________________________
>> Scons-dev mailing list
>> Scons-dev at scons.org
>> https://pairlist2.pair.net/mailman/listinfo/scons-dev
>>
>



-- 
Using Opera's mail client: http://www.opera.com/mail/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist2.pair.net/pipermail/scons-dev/attachments/20151123/1e11ce5c/attachment.html>