[Scons-dev] Excessive exec calls at startup

[Jason Kenny]

I make a new environments myself all the time. The trick is to Clone() a new environment, not create a new one from scratch each time you need a new one.

Jason
________________________________
From: Bill Deegan<mailto:bill at baddogconsulting.com>
Sent: ‎11/‎22/‎2015 5:26 PM
To: SCons developer list<mailto:scons-dev at scons.org>
Subject: Re: [Scons-dev] Excessive exec calls at startup

Johan,

SCons is functioning correctly.
It cannot cache the version (at least not as simply as you have implemented
it) because each ENV can specify a different PATH , CC, CXX, or other
variables to specify different tools.

Why are you creating a new Environment for each of your libraries?

-Bill

On Sun, Nov 22, 2015 at 3:15 PM, Johan Holmberg <johan556 at gmail.com> wrote:

> Hi!
>
> While building a project with SCons with 144 different Environment()
> objects (one for each of my libraries), I observed that a lot of external
> processes were created even before SCons did build anything. I use MacOS,
> so I turned to DTrace to snoop on new processes created:
>
>   $ sudo newproc.d > newproc.log
>
> Then I ran "scons non_existing_file" in another terminal, to see what
> SCons did at startup. According to "newproc.d" there are 3486 exec calls
> (new processes). By looking in detail in the log from "newproc.d", I see
> that "gcc --version" is called 290 times, and "g++ --version" another 290
> times (each *two* times per Environment). And each of these 580 processes
> seem to use 6 exec calls internally, like:
>
>     gcc --version
>     /Applications/Xcode.app/Contents/Developer/usr/bin/gcc --version
>     sh -c /usr/bin/xcode-select -print-path 2> /dev/null
>     /usr/bin/xcode-select -print-path
>     /usr/bin/xcrun clang --version
>
> /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang
> --version
>
> I modified the code in "src/engine/SCons/Tool/gcc.py" to cache calls to
> "detect_version" where the same gcc/g++ is asked about its version several
> times. With this change the number of exec calls went from 3486 to 18. And
> the startup time went down from 18 seconds to 8 seconds. (MacOS Yosemite on
> an iMac from 2011).
>
> I'm not very familiar with the SCons source code, so my patch is probably
> not entirely correct. But I hope something similar could be added to SCons
> to avoid wasting time at startup.
>
> Regards,
> /Johan Holmberg
>
> --- a/src/engine/SCons/Tool/gcc.py Fri Nov 20 13:54:09 2015 -0800
> +++ b/src/engine/SCons/Tool/gcc.py Sat Nov 14 17:09:59 2015 +0100
> @@ -63,11 +63,22 @@
>      # is executable, and is a GNU compiler (or accepts '--version' at
> least)
>      return detect_version(env, env.Detect(env.get('CC', compilers)))
>
> +_detected_version_cache = dict()
> +
>  def detect_version(env, cc):
>      """Return the version of the GNU compiler, or None if it is not a GNU
> compiler."""
>      cc = env.subst(cc)
>      if not cc:
>          return None
> +    cc_path = env.WhereIs(cc)
> +    try:
> +        return _detected_version_cache[cc_path]
> +    except KeyError:
> +        version = _detect_version_1(env, cc)
> +        _detected_version_cache[cc_path] = version
> +        return version
> +
> +def _detect_version_1(env, cc):
>      version = None
>      #pipe = SCons.Action._subproc(env, SCons.Util.CLVar(cc) +
> ['-dumpversion'],
>      pipe = SCons.Action._subproc(env, SCons.Util.CLVar(cc) +
> ['--version'],
>
>
> _______________________________________________
> Scons-dev mailing list
> Scons-dev at scons.org
> https://pairlist2.pair.net/mailman/listinfo/scons-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist2.pair.net/pipermail/scons-dev/attachments/20151122/6b7ef48a/attachment-0001.html>
-------------- next part --------------
_______________________________________________
Scons-dev mailing list
Scons-dev at scons.org
https://pairlist2.pair.net/mailman/listinfo/scons-dev