[Scons-dev] Excessive exec calls at startup

[Jason Kenny]

I think this is an idea we all have, and never fully made it.

If you use Parts with Scons to do the build the logic in Parts is to Clone() be default and the toolchain logic be design checks the versions only once and reuses it between different environment creations. I have not had time to refactor some stuff yet to get it to cache this state so it would exist for the next run as there are details that have to added to allow rescanning correctly if the tools changed. Nothing hard… just a time issue.

Jason

Sent from Mail for Windows 10



From: Ivan Nedrehagen
Sent: Monday, November 23, 2015 4:57 AM
To: SCons developer list;Bill Deegan
Subject: Re: [Scons-dev] Excessive exec calls at startup


Just an idea, as creating loads of new environments aren't unusual.

Perhaps caching the version like this would be impossible, but how about caching results based on executable paths and arguments?

Regards,
Ivan Nedrehagen

On Mon, 23 Nov 2015 00:26:00 +0100, Bill Deegan <bill at baddogconsulting.com> wrote:

Johan,
SCons is functioning correctly.
It cannot cache the version (at least not as simply as you have implemented it) because each ENV can specify a different PATH , CC, CXX, or other variables to specify different tools.
Why are you creating a new Environment for each of your libraries?

-Bill

On Sun, Nov 22, 2015 at 3:15 PM, Johan Holmberg <johan556 at gmail.com> wrote:
Hi!

While building a project with SCons with 144 different Environment() objects (one for each of my libraries), I observed that a lot of external processes were created even before SCons did build anything. I use MacOS, so I turned to DTrace to snoop on new processes created:

  $ sudo newproc.d > newproc.log

Then I ran "scons non_existing_file" in another terminal, to see what SCons did at startup. According to "newproc.d" there are 3486 exec calls (new processes). By looking in detail in the log from "newproc.d", I see that "gcc --version" is called 290 times, and "g++ --version" another 290 times (each *two* times per Environment). And each of these 580 processes seem to use 6 exec calls internally, like:

    gcc --version
    /Applications/Xcode.app/Contents/Developer/usr/bin/gcc --version
    sh -c /usr/bin/xcode-select -print-path 2> /dev/null
    /usr/bin/xcode-select -print-path
    /usr/bin/xcrun clang --version
    /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang --version

I modified the code in "src/engine/SCons/Tool/gcc.py" to cache calls to "detect_version" where the same gcc/g++ is asked about its version several times. With this change the number of exec calls went from 3486 to 18. And the startup time went down from 18 seconds to 8 seconds. (MacOS Yosemite on an iMac from 2011). 

I'm not very familiar with the SCons source code, so my patch is probably not entirely correct. But I hope something similar could be added to SCons to avoid wasting time at startup.

Regards,
/Johan Holmberg

--- a/src/engine/SCons/Tool/gcc.py Fri Nov 20 13:54:09 2015 -0800
+++ b/src/engine/SCons/Tool/gcc.py Sat Nov 14 17:09:59 2015 +0100
@@ -63,11 +63,22 @@
     # is executable, and is a GNU compiler (or accepts '--version' at least)
     return detect_version(env, env.Detect(env.get('CC', compilers)))
 
+_detected_version_cache = dict()
+
 def detect_version(env, cc):
     """Return the version of the GNU compiler, or None if it is not a GNU compiler."""
     cc = env.subst(cc)
     if not cc:
         return None
+    cc_path = env.WhereIs(cc)
+    try:
+        return _detected_version_cache[cc_path]
+    except KeyError:
+        version = _detect_version_1(env, cc)
+        _detected_version_cache[cc_path] = version
+        return version
+
+def _detect_version_1(env, cc):
     version = None
     #pipe = SCons.Action._subproc(env, SCons.Util.CLVar(cc) + ['-dumpversion'],
     pipe = SCons.Action._subproc(env, SCons.Util.CLVar(cc) + ['--version'],


_______________________________________________
Scons-dev mailing list
Scons-dev at scons.org
https://pairlist2.pair.net/mailman/listinfo/scons-dev



-- 
Using Opera's mail client: http://www.opera.com/mail/


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist2.pair.net/pipermail/scons-dev/attachments/20151123/f5d845fc/attachment-0001.html>