[Scons-dev] Subprocess issue on Linux?

[Tom Tanner (BLOOMBERG/ LONDON)]

I think the test round about line 144 needs to be a little different. For instance, aix and solaris (at least the versions we use at work) claim to support posix_spawn and the man pages are identical, but they don't identify themselves as linux

One other question that comes to mind from time to time: do we /really/ need to spawn a shell just to execute a command. Obviously if it include '>' and other shell specials, you should. I'd really like to make it do something like perl:
If you pass an array [ 'prog', '$TARGET', '$SOURCE' ] it doesn't execute a shell. Otherwise (passing a single string), it will examine for special characters, and if none are found it will split on white space and execute that, or it will pass that line to the shell.

NB Yes, I realise this will potentially break things, but why execute a shell if you don't have to?

----- Original Message -----
From: scons-dev at scons.org
To: scons-dev at scons.org
At: Apr  1 2014 17:13:48


I've found posix spawn can be much faster than fork/exec with large memory processes, so I'd be in favor of this. Not every system has it though so there would have to be a fallback to fork/exec. 
-- 
Gary Oberbrunner
(sent from my Android) 
On Apr 1, 2014 11:52 AM, "Kenny, Jason L" <jason.l.kenny at intel.com> wrote:

      

 
Hi guys,
 
I just got a patch to Parts internal at Intel to fix some issues with subprocess. I find myself sort of surprised by this, and honestly felt that this seems to be an issue that should be looked at in Scons as well. 
 
The problem is this. We have been building a given huge product here at Intel on RHEL 4.8. IT is old and finally time has come to start moving to a newer system. What was found when we moved to a newer RHEL 5.10 ( I know bleeding edge J ) we found that the build was twice as slow. Looking into it more it was found that it was the use of fork() in subprocess. It was found that posix_spawn() fixed this. Attached is a monkey patch that is up for internal  review to modify subprocess module a little to fix this for Unix based systems.
 
Now I am not an expert on Linux internal details. However I thought fork() was the way process got spawned on Linux. Ie fork() was the win32 CreateProcess(). I am clearly wrong, but this seems to be a problem I would not expect to have  seen on Linux. The root of the problem seems to be that the system takes a long time to setup the fork when you have a large build ( and we have a very large build ) because of the memory usage (2-6GB) needed by SCons alone. Using a different method to create  a process that does not try to “clone” memory seems to be the fix that gets the build times to be about ~2x+ faster.
 
Does anyone else have input on this? IF this is a good patch, it seem that we will want to apply it to SCons, for a speed boost.
 
Thanks
Jason
 
 
Also I attached a test file that to show fork() is the problem. You can see the difference importing the stubprocess.py file and running the test again.
_______________________________________________
Scons-dev mailing list
Scons-dev at scons.org
http://two.pairlist.net/mailman/listinfo/scons-dev


_______________________________________________ Scons-dev mailing list Scons-dev at scons.org http://two.pairlist.net/mailman/listinfo/scons-dev 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://two.pairlist.net/pipermail/scons-dev/attachments/20140402/05d10416/attachment.html>