[Scons-dev] Subprocess issue on Linux?

[Gary Oberbrunner]

I've found posix spawn can be much faster than fork/exec with large memory
processes, so I'd be in favor of this. Not every system has it though so
there would have to be a fallback to fork/exec.

-- 
Gary Oberbrunner
(sent from my Android)
On Apr 1, 2014 11:52 AM, "Kenny, Jason L" <jason.l.kenny at intel.com> wrote:


>

>

> Hi guys,

>

>

>

> I just got a patch to Parts internal at Intel to fix some issues with

> subprocess. I find myself sort of surprised by this, and honestly felt that

> this seems to be an issue that should be looked at in Scons as well.

>

>

>

> The problem is this. We have been building a given huge product here at

> Intel on RHEL 4.8. IT is old and finally time has come to start moving to a

> newer system. What was found when we moved to a newer RHEL 5.10 ( I know

> bleeding edge J ) we found that the build was twice as slow. Looking into

> it more it was found that it was the use of fork() in subprocess. It was

> found that posix_spawn() fixed this. Attached is a monkey patch that is up

> for internal review to modify subprocess module a little to fix this for

> Unix based systems.

>

>

>

> Now I am not an expert on Linux internal details. However I thought fork()

> was the way process got spawned on Linux. Ie fork() was the win32

> CreateProcess(). I am clearly wrong, but this seems to be a problem I would

> not expect to have seen on Linux. The root of the problem seems to be that

> the system takes a long time to setup the fork when you have a large build

> ( and we have a very large build ) because of the memory usage (2-6GB)

> needed by SCons alone. Using a different method to create a process that

> does not try to "clone" memory seems to be the fix that gets the build

> times to be about ~2x+ faster.

>

>

>

> Does anyone else have input on this? IF this is a good patch, it seem that

> we will want to apply it to SCons, for a speed boost.

>

>

>

> Thanks

>

> Jason

>

>

>

>

>

> Also I attached a test file that to show fork() is the problem. You can

> see the difference importing the stubprocess.py file and running the test

> again.

>

> _______________________________________________

> Scons-dev mailing list

> Scons-dev at scons.org

> http://two.pairlist.net/mailman/listinfo/scons-dev

>

>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://two.pairlist.net/pipermail/scons-dev/attachments/20140401/cbf14ba8/attachment.htm>