[Scons-dev] Multiprocessing instead of threads?

[Russel Winder]

On Mon, 2015-11-23 at 07:48 -0600, Jason Kenny wrote:
> Ideally threads in python work great, given the code in the threaded
> code is a c-extension and or some subprocess or AIO like call. Pure
> thread python code does not scale because of the GIL.

Well given that threads are no longer the things they were in the
1970s, that they now map to kernel threads, and we have fibres that are
now what threads used to be then… Python and the GIL suck at anything
multi-threaded unless you do a very specific form of blocking IO.

> Given my work At Intel on threaded/MPI systems. Multiprocessing is
> always slower then than threading. There is more overhead and work to
> be done to communicate a simple state change that from a CPU point of
> view is magnitudes slower than what would happen in a thread. Certain
> patterns scale well with multiprocessing library that allow a system
> to go across different machine. These pattern work well in python
> multiprocessing library as it allow for reduced communication and
> scaling of the CPU resources on a given box. SCons however has a
> global DAG state that has to be shared. The one way this could work
> is if the tasking system could use it. The issues is how we share
> state on what is building or finished, output text across different
> processes, dealing with pure python builders, etc. 
> 
> For me is not that this could not be useful… the issue is that it is
> not as simple as one would hope.

The moral of this story is do not even contemplate using Python for
anything where parallelism in a single process is even remotely a good
idea.

D, Chapel, even C++, anything but Python.

-- 
Russel.
=============================================================================
Dr Russel Winder      t: +44 20 7585 2200   voip: sip:russel.winder at ekiga.net
41 Buckmaster Road    m: +44 7770 465 077   xmpp: russel at winder.org.uk
London SW11 1EN, UK   w: www.russel.org.uk  skype: russel_winder

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 181 bytes
Desc: This is a digitally signed message part
URL: <https://pairlist2.pair.net/pipermail/scons-dev/attachments/20151124/8adc435c/attachment.pgp>