[Scons-dev] Fix async subprocess execution with proper handling of std* streams

[Eugene Leskinen]

Hi Anatoly,

The code supporting I/O redirection in Parts is located in
parts/part_logger.py file. Look at
http://parts.tigris.org/source/browse/parts/trunk/parts/parts/part_logger.py?revision=486&view=markup

The code uses Python threading module to create threads for each stream and
then runs a loop in every thready like your code does:

        line = ' '
        try:
            while line:
                line = self.pipein.readline()
                if line:
                    self.output.WriteStream(self.taskId, self.streamId,
line)
        except:
            # There was an error... that shouldn't happen, but still it
did. So we report it
            # to the caller and close our pipe end so that spawned program
won't block
            self.error = traceback.format_exc()
            self.pipein.close()
            self.pipein = None

Please note that public repo is a bit out-of-date. I hope Jason will find
time to pull up-to-date version from Intel internal repository.


2014-02-18 16:30 GMT+04:00 anatoly techtonik <techtonik at gmail.com>:


> On Wed, Feb 12, 2014 at 6:25 PM, Kenny, Jason L <jason.l.kenny at intel.com>

> wrote:

> >> Fix async subprocess execution with proper handling of std* streams.

> >

> > So I fixed the issue with output in Parts. I not saying it could not be

> > clean up some more. However it solves the output issues, and adds

> coloring (

> > which seems to need some fixing in certain cases.. ie mac mostly) and

> > logging support. It might not be too hard for someone to move the code

> over

> > to SCons and integrated its usage into SCons.

>

> Do you have, by any chance a description of output issues?

> It seems that I've lost my list during the last Windows partition crash.

>

> Also, can you pinpoint the parts in Parts that are relevant to the issues?

> I may have time to port them over to SCons.

>

> My approach

> https://bitbucket.org/techtonik/absub

> to make subprocess asynchronous by moving processing

> of input/output streams into separate threads. I don't remember why, but

> the

> work stalled. I remember there was some kind of error that I was not able

> to

> debug due to distributed nature of the process. I needed to write more

> tests

> for that, but it is either I run out of time, multithreaded tests were

> complicated

> or I just had an annual Windows filesystem crash.

>

> Now I am thinking about replacing concept of input/output streams with the

> concept of channel, which is a queue of strings with limited size. The

> queue

> can be rotated to become unlimited. Can be synchronous to wait (block)

> until messages are read if the queue is full. This may help to clear

> confusion

> around all that input output streams handling, pipes and descriptor

> inheritances.

>

> --

> anatoly t.

> _______________________________________________

> Scons-dev mailing list

> Scons-dev at scons.org

> http://two.pairlist.net/mailman/listinfo/scons-dev

>




-- 
Regards,
Eugene Leskinen
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://two.pairlist.net/pipermail/scons-dev/attachments/20140218/f77c152e/attachment.html>