[Scons-dev] Fix async subprocess execution with proper handling of std* streams

[anatoly techtonik]

Thanks. That would be great. ;)

On Tue, Feb 18, 2014 at 5:21 PM, Kenny, Jason L <jason.l.kenny at intel.com> wrote:

> I will get it pushed… life getting in the way.

>

>

>

> From: scons-dev-bounces at scons.org [mailto:scons-dev-bounces at scons.org] On

> Behalf Of Eugene Leskinen

> Sent: Tuesday, February 18, 2014 7:03 AM

> To: SCons developer list

> Subject: Re: [Scons-dev] Fix async subprocess execution with proper handling

> of std* streams

>

>

>

> Hi Anatoly,

>

>

>

> The code supporting I/O redirection in Parts is located in

> parts/part_logger.py file. Look at

> http://parts.tigris.org/source/browse/parts/trunk/parts/parts/part_logger.py?revision=486&view=markup

>

>

>

> The code uses Python threading module to create threads for each stream and

> then runs a loop in every thready like your code does:

>

>

>

>         line = ' '

>

>         try:

>

>             while line:

>

>                 line = self.pipein.readline()

>

>                 if line:

>

>                     self.output.WriteStream(self.taskId, self.streamId,

> line)

>

>         except:

>

>             # There was an error... that shouldn't happen, but still it did.

> So we report it

>

>             # to the caller and close our pipe end so that spawned program

> won't block

>

>             self.error = traceback.format_exc()

>

>             self.pipein.close()

>

>             self.pipein = None

>

>

>

> Please note that public repo is a bit out-of-date. I hope Jason will find

> time to pull up-to-date version from Intel internal repository.

>

>

>

> 2014-02-18 16:30 GMT+04:00 anatoly techtonik <techtonik at gmail.com>:

>

> On Wed, Feb 12, 2014 at 6:25 PM, Kenny, Jason L <jason.l.kenny at intel.com>

> wrote:

>>> Fix async subprocess execution with proper handling of std* streams.

>>

>> So I fixed the issue with output in Parts. I not saying it could not be

>> clean up some more. However it solves the output issues, and adds coloring

>> (

>> which seems to need some fixing in certain cases.. ie mac mostly) and

>> logging support. It might not be too hard for someone to move the code

>> over

>> to SCons and integrated its usage into SCons.

>

> Do you have, by any chance a description of output issues?

> It seems that I've lost my list during the last Windows partition crash.

>

> Also, can you pinpoint the parts in Parts that are relevant to the issues?

> I may have time to port them over to SCons.

>

> My approach

> https://bitbucket.org/techtonik/absub

> to make subprocess asynchronous by moving processing

> of input/output streams into separate threads. I don't remember why, but the

> work stalled. I remember there was some kind of error that I was not able to

> debug due to distributed nature of the process. I needed to write more tests

> for that, but it is either I run out of time, multithreaded tests were

> complicated

> or I just had an annual Windows filesystem crash.

>

> Now I am thinking about replacing concept of input/output streams with the

> concept of channel, which is a queue of strings with limited size. The queue

> can be rotated to become unlimited. Can be synchronous to wait (block)

> until messages are read if the queue is full. This may help to clear

> confusion

> around all that input output streams handling, pipes and descriptor

> inheritances.

>

> --

> anatoly t.

> _______________________________________________

> Scons-dev mailing list

> Scons-dev at scons.org

> http://two.pairlist.net/mailman/listinfo/scons-dev

>

>

>

>

>

> --

> Regards,

> Eugene Leskinen

>

>

> _______________________________________________

> Scons-dev mailing list

> Scons-dev at scons.org

> http://two.pairlist.net/mailman/listinfo/scons-dev

>




-- 
anatoly t.