[Scons-dev] Fix async subprocess execution with proper handling of std* streams

[anatoly techtonik]

That's cool. The document is a very good starting point. To me this
streams framework is a very useful component that should be merged
into provisional packages of Python standard library. I still need to
find time to read it thoroughly to adapt to what I know.

On Mon, Feb 24, 2014 at 9:57 PM, Kenny, Jason L <jason.l.kenny at intel.com> wrote:

> Hi Anatoly,

>

> Attached is a word document with the overview of the deign I have with thoughts on some good and bad of it. Hopefully I proofed the document enough, document writing is not my strong suit. Anyways let me know if you have any other questions, etc.. on the design.

>

> Eugene who has been helping with Parts work inside thinks he fixed a Scons/Parts rebuild issues we have been dealing with. I testing it out, if it looks good I will push this drop in a few days.

>

> Thanks for looking at this!

> Jason

>

> -----Original Message-----

> From: scons-dev-bounces at scons.org [mailto:scons-dev-bounces at scons.org] On Behalf Of anatoly techtonik

> Sent: Tuesday, February 18, 2014 10:35 AM

> To: SCons developer list

> Subject: Re: [Scons-dev] Fix async subprocess execution with proper handling of std* streams

>

> Thanks. That would be great. ;)

>

> On Tue, Feb 18, 2014 at 5:21 PM, Kenny, Jason L <jason.l.kenny at intel.com> wrote:

>> I will get it pushed… life getting in the way.

>>

>>

>>

>> From: scons-dev-bounces at scons.org [mailto:scons-dev-bounces at scons.org]

>> On Behalf Of Eugene Leskinen

>> Sent: Tuesday, February 18, 2014 7:03 AM

>> To: SCons developer list

>> Subject: Re: [Scons-dev] Fix async subprocess execution with proper

>> handling of std* streams

>>

>>

>>

>> Hi Anatoly,

>>

>>

>>

>> The code supporting I/O redirection in Parts is located in

>> parts/part_logger.py file. Look at

>> http://parts.tigris.org/source/browse/parts/trunk/parts/parts/part_log

>> ger.py?revision=486&view=markup

>>

>>

>>

>> The code uses Python threading module to create threads for each

>> stream and then runs a loop in every thready like your code does:

>>

>>

>>

>>         line = ' '

>>

>>         try:

>>

>>             while line:

>>

>>                 line = self.pipein.readline()

>>

>>                 if line:

>>

>>                     self.output.WriteStream(self.taskId,

>> self.streamId,

>> line)

>>

>>         except:

>>

>>             # There was an error... that shouldn't happen, but still it did.

>> So we report it

>>

>>             # to the caller and close our pipe end so that spawned

>> program won't block

>>

>>             self.error = traceback.format_exc()

>>

>>             self.pipein.close()

>>

>>             self.pipein = None

>>

>>

>>

>> Please note that public repo is a bit out-of-date. I hope Jason will

>> find time to pull up-to-date version from Intel internal repository.

>>

>>

>>

>> 2014-02-18 16:30 GMT+04:00 anatoly techtonik <techtonik at gmail.com>:

>>

>> On Wed, Feb 12, 2014 at 6:25 PM, Kenny, Jason L

>> <jason.l.kenny at intel.com>

>> wrote:

>>>> Fix async subprocess execution with proper handling of std* streams.

>>>

>>> So I fixed the issue with output in Parts. I not saying it could not

>>> be clean up some more. However it solves the output issues, and adds

>>> coloring ( which seems to need some fixing in certain cases.. ie mac

>>> mostly) and logging support. It might not be too hard for someone to

>>> move the code over to SCons and integrated its usage into SCons.

>>

>> Do you have, by any chance a description of output issues?

>> It seems that I've lost my list during the last Windows partition crash.

>>

>> Also, can you pinpoint the parts in Parts that are relevant to the issues?

>> I may have time to port them over to SCons.

>>

>> My approach

>> https://bitbucket.org/techtonik/absub

>> to make subprocess asynchronous by moving processing of input/output

>> streams into separate threads. I don't remember why, but the work

>> stalled. I remember there was some kind of error that I was not able

>> to debug due to distributed nature of the process. I needed to write

>> more tests for that, but it is either I run out of time, multithreaded

>> tests were complicated or I just had an annual Windows filesystem

>> crash.

>>

>> Now I am thinking about replacing concept of input/output streams with

>> the concept of channel, which is a queue of strings with limited size.

>> The queue can be rotated to become unlimited. Can be synchronous to

>> wait (block) until messages are read if the queue is full. This may

>> help to clear confusion around all that input output streams handling,

>> pipes and descriptor inheritances.

>>

>> --

>> anatoly t.

>> _______________________________________________

>> Scons-dev mailing list

>> Scons-dev at scons.org

>> http://two.pairlist.net/mailman/listinfo/scons-dev

>>

>>

>>

>>

>>

>> --

>> Regards,

>> Eugene Leskinen

>>

>>

>> _______________________________________________

>> Scons-dev mailing list

>> Scons-dev at scons.org

>> http://two.pairlist.net/mailman/listinfo/scons-dev

>>

>

>

>

> --

> anatoly t.

> _______________________________________________

> Scons-dev mailing list

> Scons-dev at scons.org

> http://two.pairlist.net/mailman/listinfo/scons-dev

>

> _______________________________________________

> Scons-dev mailing list

> Scons-dev at scons.org

> http://two.pairlist.net/mailman/listinfo/scons-dev

>




-- 
anatoly t.