[Scons-dev] Subprocess.popen() on win32?

Sun Oct 7 16:51:50 EDT 2018

On Linux I did not use the escaping function It seemed to make it worse. Popen has some code that does escaping.

>> Do we lose redirects with swapping in the Popen() ?

I was using Popen as I could redirect. The current code might redirect, but there is no control over how it works that I understand…

If you look at:
https://bitbucket.org/sconsparts/parts/src/master/parts/part_logger.py

you will see what is  do. There is more code to support this and honestly there is some clean up to do. But it works correctly. There is basically a pipe redirector class that exists and is passed on to the stdout stderr pipes of Popen() each of these are running in a thread and do nothing but grab output that is then given to a reporter class ( this is the area of have some clean up to do) that allows all output to be written to the console and to go through the logger classes I have to output the text to a given log file ( in my case I have per component loggers ( ie for the commands that happen in a given component) and a global logger that has everything as it comes. The only effect this really has on the output is that -j builds will print whole lines of text, vs mixed up lines of text from different lines.

Oh I also see I have an updated killtree functions to shut down better and a timeout value ( so a command does not lock up the build, which is great for running tests). I have an updated version of the kill logic I have to add to this in a different project of mine I need to sync with this one.

Jason

From: Scons-dev <scons-dev-bounces at scons.org> On Behalf Of Bill Deegan
Sent: Sunday, October 7, 2018 2:47 PM
To: SCons developer list <scons-dev at scons.org>
Subject: Re: [Scons-dev] Subprocess.popen() on win32?

On Sun, Oct 7, 2018 at 2:25 PM Jason Kenny <dragon512 at live.com<mailto:dragon512 at live.com>> wrote:

I can see what I can do here. I have to make a new Drop of Parts first ( having a regression in my code I have to fix )

I don’t believe there should be any limitation. The only issue I believe I had was with moving to python 27 as there was a change from 26 in quote handling. However we are on 2.7 or better so this should not be an issue. Oh there was one other thing. I have to ignore the “escape” function that is passed in the SPAWN callback. It breaks with 2.7 on Window and Linux. I also removed the Scons override for the open call as it was messing up writing to the console device ( Plus I needed to override it to allow symbolic and hard links to work on win32)

If we're using subprocess.popen() then is escaping needed on linux?

Which lead to a question for me on do we have an official way in SCons to get the stdout and stderr text from an Action that was called. I don’t recall having this before and as such I used the SPAWN to help me add logging and colorized text in Parts.

As far as I know there's currently no supported way to capture stderr or stdout from an action. (you can redirect in your command to a file).

Do we lose redirects with swapping in the Popen() ?

-Bill

Jason

From: Scons-dev <scons-dev-bounces at scons.org<mailto:scons-dev-bounces at scons.org>> On Behalf Of Bill Deegan
Sent: Friday, October 5, 2018 12:44 PM
To: SCons developer list <scons-dev at scons.org<mailto:scons-dev at scons.org>>
Subject: Re: [Scons-dev] Subprocess.popen() on win32?

Jason,

Any limitations with doing this over the current code?
Could you craft a pull request for this?

-Bill

On Fri, Oct 5, 2018 at 10:37 AM Jason Kenny <dragon512 at live.com<mailto:dragon512 at live.com>> wrote:
Been using popen with parts via overriding SPAWN var since the start.
+1 on having this be the default
Jason
Get Outlook for Android<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Faka.ms%2Fghei36&data=02%7C01%7C%7C2d46a6c59d684692660f08d62c8e6426%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636745387402640578&sdata=xQzJRpZrmVzLQdwF4iHn%2BJ%2B7PcEZ1QE1d6q5Gu41mlY%3D&reserved=0>

________________________________
From: Scons-dev <scons-dev-bounces at scons.org<mailto:scons-dev-bounces at scons.org>> on behalf of Bill Deegan <bill at baddogconsulting.com<mailto:bill at baddogconsulting.com>>
Sent: Friday, October 5, 2018 10:32:50 AM
To: SCons developer list
Subject: Re: [Scons-dev] Subprocess.popen() on win32?

Come now.
Win32.py's spawn implementation was updated in 2012.. ;)

If we universally switch to subprocess.Popen(), then we can safely chdir for non python function actions.
But passing in the cwd argument.

That seems likely to be useful..

-Bill

On Fri, Oct 5, 2018 at 10:28 AM Russel Winder <russel at winder.org.uk<mailto:russel at winder.org.uk>> wrote:
I shall be flippant in the hope of being humorous, yet with an element of
possible truth:

Perhaps no-one has bothered to update that bit of code in the last decade and
a half?

Did you mean subprocess.Popen? Worth noting that the API for subprocess
changes from call* in earlier versions to run in later. Somewhat annoying for
compatibility across many Python versions.

On Fri, 2018-10-05 at 10:17 -0700, Bill Deegan wrote:
> Greetings,
>
> Any reason we're not using subprocess.popen() on win32?
> We're still using os.spawnve() which is known to have some threading issues.
> https://bugs.python.org/issue6476<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbugs.python.org%2Fissue6476&data=02%7C01%7C%7C2d46a6c59d684692660f08d62c8e6426%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636745387402640578&sdata=BdkwYzCHfKkLtCA9MKlerT3SCCvMnEIuzyNh3z%2BGkz8%3D&reserved=0>
>
> -Bill
> _______________________________________________
> Scons-dev mailing list
> Scons-dev at scons.org<mailto:Scons-dev at scons.org>
> https://pairlist2.pair.net/mailman/listinfo/scons-dev<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpairlist2.pair.net%2Fmailman%2Flistinfo%2Fscons-dev&data=02%7C01%7C%7C2d46a6c59d684692660f08d62c8e6426%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636745387402640578&sdata=2a9RsHGYmJEVs%2BmJmXdibUUM35S91bAniHyeD%2BNk%2BnU%3D&reserved=0>
--
Russel.
===========================================
Dr Russel Winder      t: +44 20 7585 2200
41 Buckmaster Road    m: +44 7770 465 077
London SW11 1EN, UK   w: www.russel.org.uk<https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.russel.org.uk&data=02%7C01%7C%7C2d46a6c59d684692660f08d62c8e6426%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636745387402640578&sdata=aCR8VpuVx%2FrbPIYb59dC3IsW8AtfGumujeMNIX8IvsM%3D&reserved=0>

_______________________________________________
Scons-dev mailing list
Scons-dev at scons.org<mailto:Scons-dev at scons.org>
https://pairlist2.pair.net/mailman/listinfo/scons-dev<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpairlist2.pair.net%2Fmailman%2Flistinfo%2Fscons-dev&data=02%7C01%7C%7C2d46a6c59d684692660f08d62c8e6426%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636745387402796835&sdata=HIP65PCLvMytQ94%2FQxZ9BEvp9LWnL8YxgKVXlIZuPJU%3D&reserved=0>
_______________________________________________
Scons-dev mailing list
Scons-dev at scons.org<mailto:Scons-dev at scons.org>
https://pairlist2.pair.net/mailman/listinfo/scons-dev<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpairlist2.pair.net%2Fmailman%2Flistinfo%2Fscons-dev&data=02%7C01%7C%7C2d46a6c59d684692660f08d62c8e6426%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636745387402796835&sdata=HIP65PCLvMytQ94%2FQxZ9BEvp9LWnL8YxgKVXlIZuPJU%3D&reserved=0>
_______________________________________________
Scons-dev mailing list
Scons-dev at scons.org<mailto:Scons-dev at scons.org>
https://pairlist2.pair.net/mailman/listinfo/scons-dev<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpairlist2.pair.net%2Fmailman%2Flistinfo%2Fscons-dev&data=02%7C01%7C%7C2d46a6c59d684692660f08d62c8e6426%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636745387402796835&sdata=HIP65PCLvMytQ94%2FQxZ9BEvp9LWnL8YxgKVXlIZuPJU%3D&reserved=0>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist2.pair.net/pipermail/scons-dev/attachments/20181007/6fe7e391/attachment-0001.html>