[Scons-dev] revisiting CI setup

[Mats Wichmann]

time passes... so time to ask some questions again.


The travis build does two code-coverage runs (2.7 and 3.6), then seven 
regular runs (2.7, 3.5, 3.6, 3.7, 3.8-dev, pypy2 and pypy3.

The appveyor build, despite a recent caching hack that sped it up a lot, 
is still much slower, and thus is asked to do less work, but has a more 
complex scenario: it needs to test itself on three different appveyor 
images, each provisioned with a different Visual Studio version, so what 
it runs is: 2.7 on VS2017 image, 3.5 on VS2015 image, 3.6 on VS2017 
image (this is also a coverage run), 3.7 on VS2019 image.

We're having some test problems due to evolution of Windows images - in 
particular (there are github issues on this), clang tests do not run 
properly, because before it always found the mingw clang, but in the 
VS2019 image that is no longer installed by default, and the setup 
doesn't work with a "native" clang - basically this exposed an untested 
issue, rather than breaking something that was working.

3.8 is fully released now.  only the Linux CI (travis) is running the 
tests under 3.8, and it's using "3.8-dev" as opposed to an officially 
released version of 3.8. Although it's early days yet, there ought to be 
a 3.9-dev (last I checked, appveyor hadn't enabled that choice, but that 
was a while back)

what sets of these are actually important for checking that a PR does 
not introduce testing regressions?

I'm unconvinced the pypy tests add any value in this context. Maybe we 
could move them to the buildbot setup instead? In particular, pypy3 has 
never gained the performance of pypy2, and doesn't seem to be in active 
development, and the performance is a killer for our use of it.

We should be able to stop testing 2.7 once develoment flips to "4.0", 
since it won't carry the compatibility promise any longer.

Is there any other way to rearrange/improve this, including the 
mentioned buildbot - things which aren't deemed critical to run on every 
PR but which still would be useful to see the progress of could move to 
buildbot....