[Scons-dev] MSVC 2017+ Toolset Support
Joseph Brill
joseph.c.brill at gmail.com
Mon Aug 10 20:03:16 EDT 2020
> "Can I end up with 14.1 installed in both a MSVC2017 and MSVC2019 install
(and of course also all the flavors of MSVC, build tools, express,
community, enterprise), for a possible total of 8 separate installs? (or
some smaller non singular number?)"
Yes. I believe the number is 9 separate product installations for 2017
(build tools, express, community, professional, and enterprise) and 2109
(build tools, community, professional, and enterprise). I don't have
access to professional or enterprise versions and a I believe the installer
for 2017 build tools is rarer than hen's teeth (although it may be
available via Chocolatey). As far as I know, there is not an express
version of 2019. Locally, I have been using four installations for testing
of the 14.1 (14.16) toolsets: 2017 Community, 2017 Express, 2019,
Community, and 2019 Build Tools.
It is possible to specify the 2015 toolset (14.0) via MSVC 2017 and MSVC
2019 as well. The argument to the vcvars batch file is "14.0". The batch
file then effectively finds the 14.0 build tools. The working version uses
the registry keys from the msvc batch files to determine if the 14.0 tools
are installed. I believe the registry keys used by the msvc batch
files.are slightly different that the registry keys that SCons uses for
14.0.
I am time limited today and would like to spend some more time thinking
about the rest of your email before responding.
Implementation note: one advantage of the more complicated syntax is that
it is validated with a single fairly straightforward regular expression.
The components of interest are captured groups. The string match is done
once upon entry. MSVC_VERSION is immediately set to the compatible "14.X"
format for the existing code. Except for initial entry, MSVC_VERSION is
the currently used format. One reason this was necessary was that the
existing code was converting MSVC_VERSION to a floating point number. Even
adding the second digit to the MSVC_VERSION (e.g., 14.16) would/could have
"broken" the existing code. This was the key to not having to modify a
sizable chunk of the existing code. The remaining fields are stored in an
internal data structure due to the gap between determining if the tools are
installed and actually determining the host/target batch file arguments.
> (And yes. I know.. "Foolish consistency is the hobgoblin of little
minds", but I think in this case it's not foolish)
Emerson. Nice. As you are probably painfully aware by now: "I would have
written a shorter letter, but I did not have the time".
On Mon, Aug 10, 2020 at 7:23 PM Bill Deegan <bill at baddogconsulting.com>
wrote:
> Can I end up with 14.1 installed in both a MSVC2017 and MSVC2019 install
> (and of course also all the flavors of MSVC, build tools, express,
> community, enterprise), for a possible total of 8 separate installs? (or
> some smaller non singular number?)
>
> I'm not a fan of the more complicated syntax for MSVC_VERSION and would
> rather split out the selection to either separate variables for product,
> toolset, and product version and/or a callable which handles selection.
> MSVC_VERSION -> sounds like after MSVC2017 defaults to latest installed
> toolset unless you set toolset version
> MSVC_PRODUCT_VERSION -> Enterprise, Professional, Community, Express, or
> not set
> MSVC_TOOLSET_VERSION -> This may only apply for MSVC2017 or above (or
> could be used if MSVC_VERSION is not set for MSVC < 2017, though that would
> likely introduce confusion
>
> MSVC_SELECTOR -> Callable (too allow more flexibility than static versions
> above)
>
> Otherwise you end up encoding the information in a string and then you
> have to decode it.
> Seems more complicated than necessary, and we don't have any other
> syntaxes embedded in variables thus far.
>
> I realize you have lots of time invested in your MSVC_VERSION
> implementation you've explained, but the goal here is to be consistent with
> expected behavior in the face of changed MSVC.
>
> (And yes. I know.. "Foolish consistency is the hobgoblin of little minds",
> but I think in this case it's not foolish)
>
> Welcome feedback on above.
>
>
>
>
> On Sun, Aug 9, 2020 at 2:50 PM Joseph Brill <joseph.c.brill at gmail.com>
> wrote:
>
>> This email is an attempt to focus the discussion concerning support for
>> MSVC toolsets moving forward in a rational, dispassionate discourse.
>>
>> I am advocating a minor shift in orientation or "world view" for MSVC
>> 2017 and later: the semantic transition from thinking of the msvc version
>> as a product to thinking of the msvc version as a toolset. This actually
>> makes the implementation of toolset support easier.
>>
>> There is a working version of the support for toolsets at varying levels
>> of precision that can be enabled and disabled. The intent is that a
>> command-line switch could be added to enable the toolset specific
>> capabilities. By default, the toolset specific capabilities would be
>> disabled in which case the code behaves exactly like the current
>> master. This would allow field testing and evaluation of the functionality
>> in other environments and to easily test the differences between the
>> current master and the toolset functionality. The functional aspects are
>> feature complete. The working version is a product of more than one
>> man-month of full-time development activity and the result of the third
>> iteration of implementation strategy.
>>
>> Existing PR #3717 was closed and the branch renamed due in part to make
>> it easier to identify and maintain for myself as I fear that I might be
>> time limited in the near term and due in part to the existing discussion
>> unraveling fairly quickly with dim prospects of inclusion moving forward. Despite
>> being closed, I recommend viewing the "documentation" (the PR text and the
>> first comment) and the vc.py code difference in PR #3717. At any time the
>> PR request can be reopened. I would need a git guru to walk me through
>> associating the new branch name with the "old" PR.
>>
>> The implementation knocks an item or two off the internal vc.py TODO list
>> such as a single vswhere query and processing the json results for all
>> known installations. Ranked toolset lists are constructed by host/target.
>> The ranking takes into account the toolset version and product type. A
>> single toolset instance may be a member of multiple toolset lists.
>> Expensive lookups are "runtime memoized/cached" so that they are only
>> computed once. While there is a fair amount of initialization, the
>> processing burden is based on the number of products installed and the
>> number of toolsets for each product. Typical environments likely have a
>> single product installation with a spartan number of different toolsets
>> installed. There is nothing particularly "clever" done in the
>> implementation and should not be a maintenance burden. The bulk of the
>> toolset support implementation follows the code in the existing master
>> (i.e., the bottom of the file). The modifications to support toolsets
>> weighs in at approximately 1400 lines of code.
>>
>> Prior to MSVS 2017, there was a direct 1:1 mapping between the msvc
>> *product* and the msvc *toolset*. The toolset might be upgraded in an
>> update or service pack but there was no way to select a specific toolset as
>> side-by-side installations were not supported. That changed with the
>> release of MSVC 2017 which allows multiple side-by-side toolset
>> installations by target. MSVC 2017 and later, can be viewed as *toolset* oriented
>> (i.e., a MSVC 2019 installation can be viewed as a container for 14.1X,
>> 14.2X, and transparently the 14.0 toolsets). Moving forward, I believe
>> that it will be advantageous to change from a *product* orientation to a
>> *toolset* orientation. This change is only for the 2017 and later
>> installations and currently really only affects 2017.
>>
>> With MSVC 2017 and MSVC 2019, an SCons version request of "14.1" or
>> "14.2", respectively, *is a toolset request* that cannot be specified at
>> a finer granularity. The current msvc behavior uses the *default toolset* which
>> is generally the newest installed *toolset* version when the
>> "--vcvars_ver" argument is not specified for the vcvars batch file.
>>
>> For example, a version request of "14.1" is likely to use the
>> "14.16.27023" toolset when MSVC 2017 is installed. Effectively, the *product
>> request* is a *toolset request* identical to "find the newest toolset
>> version installed that starts with '14.1'". However, "14.2" (MSVC 2019)
>> will return a *toolset* based on the latest update which likely varies
>> across users and computers. Currently, a request for "14.1" will fail if
>> MSVC 2019 is installed with the 14.1 toolset. With a toolset orientation,
>> a 14.1 toolset request would be satisfied first by the native product
>> (i.e., MSVC 2017) for the toolset and then by the latest product that
>> contains the user specified toolset for a given host/target combination.
>> This two-stage query, if necessary, is implemented in the working toolset
>> oriented version.
>>
>> Currently, the actual toolset versions used for 2017 and 2019 are based
>> on end-user's installation options and frequency of updates. While two
>> different users may be using the same *product* (e.g., 2019) they may
>> not be using the same *toolset*. Similarly, the installed toolset
>> version is not necessarily the same for all host/target combinations within
>> a given product installation. The current implementation uses a default
>> *toolset* within a product across different users and environments,
>> which may not be the same for all instances, but currently cannot use a
>> *toolset* across product installations.
>>
>> There may be many reasons (e.g., enterprise and/or customer requirements)
>> that build scripts are tied to a specific toolset. If a build depends on
>> the 14.1 *toolset* it hardly seems as important in which *product* container
>> it is installed from an end-user's perspective given the two-stage
>> selection rule described earlier. Given the same toolset, the produced
>> binaries should be binary compatible independent of the product
>> installation. The same issue will arise when the next MSVC *product* is
>> released and end-users desire to use the 14.2X *toolset* combination in
>> the new product container.
>>
>> At present, the shift in toolset orientation would only affect user's
>> that desire the 14.1 toolset and do not have MSVC 2017 installed. In this
>> case, the only other product that can be used is MSVC 2019. This isolates
>> the effects of changes to a single version of MSVC.
>>
>> This could have a tangible side-effect of making build scripts more
>> portable: the build scripts can be tied to a toolset rather than a
>> toolset/product combination. Although if it is desired to specify the
>> toolset/product combination that is supported as well.
>>
>> The MSVC_VERSION format was expanded to allow:
>>
>> - specifying an individual toolset at varying degrees of precision:
>> - "14.1"
>> - "14.16"
>> - "14.16.27"
>> - "14.16.27023"
>> - specifying individual product types (attached to a specific toolset
>> or a product):
>> - Enterprise: "14.26Ent"
>> - Professional: "14.2Pro"
>> - Community: "14.26.27Com"
>> - Build Tools: "14.2BT"
>> - Express: "14.1Exp"
>> - a right assignment operator ("->") that maps a specific toolset to
>> a defined product version with an optional product type:
>> - "14.0->14.2" use the the 14.0 toolset via a 2019 installation
>> - "14.1->14.2BT" use the 14.1 toolset via a 2019 Build Tools
>> installation
>> - "14.1->14.2Com" use the 14.1 toolset via a 2019 Community
>> installation
>>
>> The translation of the extended version specification to a supplementary
>> internal data structure is done in exactly one source code location:
>> msvc_setup_env. All existing code continues to operate on the "NN.M"
>> format for MSVC_VERSION as before. Existing msvc/msvs test suites pass all
>> tests. However, issues with the extended format being used prior to entry
>> in vc.py script have yet to be identified, if any. The working version is
>> completely self-contained in the vc.py script.
>>
>> Without a product qualifier (e.g., "BT"), there is a *subtle but
>> significant* change semantically: the product selected will be the
>> newest/latest toolset that matches the user specification within installed
>> toolsets for a given (host,target) combination. Effectively, without a
>> product qualifier, any product type could be returned based on the
>> installed toolsets. This only affects multiple installations of the same
>> product version (e.g., 14.1 and 14.Exp and/or 14.2Com and 14.2BT). For end
>> users that only have one product installed, there is no reason to specify a
>> product type.
>>
>> This means that a user does not need to specify "14.1Exp" explicitly if
>> there is a single 14.1 product installation. On the flip side, it also
>> means that "14.1Exp" may be returned for a "14.1" request. In local
>> testing, "14.1Exp" was returned for an "arm" target. In the local
>> installations of 14.1 Community and 14.1 Express, the "newest" toolset for
>> an arm target was in the 14.1 Express installation as it was installed the
>> most recently. With a product qualifier, a specific product version and
>> type is selected.
>>
>> To be clear, I do not need any fixes, enhancements, and am not reporting
>> any bugs. The toolset version development was done in the spirit of
>> contributing to software that I have used since 2.X. I had a development
>> window to contribute to the windows/msvc support and in process address two
>> of the outstanding msvc items on the issues list. I was motivated by
>> both the post in the scons users mailing list shown below and issue #3664.
>> As I am familiar with the msvc detection code, my initial thoughts were
>> "how hard could it be?"
>>
>> Regards,
>>
>> Joe
>>
>> https://pairlist4.pair.net/pipermail/scons-users/2020-June/008216.html
>>
>> >* The problem is scons don't find any 14.1(2017) installation, he only find
>> *>* 14.2(2019).
>> *> >* Is there something I could do to tell scons that 141 is installed somewhere
>> *>* in vs2019 installation?
>> *
>> at the moment, no... it's a work in progress.
>>
>> See https://github.com/SCons/scons/issues/3664
>>
>> There wasn't really a "Windows native" person involved in the most
>> recent times, so we didn't end up understanding about the model of
>> multiple versions under one VS install - and in particular that people
>> would want to get 14.1 from VS2019 rather than keep VS2017 installed.
>> For the most recent VS installs, vswhere is used to detect them, but we
>> find only the "default version" under each product vswhere reports, so
>> 14.2.latest for 2019, 14.1 for 2017.
>>
>> _______________________________________________
>> Scons-dev mailing list
>> Scons-dev at scons.org
>> https://pairlist2.pair.net/mailman/listinfo/scons-dev
>>
> _______________________________________________
> Scons-dev mailing list
> Scons-dev at scons.org
> https://pairlist2.pair.net/mailman/listinfo/scons-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist2.pair.net/pipermail/scons-dev/attachments/20200810/1dc86875/attachment-0001.html>
More information about the Scons-dev
mailing list