[Scons-dev] CSIG and caching for subst in Parts

[Jason Kenny]

Hello,

As request by bdbaddog I am adding to the mailing list.

I am adding in my next drop of Parts some work to allow subst to work faster. It is restricted at the moment to Parts Mapper objects at this time until it is better understood how this could be expanded better in a more general way.  The core of the fix is to make a cache key that allows me to store a result given that I know at this point it is not going to change. This is a little more complex in that with Parts I have builders that deal with what I call dynamic scanners, or scanners that add build actions not known until during the build stage ( ie not after the read stage of all build files).

The current code for this caching is here https://bitbucket.org/sconsparts/parts/src/7a186e7a9c56ff117c8a32f603d2e048704fcbc6/src/parts/mappers.py#lines-140:171

It is based on the use of mapping what environment is used for generating the value. To make this work I had to add a get_csig() call to the environment. This allows me to know that $FOO_NAME is "joe" in this environment and "bill" in a different one.

The current code for this at this time is here:
https://bitbucket.org/sconsparts/parts/src/7a186e7a9c56ff117c8a32f603d2e048704fcbc6/src/parts/overrides/env_csig.py

I am still refactoring this at the moment as I had some older code that needs to be merged in with this ( ie ongoing cleanup I have to do). This code is here https://bitbucket.org/sconsparts/parts/src/7a186e7a9c56ff117c8a32f603d2e048704fcbc6/src/parts/common.py#lines-152:181

I plan to make a get_content() function in parts.core.util that will be used instead as a reusable function that can deal given objects as needed.

This sig work is also part of me cleaning up and resetting code that is turned off in Parts that allows for smart loading and caching of results to speed up the incremental build. That is a different subject from here which is about what I am putting together for making a better reproducible sig for the environment and how this can be used to improve subst() logic.

the early number for a no-op check that everything is up-to-date build I have with 315 parts ( think of this as Sconscripts) when from about 20 minutes to about 4. however as I tweak with an updated get_content function I expect this to probably go up a little. I also have better times for the full build but did not get any formal numbers to share as of yet

Jason
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist2.pair.net/pipermail/scons-dev/attachments/20200505/ae8c2805/attachment.html>