[Scons-dev] Variable Substitution

[Jason Kenny]

I don’t think I abuses the values.. but pushed the subst() system beyond what it was orginally design for. The code you are asking for should be:

https://bitbucket.org/sconsparts/parts/src/6a57ea6ae48acf491cda5ab8c1d45fd53d4543c2/parts/mappers.py?at=master&fileviewer=file-view-default#mappers.py-258
lines 258 – 310

replacement for

_concat
_concat_ixes

This is mapped BTW with a fix to scanners

https://bitbucket.org/sconsparts/parts/src/6a57ea6ae48acf491cda5ab8c1d45fd53d4543c2/parts/overrides/scanner.py?at=master&fileviewer=file-view-default

This in general prevent most of the extra subst calls and prevents an issue with Scons getting confused with state that is stroed in the DB that I have seen cause a rebuild after a fresh build of everything. This only happens once. The main goal of this file is reduction of subst calls and to avoid the issue with the rebuild

I should point out all the main overirde ( minus maybe one or two of them) are all in the
https://bitbucket.org/sconsparts/parts/src/6a57ea6ae48a/parts/overrides/?at=master

location. Anything here are items tweaked to improve the Scons system in small, or larger ways. Most all these are easy to move to Scons, and few might call some Parts infra ( such as output code for better error handling or print relacement api.output functions for Part to handle coloring and verbose/ trace messages with logging)

Jason




From: Bill Deegan [mailto:bill at baddogconsulting.com]
Sent: Wednesday, July 26, 2017 6:38 PM
To: Jason Kenny <dragon512 at live.com>; SCons developer list <scons-dev at scons.org>
Subject: Re: Variable Substitution

Jason,

Any chance you can point to your improved _concat_ixes(), and any other routines you’ve improved and/or a pull request per improved function?

It does look like you really abused the values.
Unless you’re playing games with them in scanners or emitters..

-Bill

On Mon, Jul 24, 2017 at 7:52 PM Jason Kenny <dragon512 at live.com<mailto:dragon512 at live.com>> wrote:
Hi Bill

First sorry I double posted this.. the first case I did not finish, fumbled a key and posted before I was ready… This version should be “better”.

Given Parts makes components I needed a way to share data. I tried the Export Import logic, but had issues with it. In the end I used Import to pass in the env and other functions I add in Parts.

This is why all .parts files start with Import(‘*’). I could get around this by taking more control, but this seemed to be counter productive to my hope of getting Parts logic into SCons.

So to pass data around and to request what data is needed we have a DependsOn() function ( and a Exportxxx()/SDKxxx functions that push values to be shareable” that allows the user define they depend on a component with certain requires, like version or other features ( given more than on Part is defined in the SConstruct as it common if you cross build in a single pass). I this call you can define some values that you require from that component via the REQ object. Certain values are passed automatically such as CPPPATH, and LIBS.. ie values you need to compile against the component, while other values need to be explicitly requested, such as CPPFLAGS or custom values that may have been exporting from the Component. When you DependOn([Component(A)) Parts will add some strings to map implicated values. These may not get expanded at all and at worse waste some space in the environment. Explicit value defined by the REQ meta object also get mapped. For this e-mail, I will keep it simple and leave out some features.

The main point is that the string we add is something like “{PARTIDEXPORTS(‘sub1 at version:1.*’,’build’,’LIBS’,’1’)}”

This call a parts.mapper object ( we have a few defined in Parts). This object will look up the needed object and get the requested data in the dependent component and add it to the environment variable in question. This subst happens when SCons subst the code. We do some tricks to avoid extra look ups and to replace values that are known. This way Part A can depend on Part B and C and we can add to the LIBS and LIBSPATH variable the needed information without the developer bothering to do this. What is also nice about this is that if B was making a Boo.lib files and it changes to a BOB1.2.lib file, the only change needed is the in B.part files. The change will auto propagate correctly up the depends chain and everything will just rebuild. As I stated this happens via calling a python object in the subst() call and having the code correctly return a value and or a value and update in place a value in the list ( as is common with CPPFLAGS, LIBS, etc..)  In the end, you do a build ( and you can look at samples in Parts to see this happening) when the part file is done being read we might see something like:

Given a setup such as

a->[B,C]
B->[]
C->[D]

Env[‘LIBS’] -> [‘a.lib’,’ {PARTIDEXPORTS(‘B at version:2.*’,’build’,’LIBS’,’1’)}’, {PARTIDEXPORTS(‘C at version:1.*’,’build’,’LIBS’,’1’)}”]

But when the subst for like “LINKCOM” happens it will get for the $LIBS a list like -> [‘a.lib’,’b.lib’,’c.lib’, ’d.lib’]. technically this happens with the Scanner first for items LIBS, LIBPATH, CPPPATH, so when the command subst happens we already replaced ourselves to prevent redundant logic from happening.

The mapper code in all the glory can be seen here: https://bitbucket.org/sconsparts/parts/src/6a57ea6ae48acf491cda5ab8c1d45fd53d4543c2/parts/mappers.py?at=master&fileviewer=file-view-default

If you look at the history this was simpler code at first, but then Gary complained I made it to C\C++ specific so it could never get into SCons, I tweaked to make it more general, then I needed to make it better to allow custom values.  Other tweaks have been made as well over time to address bugs or feature issues. The issue is that this code goes off a lot and takes time to process in Large builds. Work has been done to replace the SCons version of _concat_ixes(), and some other objects to make them not faster. The last major update we had made was to try to fix an issue in which we had a Vtune with some messed up depends circular depends in which the mapper code would flip an include path around with mid-level dependent components, cause a rebuild because SCons saw a change in the environment. This code has been a sore spot for me to a large degree as it never quite worked right in certain complex cases was done to delay subst values because we would not know at load time which variant of Part C you needed to get value from until we loaded all the components. The current form is a bit complex and ugly honestly. There is some odd work around for some odd behaviors with CmdStringHolder class.

Given what I have learned... The plan is to replace this given I get time to do this via implementing:


  1.  A new Parts format using @decorators to delay process functions ( ie make the Parts file loading not an exec off a lot of python code, but a register callback) This will allow me to load all the “parts” and have needed information such as name, version, and other platform information before I try to process what the component will emit as build actions.
  2.  For the existing format implement a “continuous loader”.. or load a Parts file until I get to a DependsOn statement. At the point, I pause and start loading another Part until all the primary “top” level Parts are loaded. At that point, I know enough information to know what is at the bottom and what is at the top… This can be done with threads in earlier versions of python or with python3 (3.5 in my head) asyncio and yield “task” logic.

In both of these cases, this allows me to delay “processing” the expensive stuff, such as env.Program([list of 1000 files]) until I really need to because I know it is the build “target” list. I also can on first pass ( ie no DB) can load the Parts in order and insert the needed values directly. Ie add “B.lib” or “b/includes” to the needed item. I don’t need to subst and code. At most I might need to move some values around, such as libs to make sure the linker is happy and we avoid double ( or more) inserts of the same values. This would allow Parts to start SCons up much faster as we can:


  1.  Avoid loading all the components if the target is to not build “all” or “.”

  1.  Avoid a lot of subst() calls during the build allowing SCons, while allowing me a nice way to share data in an organized way. This will also allow me to share objects between components vs only strings. This will be good as it will allow a “part” to define a builder and export it to be shared by other components. Which is a more common request than I thought. I have ugly work around at the moment to do this today.. but I rather not speak about them 😊
  2.  Independent of mappers, this allows for other speed ups such as possible avoiding of code execution such as env.Builder([lots of objects]) what would not be called or better yet avoid calls to SCons Glob() or the Parts Pattern() which scan the disk for files. This was found in profiling to be one of the biggest reasons why it took the time to load a part. We had cases such as boost in which a few scans happened ( one for headers to export and few different ones for sources to build different boost components) which would result in the file taking 30 seconds to load not  average .02 – .5 seconds on a laptop spinning disk the other parts generally took that did not scan large number of files. ( this is one reason for the use the cache better argument I made before, as we should only need to scan a directory at most once, and then only again for a change. I had a prototype this in Parts once and it showed a big improvement in load time for the read phase.. never was finished it ☹, but for me the point is that once we know what it is we should save it and not reproduce calculating it until something says we need too. File and directory nodes, and  subst string values I had used to pass libs, paths and other values between components are great examples of this )

I hope this helps… let me know if you have other questions.

You can also install Parts with SCons on a box and run a some of my samples.. add a --trace=mappers should give you a big dump of data. ( ie add --verbose and --trace will dump everything… adding the [category] printed with a verbose or trace message to the –trace or –verbose separated with commas will allow you to filter to a smaller set of data. It will help show what is going on and how Parts shares data

Jason

In case you want a simple example of what this might look like without playing with code

# hello part depends on print_msg parts
PartName('hello')
env.DependsOn([Component('print_msg','1.*')])
ret=env.Program("hello","hello.c")
env.InstallTarget(ret)

------------
# the print_msg.part
PartVersion('1.0.0')
PartName('print_msg')

#files
cpp_files=['print_msg.c']
…
outputs=env.SharedLibrary('print_msg',cpp_files)

#This will auto export the print_msg.lib file as well other stuff related to install sandbox and possible package creation
env.InstallTarget(outputs)
# this exports the print_msg.h include path
env.SdkInclude(['print_msg.h'])

---
#SConstruct

from parts import *
Part('hello/hello.parts')
Part('print_msg/print_msg.parts')

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist2.pair.net/pipermail/scons-dev/attachments/20170727/0d4bef9f/attachment-0001.html>