[Scons-dev] Variable Substitution

[Bill Deegan]

Jason,

Any chance you can point to your improved _concat_ixes(), and any other
routines you’ve improved and/or a pull request per improved function?

It does look like you really abused the values.
Unless you’re playing games with them in scanners or emitters..

-Bill

On Mon, Jul 24, 2017 at 7:52 PM Jason Kenny <dragon512 at live.com> wrote:

> Hi Bill
>
>
>
> First sorry I double posted this.. the first case I did not finish,
> fumbled a key and posted before I was ready… This version should be
> “better”.
>
>
>
> Given Parts makes components I needed a way to share data. I tried the
> Export Import logic, but had issues with it. In the end I used Import to
> pass in the env and other functions I add in Parts.
>
>
>
> This is why all .parts files start with Import(‘*’). I could get around
> this by taking more control, but this seemed to be counter productive to my
> hope of getting Parts logic into SCons.
>
>
>
> So to pass data around and to request what data is needed we have a
> DependsOn() function ( and a Exportxxx()/SDKxxx functions that push values
> to be shareable” that allows the user define they depend on a component
> with certain requires, like version or other features ( given more than on
> Part is defined in the SConstruct as it common if you cross build in a
> single pass). I this call you can define some values that you require from
> that component via the REQ object. Certain values are passed automatically
> such as CPPPATH, and LIBS.. ie values you need to compile against the
> component, while other values need to be explicitly requested, such as
> CPPFLAGS or custom values that may have been exporting from the Component.
> When you DependOn([Component(A)) Parts will add some strings to map
> implicated values. These may not get expanded at all and at worse waste
> some space in the environment. Explicit value defined by the REQ meta
> object also get mapped. For this e-mail, I will keep it simple and leave
> out some features.
>
>
>
> The main point is that the string we add is something like
> “{PARTIDEXPORTS(‘sub1 at version:1.*’,’build’,’LIBS’,’1’)}”
>
>
>
> This call a parts.mapper object ( we have a few defined in Parts). This
> object will look up the needed object and get the requested data in the
> dependent component and add it to the environment variable in question.
> This subst happens when SCons subst the code. We do some tricks to avoid
> extra look ups and to replace values that are known. This way Part A can
> depend on Part B and C and we can add to the LIBS and LIBSPATH variable the
> needed information without the developer bothering to do this. What is also
> nice about this is that if B was making a Boo.lib files and it changes to a
> BOB1.2.lib file, the only change needed is the in B.part files. The change
> will auto propagate correctly up the depends chain and everything will just
> rebuild. As I stated this happens via calling a python object in the
> subst() call and having the code correctly return a value and or a value
> and update in place a value in the list ( as is common with CPPFLAGS, LIBS,
> etc..)  In the end, you do a build ( and you can look at samples in Parts
> to see this happening) when the part file is done being read we might see
> something like:
>
>
>
> Given a setup such as
>
>
>
> a->[B,C]
>
> B->[]
>
> C->[D]
>
>
>
> Env[‘LIBS’] -> [‘a.lib’,’ {PARTIDEXPORTS(‘B at version:2.*’,’build’,’LIBS’,’1’)}’,
> {PARTIDEXPORTS(‘C at version:1.*’,’build’,’LIBS’,’1’)}”]
>
>
>
> But when the subst for like “LINKCOM” happens it will get for the $LIBS a
> list like -> [‘a.lib’,’b.lib’,’c.lib’, ’d.lib’]. technically this happens
> with the Scanner first for items LIBS, LIBPATH, CPPPATH, so when the
> command subst happens we already replaced ourselves to prevent redundant
> logic from happening.
>
>
>
> The mapper code in all the glory can be seen here:
> https://bitbucket.org/sconsparts/parts/src/6a57ea6ae48acf491cda5ab8c1d45fd53d4543c2/parts/mappers.py?at=master&fileviewer=file-view-default
>
>
>
> If you look at the history this was simpler code at first, but then Gary
> complained I made it to C\C++ specific so it could never get into SCons, I
> tweaked to make it more general, then I needed to make it better to allow
> custom values.  Other tweaks have been made as well over time to address
> bugs or feature issues. The issue is that this code goes off a lot and
> takes time to process in Large builds. Work has been done to replace the
> SCons version of _concat_ixes(), and some other objects to make them not
> faster. The last major update we had made was to try to fix an issue in
> which we had a Vtune with some messed up depends circular depends in which
> the mapper code would flip an include path around with mid-level dependent
> components, cause a rebuild because SCons saw a change in the environment.
> This code has been a sore spot for me to a large degree as it never quite
> worked right in certain complex cases was done to delay subst values
> because we would not know at load time which variant of Part C you needed
> to get value from until we loaded all the components. The current form is a
> bit complex and ugly honestly. There is some odd work around for some odd
> behaviors with CmdStringHolder class.
>
>
>
> Given what I have learned... The plan is to replace this given I get time
> to do this via implementing:
>
>
>
>    1. A new Parts format using @decorators to delay process functions (
>    ie make the Parts file loading not an exec off a lot of python code, but a
>    register callback) This will allow me to load all the “parts” and have
>    needed information such as name, version, and other platform information
>    before I try to process what the component will emit as build actions.
>    2. For the existing format implement a “continuous loader”.. or load a
>    Parts file until I get to a DependsOn statement. At the point, I pause and
>    start loading another Part until all the primary “top” level Parts are
>    loaded. At that point, I know enough information to know what is at the
>    bottom and what is at the top… This can be done with threads in earlier
>    versions of python or with python3 (3.5 in my head) asyncio and yield
>    “task” logic.
>
>
>
> In both of these cases, this allows me to delay “processing” the expensive
> stuff, such as env.Program([list of 1000 files]) until I really need to
> because I know it is the build “target” list. I also can on first pass ( ie
> no DB) can load the Parts in order and insert the needed values directly.
> Ie add “B.lib” or “b/includes” to the needed item. I don’t need to subst
> and code. At most I might need to move some values around, such as libs to
> make sure the linker is happy and we avoid double ( or more) inserts of the
> same values. This would allow Parts to start SCons up much faster as we can:
>
>
>
>
>
>    1. Avoid loading all the components if the target is to not build
>    “all” or “.”
>
>
>    1. Avoid a lot of subst() calls during the build allowing SCons, while
>    allowing me a nice way to share data in an organized way. This will also
>    allow me to share objects between components vs only strings. This will be
>    good as it will allow a “part” to define a builder and export it to be
>    shared by other components. Which is a more common request than I thought.
>    I have ugly work around at the moment to do this today.. but I rather not
>    speak about them 😊
>    2. Independent of mappers, this allows for other speed ups such as
>    possible avoiding of code execution such as env.Builder([lots of objects])
>    what would not be called or better yet avoid calls to SCons Glob() or the
>    Parts Pattern() which scan the disk for files. This was found in profiling
>    to be one of the biggest reasons why it took the time to load a part. We
>    had cases such as boost in which a few scans happened ( one for headers to
>    export and few different ones for sources to build different boost
>    components) which would result in the file taking 30 seconds to load not
>     average .02 – .5 seconds on a laptop spinning disk the other parts
>    generally took that did not scan large number of files. ( this is one
>    reason for the use the cache better argument I made before, as we should
>    only need to scan a directory at most once, and then only again for a
>    change. I had a prototype this in Parts once and it showed a big
>    improvement in load time for the read phase.. never was finished it ☹,
>    but for me the point is that once we know what it is we should save it and
>    not reproduce calculating it until something says we need too. File and
>    directory nodes, and  subst string values I had used to pass libs, paths
>    and other values between components are great examples of this )
>
>
>
> I hope this helps… let me know if you have other questions.
>
>
>
> You can also install Parts with SCons on a box and run a some of my
> samples.. add a --trace=mappers should give you a big dump of data. ( ie
> add --verbose and --trace will dump everything… adding the [category]
> printed with a verbose or trace message to the –trace or –verbose separated
> with commas will allow you to filter to a smaller set of data. It will help
> show what is going on and how Parts shares data
>
>
>
> Jason
>
>
>
> In case you want a simple example of what this might look like without
> playing with code
>
>
>
> # hello part depends on print_msg parts
>
> PartName('hello')
>
> env.DependsOn([Component('print_msg','1.*')])
>
> ret=env.Program("hello","hello.c")
>
> env.InstallTarget(ret)
>
>
>
> ------------
>
> # the print_msg.part
>
> PartVersion('1.0.0')
>
> PartName('print_msg')
>
>
>
> #files
>
> cpp_files=['print_msg.c']
>
> …
>
> outputs=env.SharedLibrary('print_msg',cpp_files)
>
>
>
> #This will auto export the print_msg.lib file as well other stuff related
> to install sandbox and possible package creation
>
> env.InstallTarget(outputs)
>
> # this exports the print_msg.h include path
>
> env.SdkInclude(['print_msg.h'])
>
>
>
> ---
>
> #SConstruct
>
>
>
> from parts import *
>
> Part('hello/hello.parts')
>
> Part('print_msg/print_msg.parts')
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist2.pair.net/pipermail/scons-dev/attachments/20170726/98d83b16/attachment-0001.html>