[Scons-dev] SCons-like build system

[Constantine]

> Parts check a number of things to see if a Part might be out of date, 
> and if it could be, we load it, else we don’t read the file, and 
> instead load cached information if and only if there is a component 
> that depends on it. 
I.e. Parts checks all sources, implicit dependencies, targets etc. by 
itself.
I mean - not using SCons standard logic. And it works faster than if 
SCons checked all these things.
Right?
Then what is the secret of Parts? How does it check all signatures much 
faster than SCons?

For me it sounds like if I split a big project to many totally 
independent small projects (SConstruct`s) and write a shell script:
|    scons -C subprj1||&& \||
||    scons -C subprj2 && \||
||    scons -C subprj3 && \||
||    ...||
||    scons -C subprj200||
|
Then this shell script would work faster then one big project, but it 
would be still significantly slower than Parts.
Right?

Thank you.
Best regards,
Constantine.

> In general Parts uses the notion of a component ( or a file that 
> defines something) to make a “component” Dag. It checks to see if 
> anything modified that state ( parts file changed, Sconstruct changed, 
> configuration/toolchains was modified, source  or output files for 
> component changed etc… which are mostly a Scons timestamp-MD5 check. 
> If any component is out of date or might be out of sync we mark it as 
> such and load any dependent component ( no need to check them as we 
> know there is “some” modification). This loads the items into SCons to 
> be processed. The resulting time to load and process is *much* smaller 
> on average.
>
> Currently Parts is looking at removing our “mapper” logic, which is an 
> abuse of the subst() system in SCons to share data between components, 
> and moving to what is called “continuous loader” which will load the 
> Components in dependency order (making data sharing simpler). This 
> should also make the current logic for checking state simpler and faster.
>
> From there the idea to look at is to store the actions defined in the 
> Component and replay them as part of the rebuild, as this can be done 
> once we know we have a sub tree that has nothing new added to it. 
> While those items build we continue to load more data ( as the build 
> would be waiting for build actions to complete). Given the common case 
> for a incremental build this should work out. As any relationships 
> defined explicitly or implicitly will not change. The most common 
> change will be in the source files, which will be seem as part of a 
> change that effect loading a component build file, or by the component 
> build file being changed. At any rate the dependent information gets 
> loaded as needed as processed to see what is safe or not safe to say. 
> This idea need more flushing out as there are details that could mess 
> up a correct build, or it might show to not speed anything up enough.
>
> Jason
>
> *From:*Scons-dev [mailto:scons-dev-bounces at scons.org] *On Behalf Of 
> *Constantine
> *Sent:* Sunday, January 25, 2015 3:24 AM
> *To:* SCons developer list; Tom Tanner
> *Subject:* Re: [Scons-dev] SCons-like build system
>
> Do I understand correctly?
> That the main idea is that using the Parts a big project is split into 
> many small parts.
> And then SCons builds these parts more-or-less independently.
> I.e. SCons processes many small DAGs instead of one big DAG.
>
> Thank you.
> Best regards,
> Constantine.
>
> On 01/23/15 18:03, Kenny, Jason L wrote:
>
>     I understand this problem. This is one main thing Parts tries to
>     address. It uses information about the different component to
>     figure out what not to load. This requires the build to be laid
>     out in some sort of component based like build, vs a recursive
>     make, which may not be a big deal, depending on how your project
>     is laid out. The result for example of a massive build I have here
>     is a “do nothing” build takes ~7-20 second vs the normal ~15-20
>     minutes it would normally take. Incremental build also are reduced
>     to minutes ( depending on what changed of course)
>
>     I honestly believe this can be better yet.. but we requires more
>     work.  The main issue with spawn issue that we pushed, was found
>     when a build for this large product moved from rhel4 to rhel5. The
>     time increased from 2 hours to 4+. It was the spawning issue.
>     Something changed that really made this worse. Once we had this
>     fixed rhel5 build went to 2 hours and the old rhel4 based build
>     went to ~1.5 hours.
>
>     Jason
>
>     *From:*Scons-dev [mailto:scons-dev-bounces at scons.org] *On Behalf
>     Of *Tom Tanner (BLOOMBERG/ LONDON)
>     *Sent:* Thursday, January 22, 2015 2:49 AM
>     *To:* scons-dev at scons.org <mailto:scons-dev at scons.org>
>     *Subject:* Re: [Scons-dev] SCons-like build system
>
>     Having been poking around in this, I see that 'posix_spawn' cleans
>     to be availble on both aix and solaris (at least they have man
>     pages). Which is a relief for me. However, my current grief is
>     that a 'do nothing' build is still taking 20 minutes to traverse
>     the tree. *part* of this is because I check the md5sum on each
>     file to make sure someone hasn't gone and edit/patched/hacked a
>     generated target (which happens more often than is desirable), but
>     that isn't the bulk of the time taken.
>
>     I suspect I could leverage the fact that we use git to get the
>     sha1 of certain repositories that are known not to be changed by
>     us and if we find a file therein, to use the combined sha1 of the
>     repos might improve that at the cost of potentially spurious
>     rebuilds if one of the repos changed but the others didn't as i
>     wouldn't be reading all the files.
>
>
>
> _______________________________________________
> Scons-dev mailing list
> Scons-dev at scons.org
> https://pairlist2.pair.net/mailman/listinfo/scons-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist2.pair.net/pipermail/scons-dev/attachments/20150127/97a99173/attachment-0001.html>