[Scons-dev] scons daemon

Schleimer, Ben bensch128 at yahoo.com
Thu Jan 14 18:24:33 EST 2016 

Hi Bill,   I decided to have the daemon copy the source tree because I wanted to make sure that once a background build starts, it can complete or fail without additional source file changes. 

About copying the directory: I guess that if the scons scan can correctly locate every source file and dependency, then only the source and SConscripts/SConstruct files need to be copied to the temp directory.
I did assume that the SConstruct file was in the top level of each source tree. It could be changed that the SConscript files are used to indicate the top level instead. I didn't consider that a usage case.

About memory usage: It looks like the memory usage roughly doubles because the parent process retains the data from the scan and then the child process retains the memory during it's build.

My memory stats:
Num source_files=107, num target_files=21, num SConscripts=728.772MB res for parent process24.5MB res for child process
I will try to add some tests.Where do you suggest putting them?

Can you give me a stub in the Wiki to add documentation?

Sincerely,Ben
 

    On Thursday, January 14, 2016 1:55 PM, Bill Deegan <bill at baddogconsulting.com> wrote:
 
 

 A few thoughts..

Seems like you are assuming that the SConstruct is in the top directory of a source tree.
This is not a safe assumption. I've seen builds where the SConscripts are scattered over several filesystems (building a project from several shared source repositories).
Can you add some tests? 
And documentation?
Any comments on the memory footprint of running this on a larger source tree? (where SCons is using a fair amount of memory already)

Is it really necessary to make a full copy of the source tree to a temp dir?
How would this impact the use of VariantDir?

-Bill

On Thu, Jan 14, 2016 at 11:56 AM, Schleimer, Ben via Scons-dev <scons-dev at scons.org> wrote:

Hi,
    I finished up the watch mode as best as I could and posted a pull request for it here https://bitbucket.org/scons/scons/pull-requests/289/watch-mode/diff

I am hoping that people will try it out for their incremental builds and see it if is working for them or not and if it helps their iteration time.
I tried to minimize changes in the core scons code. The only major change I had to make was to add
   SCons.Script._SConscript.global_list_of_processed_sconscipt_files


This is used to track where the SConscript files are so that a commonprefix can be established for all of the SConstruct files.

Sincerely
Ben


On Tuesday, December 29, 2015 8:23 PM, "Schleimer, Ben via Scons-dev" <scons-dev at scons.org> wrote:


>
>
>Hi William,
>
>   The non-interactive timings are not affected by the patch.
>
>
>Besides, I'm giving up on that patch and writing a completely separate Watch mode. It'll be started by "scons --watch" and it'll be a background compilation triggered by file changes.
>
>
>
>Once I get something simple working, I'll post the pull request so people can criticize it and hopefully improve it so that it works in a general way.
>
>
>Sincerely
>Ben
>
>
>
>
>On Tuesday, December 29, 2015 7:25 PM, William Blevins <wblevins001 at gmail.com> wrote:
>
>
>>
>>
>>No, I just wanted the profile from before the patch and after the patch on the same code. Unless I misunderstood your output from last time, you only supplied the information regarding the patched version correct?
>>
>>
>>V/R,
>>William
>>
>>
>>On Tue, Dec 29, 2015 at 11:25 PM, Schleimer, Ben via Scons-dev <scons-dev at scons.org> wrote:
>>
>>Hi,
>>>
>>>
>>>
>>>> Try running with --debug=time
>>>
>>>
>>>Did you want to see all of the build output?
>>>
>>>
>>>Ben
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>On Tuesday, December 29, 2015 2:57 PM, Bill Deegan <bill at baddogconsulting.com> wrote:
>>>
>>>
>>>>
>>>>
>>>>Try running with --debug=time
>>>>
>>>>
>>>>--debug=time
>>>>Prints various time profiling information:
the time spent executing each individual build command;
the total build time (time SCons ran from beginning to end);
the total time spent reading and executing SConscript files;
the total time spent SCons itself spend running
(that is, not counting reading and executing SConscript files);
and both the total time spent executing all build commands
and the elapsed wall-clock time spent executing those build commands.
(When scons is executed without the -j option,
the elapsed wall-clock time will typically
be slightly longer than the total time spent
executing all the build commands,
due to the SCons processing that takes place
in between executing each command.
When scons is executed with the -j option,
and your build configuration allows good parallelization,
the elapsed wall-clock time should
be significantly smaller than the
total time spent executing all the build commands,
since multiple build commands and
intervening SCons processing
should take place in parallel.)
>>>>-Bill
>>>>
>>>>
>>>>
>>>>
>>>>On Tue, Dec 29, 2015 at 2:47 PM, William Blevins <wblevins001 at gmail.com> wrote:
>>>>
>>>>I guess my point here was that I wanted to know where the time savings was happening since SCons does split out some process timings.
>>>>>
>>>>>
>>>>>I just wanted to see a regular build timing with and without your patch.
>>>>>
>>>>>
>>>>>V/R,
>>>>>William
>>>>>
>>>>>
>>>>>On Tue, Dec 29, 2015 at 7:07 PM, Schleimer, Ben via Scons-dev <scons-dev at scons.org> wrote:
>>>>>
>>>>>Hi William,
>>>>>>
>>>>>>
>>>>>>>Can you give the time saving using --debug=time for -j1 and -j4?
>>>>>>>
>>>>>>
>>>>>>   Sure,
>>>>>>
>>>>>>regular scons build (scons -jX --debug=time)
>>>>>>a clean build with -j1 is:
>>>>>>Total build time: 20.901390 seconds
>>>>>>Total SConscript file execution time: 0.149543 seconds
>>>>>>Total SCons execution time: 0.444850 seconds
>>>>>>Total command execution time: 20.306997 seconds
>>>>>>
>>>>>>
>>>>>>an incremental build with -j1 is:
>>>>>>Total build time: 2.439754 seconds
>>>>>>Total SConscript file execution time: 0.151794 seconds
>>>>>>Total SCons execution time: 0.337350 seconds
>>>>>>Total command execution time: 1.950610 seconds
>>>>>>
>>>>>>
>>>>>>a clean build with -j4 is:
>>>>>>Total build time: 7.158301 seconds
>>>>>>Total SConscript file execution time: 0.153021 seconds
>>>>>>Total SCons execution time: 0.107556 seconds
>>>>>>Total command execution time: 6.897724 seconds
>>>>>>
>>>>>>
>>>>>>an incremental build with -j4 is:
>>>>>>Total build time: 2.368260 seconds
>>>>>>Total SConscript file execution time: 0.151281 seconds
>>>>>>Total SCons execution time: 0.183930 seconds
>>>>>>Total command execution time: 2.033049 seconds
>>>>>>
>>>>>>
>>>>>>interactive build (scons --interactive -jX --debug=time)
>>>>>>a clean build with -j1:
>>>>>>time to do a build = 19.983217001 sec
>>>>>>
>>>>>>an incremental build with -j1:
>>>>>>time to do a build = 2.2111852169 sec
>>>>>>
>>>>>>a clean build with -j4:
>>>>>>time to do a build = 6.93614792824 sec
>>>>>>
>>>>>>an incremental build with -j4:
>>>>>>time to do a build = 2.17438697815 sec
>>>>>>
>>>>>>
>>>>>>It's consistently 0.2 seconds faster with the interactive build.
>>>>>>Not a huge amount but I'm not using that many SConscript files.
>>>>>>(5 SConscript files and 2 SConstruct files)
>>>>>>
>>>>>>
>>>>>>
>>>>>>Cheers
>>>>>>
>>>>>>
>>>>>>Ben
>>>>>>_______________________________________________
>>>>>>Scons-dev mailing list
>>>>>>Scons-dev at scons.org
>>>>>>https://pairlist2.pair.net/mailman/listinfo/scons-dev
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>>_______________________________________________
>>>>>Scons-dev mailing list
>>>>>Scons-dev at scons.org
>>>>>https://pairlist2.pair.net/mailman/listinfo/scons-dev
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>>
>>>_______________________________________________
>>>Scons-dev mailing list
>>>Scons-dev at scons.org
>>>https://pairlist2.pair.net/mailman/listinfo/scons-dev
>>>
>>>
>>
>>
>>
>
>_______________________________________________
>Scons-dev mailing list
>Scons-dev at scons.org
>https://pairlist2.pair.net/mailman/listinfo/scons-dev
>
>
>
_______________________________________________
Scons-dev mailing list
Scons-dev at scons.org
https://pairlist2.pair.net/mailman/listinfo/scons-dev




 
  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist2.pair.net/pipermail/scons-dev/attachments/20160114/366375ea/attachment-0001.html>

More information about the Scons-dev mailing list