[Scons-dev] [Scons-users] Bug in up_to_date for directory targets
William Blevins
wblevins001 at gmail.com
Tue Mar 21 18:33:42 EDT 2017
I also checked scons 2.4.1 and 2.3.0, so this goes back a while apparently?
Or maybe this never worked right...
Can someone confirm?
V/R,
William
On Tue, Mar 21, 2017 at 6:09 PM, William Blevins <wblevins001 at gmail.com>
wrote:
> Moving to scons-dev list. Notice email change ^
>
> When I look at the "sconsign -c .scons.dblite ", I noticed that the source
> file does not have a csig, so the default decider will not cause a rebuild
> (MD5-only).
>
> $ sconsign .sconsign.dblite
>> === .:
>> SConstruct: None 1490133556 120
>> permissions: None 1490133631 5
>> === /bin:
>> chmod: None 1487766225 60296
>> mkdir: None 1487766225 81032
>>
>
> The dependency tree looks right:
>
> $ scons --tree=all,status
>> scons: Reading SConscript files ...
>> scons: done reading SConscript files.
>> scons: Building targets ...
>> scons: `.' is up to date.
>> E = exists
>> R = exists in repository only
>> b = implicit builder
>> B = explicit builder
>> S = side effect
>> P = precious
>> A = always build
>> C = current
>> N = no clean
>> H = no cache
>>
>> [E b C ]+-.
>> [E C ] +-SConstruct
>> [E B C ] +-aaa
>> [E C ] | +-permissions
>> [E C ] | +-/bin/mkdir
>> [E C ] | +-/bin/chmod
>> [E C ] +-permissions
>>
>
> Even if I change the decider I still don't get a rebuild:
>
> env = DefaultEnvironment()
>>
>> env.Decider('MD5-timestamp')
>> env.Command(Dir('aaa'), File('permissions'), ['mkdir $TARGET', 'chmod
>> `cat permissions` $TARGET'])
>>
>
> Information:
>
> $ scons --version
>> SCons by Steven Knight et al.:
>> script: v2.5.1.rel_2.5.1:3735:9dc6cee5c168[MODIFIED], 2016/11/03
>> 14:02:02, by bdbaddog on mongodog
>> engine: v2.5.1.rel_2.5.1:3735:9dc6cee5c168[MODIFIED], 2016/11/03
>> 14:02:02, by bdbaddog on mongodog
>> engine path: ['/usr/lib/scons/SCons']
>> Copyright (c) 2001 - 2016 The SCons Foundation
>>
>
>>
> $ python --version
>>
> Python 2.7.13
>>
>
> SCons distribution per Debian Apt 2.5.1-1
>
> Thoughts?
> William
>
>
>
>
>
>
> On Tue, Mar 21, 2017 at 5:39 PM, William Blevins <wblevins001 at gmail.com>
> wrote:
>
>> I see what you are saying. I need to look into it some. Did this work in
>> a previous version of SCons? What version did it stop working in? What
>> version of Python are you using?
>>
>> It's a bit strange. I needed to fix your example, but I got it to
>> replicate. At a glance, the dependency tree looks correct, but nothing
>> happens when the permissions file is updated.
>>
>> V/R,
>> William
>>
>> On Tue, Mar 21, 2017 at 2:09 PM, Manish Vachharajani <
>> manishv at unbounded.systems> wrote:
>>
>>> Let me try to get the discussion back to my main point. There is a bug
>>> in scons and there should be a ticket for it and a fix (unless I've
>>> misunderstood something, which from the discussion I don't think I have).
>>> The nature of the bug is this:
>>>
>>> 1) The user tells scons that there is a target (foo in my example)
>>> 2) The target depends on a source (foo-contents.txt in my example)
>>> 3) When the target is out of date with respect to the source, rerun the
>>> builder
>>>
>>> It isn't scons' job, in my opinion, to second guess me and decide that
>>> it doesn't really need to run the builder, or that the target isn't the
>>> precise output set of the builder etc. The point is that there is a target
>>> which is out of date with respect to the source and scons is not running
>>> the builder. Everything else, IMHO, is an aside. Does that make it
>>> clear? I'm not trying to be rude but something is getting lost when I'm
>>> trying to be indirect.
>>>
>>> I'm glad to discuss what I believe is the root cause based on my
>>> investigations in the python debugger. 99% of the time this doesn't matter
>>> because there is almost never a directory only set of targets with a
>>> builder that isn't simply a mkdir builder or equivalent. Note, however, if
>>> the builder were something like:
>>>
>>> env.Command(['/a/b/c'],
>>> ['permissions.txt'],
>>> [mkdir $TARGET,
>>> chmod `permissions.txt` $TARGET])
>>>
>>> scons would fail to update the permissions on the directory when
>>> permissions.txt changed if /a/b/c already existed. I hope that makes it
>>> more clear the exact nature of the issue. My example is simple because it
>>> is the smallest SConstruct I could create to demonstrate the issue.
>>>
>>> For my specific case, I have workarounds, and a custom builder that
>>> uses them. A custom builder doesn't help without the workarounds because
>>> the problem is in the up to date logic, not the scanner and action logic in
>>> scons. Oh, and while package managers let you see content, it doesn't help
>>> if the package is going to fetched from the network and I need to calculate
>>> dependencies before the package is fetched.
>>>
>>> Manish
>>>
>>>
>>>
>>> On Tue, Mar 21, 2017 at 10:53 AM, William Blevins <wblevins001 at gmail.com
>>> > wrote:
>>>
>>>> Manish,
>>>>
>>>> Ah, ok. I was confused when your original example seemed so simple. You
>>>> should probably create a custom builder then:
>>>> https://bitbucket.org/scons/scons/wiki/ToolsForFools
>>>>
>>>> It isn't a bug that SCons doesn't rebuild targets it doesn't have. The
>>>> issue is that you need to tell SCons about said targets. Do this via a
>>>> custom builder.
>>>>
>>>> I haven't used npm, but most package managers have a way to list the
>>>> contents of a package, so just convert that list into a target set. I would
>>>> need more information about the source -> target dependency chain in terms
>>>> of actual vs expected in order to say much more.
>>>>
>>>> Sorry for the fix response and run, but normally I don't get time to
>>>> answer emails this early in the day. Let us know if you think a custom
>>>> builder will not solve your problem.
>>>>
>>>> V/R,
>>>> William
>>>>
>>>> On Tue, Mar 21, 2017 at 12:22 PM, Manish Vachharajani <
>>>> manishv at unbounded.systems> wrote:
>>>>
>>>>> If you look at my original message, the problem is that in my real use
>>>>> case, I have no idea what the file targets will be. The only target I know
>>>>> will be built by the builder is a set of directories. Your example simply
>>>>> causes scons to have a file target in the second Command builder which
>>>>> works around the bug as I describe in my original email. In real life, my
>>>>> builder looks something like this;
>>>>>
>>>>> env.Command(['node_modules/minimist', 'node_modules/@types/node,
>>>>> ...], #these are directories
>>>>> ['package.json'],
>>>>> ['npm install --global-style'],
>>>>> chdir=Dir('.')) #chdir because npm doesn't have a way to
>>>>> install elsewhere
>>>>>
>>>>> In this case, I have no idea what files npm install will create. The
>>>>> targets array is actually created by reading package.json and looking at
>>>>> the dependencies and devDependencies fields of the main object, so those
>>>>> aren't even hard-coded in the SConstruct/SConscript.
>>>>>
>>>>> Again, if I fake out scons by have a 'node_modules/.built' File target
>>>>> in the list and add a 'touch node_modules/.built' command to the builder
>>>>> after 'npm install', all will work as expected because there is a File
>>>>> target. In your example, things work because the second builder has a file
>>>>> target and SCons will only run the first builder in your example when foo
>>>>> does not exist, regardless of the state of foo-contents.txt in relation to
>>>>> what is in .sconsign. That is the essence of the bug, if the target
>>>>> directory exists and it is the only target, the builder will not run
>>>>> regardless of the state of the sources.
>>>>>
>>>>> Manish
>>>>>
>>>>>
>>>>> On Mon, Mar 20, 2017 at 9:09 PM, William Blevins <
>>>>> wblevins001 at gmail.com> wrote:
>>>>>
>>>>>> Manish,
>>>>>>
>>>>>> * Why does my example not work?
>>>>>>
>>>>>> SCons Comand Builder parameters are target(s), source(s), command(s):
>>>>>> http://scons.org/doc/HTML/scons-user.html#chap-builders-commands
>>>>>>
>>>>>> Foo is the target parameter, and foo-contents.txt is the source
>>>>>> parameter. Targets explicitly depend on sources. In your example, directory
>>>>>> foo depends on source file foo-contents.txt; target file copy does not
>>>>>> depend on source file.
>>>>>>
>>>>>> * How do I fix this issue?
>>>>>>
>>>>>> 1. Call Command Builder correctly: Command('foo', 'foo-contents',
>>>>>> 'mkdir $TARGET') which creates file dependency on directory, and then
>>>>>> Command('foo/foo-contents.txt', 'foo-contents.txt', 'cp $SOURCE
>>>>>> $TARGET') which creates file target dependency on file source.
>>>>>>
>>>>>> 2: [PREFERRED] Use the SCons FileSystem factory
>>>>>> http://scons.org/doc/HTML/scons-user.html#chap-factories
>>>>>>
>>>>>> V/R,
>>>>>> William
>>>>>>
>>>>>>
>>>>>> On Mon, Mar 20, 2017 at 1:47 PM, Manish Vachharajani <
>>>>>> manishv at unbounded.systems> wrote:
>>>>>>
>>>>>>> I believe there is a bug in how up to date is computed for file
>>>>>>> nodes and directory targets. Below is a SConstruct and run output
>>>>>>> illustrating the bug.
>>>>>>>
>>>>>>> ##############
>>>>>>> # SConstruct
>>>>>>> env = Environment()
>>>>>>> targets = env.Command('foo',
>>>>>>> 'foo-contents.txt',
>>>>>>> ['mkdir foo',
>>>>>>> 'cp foo-contents.txt foo'])
>>>>>>> env.Default(targets)
>>>>>>>
>>>>>>> ##############
>>>>>>> # Shell log
>>>>>>>
>>>>>>> # Create the file that will be installed into foo/foo-contents.txt
>>>>>>> $ echo "Hello, " >> foo-contents.txt
>>>>>>> # All is well, scons runs the command builder as expected
>>>>>>> $ scons
>>>>>>>
>>>>>>> scons: Reading SConscript files ...
>>>>>>> scons: done reading SConscript files.
>>>>>>> scons: Building targets ...
>>>>>>> mkdir foo
>>>>>>> cp foo-contents.txt foo
>>>>>>> scons: done building targets.
>>>>>>>
>>>>>>> # All is well, the target is up to date as expected
>>>>>>> $ scons
>>>>>>>
>>>>>>> scons: Reading SConscript files ...
>>>>>>> scons: done reading SConscript files.
>>>>>>> scons: Building targets ...
>>>>>>> scons: `foo' is up to date.
>>>>>>> scons: done building targets.
>>>>>>>
>>>>>>> # Uh-oh, source is updated but the target is considered up to date!
>>>>>>> $ echo "World!" >> foo-contents.txt
>>>>>>> $ scons
>>>>>>>
>>>>>>> scons: Reading SConscript files ...
>>>>>>> scons: done reading SConscript files.
>>>>>>> scons: Building targets ...
>>>>>>> scons: `foo' is up to date.
>>>>>>> scons: done building targets.
>>>>>>>
>>>>>>> For the bug to trigger, all targets for a builder must be
>>>>>>> directories. The target directory must exist and the source file must
>>>>>>> exist and be out of date due to a content update (resulting in a signature
>>>>>>> mismatch).
>>>>>>>
>>>>>>> I think the problem is that is_up_to_date for File nodes only checks
>>>>>>> the signatures of its children, which avoids this bug as long as one of the
>>>>>>> builder targets is a File and not a Directory. However, I haven't done
>>>>>>> enough analysis to be sure that this is the issue.
>>>>>>>
>>>>>>> A work around is to make sure that you have a file (or create a
>>>>>>> dummy .built file) and include it as a target.
>>>>>>>
>>>>>>> This is showing up specifically for us when trying to calculate
>>>>>>> dependencies for a package manager (npm) where we have no insight into the
>>>>>>> contents of the resulting directory.
>>>>>>>
>>>>>>> Manish
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Scons-users mailing list
>>>>>>> Scons-users at scons.org
>>>>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Scons-users mailing list
>>>>>> Scons-users at scons.org
>>>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>>>
>>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Scons-users mailing list
>>>>> Scons-users at scons.org
>>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>>
>>>>>
>>>>
>>>> _______________________________________________
>>>> Scons-users mailing list
>>>> Scons-users at scons.org
>>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>>
>>>>
>>>
>>> _______________________________________________
>>> Scons-users mailing list
>>> Scons-users at scons.org
>>> https://pairlist4.pair.net/mailman/listinfo/scons-users
>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://pairlist2.pair.net/pipermail/scons-dev/attachments/20170321/f43fd4d3/attachment-0001.html>
More information about the Scons-dev
mailing list