[Scons-dev] sconsign database discussion

[Mats Wichmann]

I know there isn't a lot of discussion going on here, but thought this 
would be a good place to have a recorded examination of this topic.

There's been some discussion of making sconsign be a "real" database, 
not least because the "dump the entire dictionary (pickling as we go) to 
database on exit" behavior is pretty irritating - quite a slow operation 
if you didn't do any work.

I prototyped using an existing pypi package to make a disk-backed 
(sqlite) version of this, on the theory that something that exists is a 
better starting point for a quick prototype  than something that doesn't 
exist yet :)   Turns out it was quite simple to make the sconsign 
utility be able to dump the database to a new format, and reading and 
displaying such a converted database gives the same results as reading 
and displaying .sconsign.dblite.

But before going much further down this path, the current implementation 
actually imposes a bit on the internal design: scons thinks that a disk 
signature database is something you read in, close, and then ignore, 
using only the in-memory dict that has been created from it, until you 
quit, when you write out a new copy and replace the old one with it. 
Yes, a good piece of that logic is in the SCons/dblite.py module, so 
it's not *fully* wired into scons as a whole, but the assumption is 
still there. Is it okay to switch to a version where a change to the 
signature dict while running would commit behind the scenes to the 
sqlite db, and thus that the in-memory db is not disconnected from the 
on-disk db?

There are also the cases of having additional databases... if you 
specify a repository, its sconsign is used but in read-only mode, that 
is, we never update that.  What should the disk-backed version do if 
asked to open read-only?

What should a disk-cached version do with the request to be read-only? 
When I hacked it together, all it does is this:

     def __setitem__(self, key, value):
         # just skip setting if db is read-only
         # should we raise an error instead?
         if self._writable:
             self._dict[key] = value

The comment asks part of the question... is "silently ignore updates" 
sensible?  That doesn't allow the in-memory part to be updated either - 
but should you ever even be updating the in-memory part of a copy you 
asked  to open read-only?

I'm assuming co-existence would be desirable - that you could set a 
sqlite version for your main project, but still want to be able to read 
a dblite version from a repository at the same time?

Thoughts?