[Scons-dev] File-based build tools will never handle Java

[Greg Ward]

On 06 September 2012, Chris BeHanna said:

> 	:nod:

> 

> 	That performance hit is why $EMPLOYER is abandoning SCons.  :-(


OT: mind if I ask what they/you are switching to? I've spent the last
day learning tup, and it has given me some interesting ideas. Curious
what else is out there that people are adopting.


> > However, something valuable is lost when you simplify dependencies so

> > dramatically. Specifically, I implemented incremental testing, which

> > was a win, but not nearly as much of a win as it could have been. E.g.

> > if you modify a source file in a low-level component that everything

> > else depends on, then you end up recompiling the world (no big deal:

> > javac is fast enough), and re-running all the tests (not nice: the

> > whole test suite is 20-30 min).

> 



> Can the tests be broken down to a per-jar basis?  If so, and if you

> can identify the jar or jars that changed, then you only run the tests

> for the jars that changed.


Oops, I overstated things for dramatic purposes. In fact, in the end I
had something that works pretty much as you described. Here's a
simplified version of how things looked:

  common-main.jar:
    com/example/utils/StringUtils.class
    com/example/utils/DateUtils.class
  common-test.jar:
    com/example/utils/TestStringUtils.class
    com/example/utils/TestDateUtils.class

  guilibs-main.jar:
    com/example/swingutils/CustomTable.class
  guilibs-test.jar:
    com/example/swingutils/TestCustomTable.class

  bigapp-main.jar:
    com/example/bigapp/ApplicationLogic.class
  bigapp-test.jar:
    com/example/bigapp/TestApplicationLogic.class

(Expand that to 75-80 jar files and ~20,000 .class files to get a feel
for how things really looked.)

The DAG looks like this:

  bigapp-test.jar : source/test/com/example/bigapp/TestApplicationLogic.java
                    guilibs-main.jar
                    common-main.jar
		    junit.jar
		    javac
		    jar
  bigapp-main.jar : source/test/com/example/bigapp/ApplicationLogic.java
                    guilibs-main.jar
                    common-main.jar
		    javac
		    jar

No .class files, just as you suggested: that's the sausage. The build
action is [Remove(classdir), Mkdir(classdir), javac -d classdir, jar].

So if I modify ApplicationLogic.java, here's what SCons does:

  ApplicationLogic.java changed:
    javac + jar to rebuild bigapp-main.jar
  bigapp-main.jar changed:
    javac + jar to rebuild bigapp-test.jar
  bigapp-test.jar changed:
    run com.example.bigapp.TestApplicationLogic
    touch runtests/com.example.bigapp.TestApplicationLogic.stamp

Fine so far, right? Now what happens if I modify DateUtils.java?

  DateUtils.java changed:
    javac + jar to rebuild common-main.jar
  common-main.jar changed:
    javac + jar to rebuild common-test.jar
    javac + jar to rebuild guilibs-main.jar
    javac + jar to rebuild bigapp-main.jar
    run com.example.utils.TestDateUtils
    touch runtests/com.example.utils.TestDateUtil.stamp
    run com.example.utils.TestStringUtils
    touch runtests/com.example.utils.TestStringUtils.stamp

So you can see that TestStringUtils is run unnnecessarily. When you
modify a component with 3000 classes and 1000 tests, that's a lot of
unnnecessary test runs.

(No, we should not have had components with 3000 classes in them. I'd
say it was legacy code, but some of that code was brand new. *sigh*)

       Greg
-- 
Greg Ward                                http://www.gerg.ca/
Any priest or shaman must be presumed guilty until proven innocent.