[Scons-dev] Java Development

Fri Aug 1 19:09:31 EDT 2014

On Friday, August 01, 2014 04:01:54 PM Bill Deegan wrote:
> Jason,
> 
> Last I knew the output files differed by java version and not just java
> vendor.. ;)
> Any idea what version(s)?

The outputs have not changed for the Oracle javac between 1.6 and 1.7.

I don't know about 1.8, but it seems unlikely that it changed between 1.7 and 
1.8.  Easy enough to test if someone has Java 8 installed on their machine.

> 
> -Bill
> 
> 
> On Fri, Aug 1, 2014 at 3:36 PM, Kenny, Jason L <jason.l.kenny at intel.com>
> 
> wrote:
> >  For the tools here that use Java on the teams I know about use the
> > 
> > standard JDK from Oracle.
> > 
> > 
> > 
> > Jason
> > 
> > 
> > 
> > *From:* Scons-dev [mailto:scons-dev-bounces at scons.org] *On Behalf Of
> > *William Blevins
> > *Sent:* Friday, August 1, 2014 5:03 PM
> > 
> > *To:* SCons developer list
> > *Subject:* Re: [Scons-dev] Java Development
> > 
> > 
> > 
> > I can try to take a look at it, but if I understand correctly, the problem
> > with the Java emitter is that there is no standard for how a Java compiler
> > classes file creation, which means that the emitter needs to be different
> > for every Java compiler.
> > 
> > 
> > 
> > Which JDK do you use?
> > 
> > 
> > 
> > V/R,
> > 
> > William
> > 
> > 
> > 
> > On Fri, Aug 1, 2014 at 5:49 PM, Kenny, Jason L <jason.l.kenny at intel.com>
> > 
> > wrote:
> >  We just updated in Parts trunk the Java tool. I believe it a few tweak
> > 
> > based on the standard Scons tools. The note I have on this fix is “Fixed
> > Java tool emitter to correctly handle derived targets.”
> > 
> > 
> > 
> > You might want to look at it and see if it helps at all and is work
> > grabbing into SCons.
> > 
> > 
> > 
> > Here is a link the SVN file online..
> > http://parts.tigris.org/source/browse/parts/trunk/parts/parts/tools/javac.
> > py?revision=516&view=markup
> > 
> > 
> > 
> > Jason
> > 
> > 
> > 
> > *From:* Scons-dev [mailto:scons-dev-bounces at scons.org] *On Behalf Of
> > *William Blevins
> > *Sent:* Friday, August 1, 2014 4:23 PM
> > *To:* SCons developer list
> > *Subject:* Re: [Scons-dev] Java Development
> > 
> > 
> > 
> > SCons Java doesn't need to be that fancy, but I think the root problems
> > can be solved. The SCons java tool simply doesn't get the love that some
> > of
> > the other tools get.
> > 
> > 
> > 
> > I don't have the experience with the code base to know if my ideas are
> > workable, so I will need some guidance.
> > 
> > 
> > 
> > On Thu, Jul 31, 2014 at 11:04 PM, Mark A. Flacy <mflacy at verizon.net>
> > 
> > wrote:
> >  On Sunday, July 27, 2014 03:10:34 PM William Deegan wrote:
> > > William,
> > > On July 25, 2014 at 8:27:02 PM, William Blevins (wblevins001 at gmail.com)
> > > wrote:
> > > 
> > > Team,
> > > 
> > > I want to get another thread going for SCons Java development.
> > > 
> > > The SCons Java tool has a ton of error reports on Tigris including 7
> > > priority 1 issues.  At the moment, this tool doesn't stand a chance
> > 
> > against
> > 
> > > other Java competitors, and not because they are great tools.  I frankly
> > > hate ANT.  I have used Java support from SCons and it's seriously
> > 
> > painful;
> > 
> > > nothing like the C++ support.  Some other developers have made
> > > statements
> > > like "No one outside SCons builds Java programs more complicated than
> > 
> > hello
> > 
> > > world."  The SCons tool framework is great, and I would really like to
> > 
> > see
> > 
> > > the Java toolkit see some love.  It has potential to be a hidden gem,
> > 
> > and I
> > 
> > > want help out with this, but I don't have the experience to do this on
> > > my
> > > own, so firstly I'd like to list some of the biggest hurdles to users
> > 
> > SCons
> > 
> > > Java.  I'm not gonna try and propose any solutions at the moment.  I
> > > just
> > > want to see if I can get the group thinking about the problems.
> > > 
> > > 1. Adding resource files to a jar causes SCons
> > > segfault: http://scons.tigris.org/issues/show_bug.cgi?id=2550
> > > 
> > > I have firsthand experience with this bug.  The only way I could figure
> > 
> > out
> > 
> > > how to workaround it was to make a separate jar just for resources.
> > > 
> > > 2. Java emitter almost never gets the java output correct.
> > > 
> > > One of the many things I hate about ANT is that ANT is stupid.  It
> > > always
> > > executes a build even if code is up-to-date and I usually have to
> > > explicitly clean.  SCons COULD resolve both problems if the emitters
> > > worked.  The only way to get remotely consistent working build is to
> > > call
> > > Jar( 'buildDir' ) when everyone wants to do Jar( [ 'class1', ...
> > 
> > 'classN' ]
> > 
> > > ).
> > > 
> > > 3. Dependencies:
> > > 
> > > SCons does not automatically add classpath items as dependencies.  Why
> > 
> > do I
> > 
> > > need to do this manually?  This is what SCons does!  It's the heart and
> > > soul!
> > > 
> > > I believe this is because of item #6.
> > > 
> > > 4. Consistency:
> > > 
> > > Classpath tokens (among other items) do not behave the same as other
> > > builders. Example: I cannot use "#jar/item.jar" in the classpath without
> > > expanding via something like File(...).get_path().
> > > 
> > > 5. Interfaces:
> > > 
> > > Java(...) parameters and internal handling aren't intuitive and only
> > 
> > handles
> > 
> > > sources = 'directory' correctly.  It doesn't do lists of java files or
> > 
> > list
> > 
> > > of directories in a sane manner.
> > > http://scons.tigris.org/issues/show_bug.cgi?id=1772
> > > 
> > > Personally, I don't think that Java and Jar should be separate
> > > functions.
> > > 
> > >  How do you get to the classes then?  What about Javah!  I have an idea,
> > > 
> > > but that's outside the scope of this rant.
> > > 
> > > 6. Performance:
> > > 
> > > The dependency structure for Java exposes class files in a way that
> > 
> > creates
> > 
> > > tons of false positives.
> > > 
> > > SCons current:
> > > 
> > > classes1 = Java(...)
> > > 
> > > classes2 = Java(...)
> > > 
> > > Depends( classes2, classes1 ) # O( N^2 ) dependency graph with tons of
> > 
> > false
> > 
> > > positives
> > > 
> > > SCons if I have anything useful to say about it
> > > 
> > > jar1 = Jar( classes1 )
> > > 
> > > jar2 = Jar( classes2 )
> > > 
> > > Depends( jar2, jar1 ) # O( 1 ) which obviously fails in parallel builds
> > > currently.
> > > 
> > > I am currently data mining a production Java codebase to prove my point.
> > > Dirk and I have already discussed this issue somewhat; thanks Dirk :)
> > > 
> > > This actually causes the Task Master thread to get blocked on large jars
> > > reducing parallel efficiency in builds to None.
> > > 
> > > 
> > > The main issue here (if I understand SCons’ internals enough) is that
> > > SCons’s doing all dependencies on a per file basis. For many types of
> > > builds that works fine. For Java (building jars, and other issues) and
> > 
> > some
> > 
> > > other types of builds, that’s very inefficient.
> > > 
> > > There’s no “blob” of files where you have N inputs and M outputs, and
> > 
> > thus
> > 
> > > you get the N*M arcs in the graph.
> > > 
> > > Currently the only similar but not really similar enough is the Dir()
> > 
> > Node
> > 
> > > type.  But that has it’s own problems, which could be solved by a N*M
> > 
> > type
> > 
> > > node.
> > > 
> > > 
> > > 
> > > Other Java issues could likely be resolved building on top of such a new
> > > Node type.
> > > 
> > > Though resolving the anonymous and inner classes in a java file creating
> > > more than one class file and what it might be named is also still an
> > 
> > issue
> > 
> > > which the scanner and emitter try to solve by parsing the java files and
> > > figuring out the proper naming.  This of course is not (as I understand
> > 
> > it)
> > 
> > > formally defined as part of the java language and thus is a per compiler
> > > implementation detail.
> > 
> > In my opinion, you are wasting your time.
> > 
> > The way java produces artifacts is so different from the model that SCons
> > "expects" that you are attempting to cram an elephant down a shrew's
> > throat.
> > (or attempting to feed a Great White shark with ants, if you prefer) to
> > get
> > this to work.
> > 
> > SCons works *great* when there's one output per input, especially when you
> > can
> > deduce the output name from the input name and little else.  It also works
> > great when it is fairly cheap to determine what files a given file depends
> > upon.
> > 
> > Sadly, neither of those conditions exist with java source.
> > 
> > 
> > Between 2003 and 2010, I developed and maintained a python based build
> > tool
> > for a Java project that contained ~10K java source files.  It would
> > analyse the
> > java source files to figure out the package-level compile dependencies and
> > then
> > send lists of files to be compiled as a unit to a persistent compile
> > server (in
> > fact, you could run multiple compile servers at once to get a large degree
> > of
> > parallelism).  It would then analyze the generated class files to see if
> > the
> > total public/protected interface for the package changed, preventing
> > recompiles if all you did was change an implementation detail versus a
> > visible
> > method or attribute.
> > 
> > You could also register various hooks against packages (such as run rmic)
> > if
> > the package compiled correctly.  We did some funky stuff with
> > autogenerating
> > WSDL from java source and then generate client classes from the WSDL and
> > compiling those classes also.  Almost all of that was data driven.  It did
> > other neat stuff that I don't remember off the top of my head, but I've
> > got all
> > the source sitting around in various tla/hg repositories.
> > 
> > Nobody uses it any more.  Even me, and I thought about providing some it
> > in an
> > open source build tool.
> > 
> > For our Windows users, it was faster to nuke the output directories and
> > recompile everything in one invocation of javac. (Well, 2 invocations
> > since
> > some of the funky stuff was to use the .class files from the first compile
> > to
> > generate the wsdl and use *that* wsdl to generate java source as part of
> > the
> > second bulk compile.) Most of the time difference had to do with the way
> > the
> > python tool talked to the compile processes by sending the list of file
> > names
> > through a pipe to the other process and then flushing the pipe.  The rest
> > had
> > to do with the time spent parsing the files (and I maintained a cache of
> > parsed
> > file information that contained the last mtime of the source files and
> > would not
> > process a file whose mtime was the same as what was recorded).
> > 
> > For our Linux users (me, for one), it was not quite a wash to change the
> > behavior but the delta wasn't enough to worry about.
> > 
> > I'm more than willing to discuss this and also to be proven wrong.
> > 
> > --
> > Mark A. Flacy
> > _______________________________________________
> > Scons-dev mailing list
> > Scons-dev at scons.org
> > http://two.pairlist.net/mailman/listinfo/scons-dev
> > 
> > 
> > 
> > 
> > _______________________________________________
> > Scons-dev mailing list
> > Scons-dev at scons.org
> > http://two.pairlist.net/mailman/listinfo/scons-dev
> > 
> > 
> > 
> > _______________________________________________
> > Scons-dev mailing list
> > Scons-dev at scons.org
> > http://two.pairlist.net/mailman/listinfo/scons-dev
-- 
Mark A. Flacy