[Scons-dev] Java Development

Fri Aug 1 18:36:37 EDT 2014

For the tools here that use Java on the teams I know about use the standard JDK from Oracle.

Jason

From: Scons-dev [mailto:scons-dev-bounces at scons.org] On Behalf Of William Blevins
Sent: Friday, August 1, 2014 5:03 PM
To: SCons developer list
Subject: Re: [Scons-dev] Java Development

I can try to take a look at it, but if I understand correctly, the problem with the Java emitter is that there is no standard for how a Java compiler classes file creation, which means that the emitter needs to be different for every Java compiler.

Which JDK do you use?

V/R,
William

On Fri, Aug 1, 2014 at 5:49 PM, Kenny, Jason L <jason.l.kenny at intel.com<mailto:jason.l.kenny at intel.com>> wrote:
We just updated in Parts trunk the Java tool. I believe it a few tweak based on the standard Scons tools. The note I have on this fix is “Fixed Java tool emitter to correctly handle derived targets.”

You might want to look at it and see if it helps at all and is work grabbing into SCons.

Here is a link the SVN file online.. http://parts.tigris.org/source/browse/parts/trunk/parts/parts/tools/javac.py?revision=516&view=markup

Jason

From: Scons-dev [mailto:scons-dev-bounces at scons.org<mailto:scons-dev-bounces at scons.org>] On Behalf Of William Blevins
Sent: Friday, August 1, 2014 4:23 PM
To: SCons developer list
Subject: Re: [Scons-dev] Java Development

SCons Java doesn't need to be that fancy, but I think the root problems can be solved. The SCons java tool simply doesn't get the love that some of the other tools get.

I don't have the experience with the code base to know if my ideas are workable, so I will need some guidance.

On Thu, Jul 31, 2014 at 11:04 PM, Mark A. Flacy <mflacy at verizon.net<mailto:mflacy at verizon.net>> wrote:
On Sunday, July 27, 2014 03:10:34 PM William Deegan wrote:
> William,
> On July 25, 2014 at 8:27:02 PM, William Blevins (wblevins001 at gmail.com<mailto:wblevins001 at gmail.com>)
> wrote:
>
> Team,
>
> I want to get another thread going for SCons Java development.
>
> The SCons Java tool has a ton of error reports on Tigris including 7
> priority 1 issues.  At the moment, this tool doesn't stand a chance against
> other Java competitors, and not because they are great tools.  I frankly
> hate ANT.  I have used Java support from SCons and it's seriously painful;
> nothing like the C++ support.  Some other developers have made statements
> like "No one outside SCons builds Java programs more complicated than hello
> world."  The SCons tool framework is great, and I would really like to see
> the Java toolkit see some love.  It has potential to be a hidden gem, and I
> want help out with this, but I don't have the experience to do this on my
> own, so firstly I'd like to list some of the biggest hurdles to users SCons
> Java.  I'm not gonna try and propose any solutions at the moment.  I just
> want to see if I can get the group thinking about the problems.
>
> 1. Adding resource files to a jar causes SCons
> segfault: http://scons.tigris.org/issues/show_bug.cgi?id=2550
>
> I have firsthand experience with this bug.  The only way I could figure out
> how to workaround it was to make a separate jar just for resources.
>
> 2. Java emitter almost never gets the java output correct.
>
> One of the many things I hate about ANT is that ANT is stupid.  It always
> executes a build even if code is up-to-date and I usually have to
> explicitly clean.  SCons COULD resolve both problems if the emitters
> worked.  The only way to get remotely consistent working build is to call
> Jar( 'buildDir' ) when everyone wants to do Jar( [ 'class1', ... 'classN' ]
> ).
>
> 3. Dependencies:
>
> SCons does not automatically add classpath items as dependencies.  Why do I
> need to do this manually?  This is what SCons does!  It's the heart and
> soul!
>
> I believe this is because of item #6.
>
> 4. Consistency:
>
> Classpath tokens (among other items) do not behave the same as other
> builders. Example: I cannot use "#jar/item.jar" in the classpath without
> expanding via something like File(...).get_path().
>
> 5. Interfaces:
>
> Java(...) parameters and internal handling aren't intuitive and only handles
> sources = 'directory' correctly.  It doesn't do lists of java files or list
> of directories in a sane manner.
> http://scons.tigris.org/issues/show_bug.cgi?id=1772
>
> Personally, I don't think that Java and Jar should be separate functions.
>  How do you get to the classes then?  What about Javah!  I have an idea,
> but that's outside the scope of this rant.
>
> 6. Performance:
>
> The dependency structure for Java exposes class files in a way that creates
> tons of false positives.
>
> SCons current:
>
> classes1 = Java(...)
>
> classes2 = Java(...)
>
> Depends( classes2, classes1 ) # O( N^2 ) dependency graph with tons of false
> positives
>
> SCons if I have anything useful to say about it
>
> jar1 = Jar( classes1 )
>
> jar2 = Jar( classes2 )
>
> Depends( jar2, jar1 ) # O( 1 ) which obviously fails in parallel builds
> currently.
>
> I am currently data mining a production Java codebase to prove my point.
> Dirk and I have already discussed this issue somewhat; thanks Dirk :)
>
> This actually causes the Task Master thread to get blocked on large jars
> reducing parallel efficiency in builds to None.
>
>
> The main issue here (if I understand SCons’ internals enough) is that
> SCons’s doing all dependencies on a per file basis. For many types of
> builds that works fine. For Java (building jars, and other issues) and some
> other types of builds, that’s very inefficient.
>
> There’s no “blob” of files where you have N inputs and M outputs, and thus
> you get the N*M arcs in the graph.
>
> Currently the only similar but not really similar enough is the Dir() Node
> type.  But that has it’s own problems, which could be solved by a N*M type
> node.
>
>
>
> Other Java issues could likely be resolved building on top of such a new
> Node type.
>
> Though resolving the anonymous and inner classes in a java file creating
> more than one class file and what it might be named is also still an issue
> which the scanner and emitter try to solve by parsing the java files and
> figuring out the proper naming.  This of course is not (as I understand it)
> formally defined as part of the java language and thus is a per compiler
> implementation detail.
In my opinion, you are wasting your time.

The way java produces artifacts is so different from the model that SCons
"expects" that you are attempting to cram an elephant down a shrew's throat.
(or attempting to feed a Great White shark with ants, if you prefer) to get
this to work.

SCons works *great* when there's one output per input, especially when you can
deduce the output name from the input name and little else.  It also works
great when it is fairly cheap to determine what files a given file depends upon.

Sadly, neither of those conditions exist with java source.

Between 2003 and 2010, I developed and maintained a python based build tool
for a Java project that contained ~10K java source files.  It would analyse the
java source files to figure out the package-level compile dependencies and then
send lists of files to be compiled as a unit to a persistent compile server (in
fact, you could run multiple compile servers at once to get a large degree of
parallelism).  It would then analyze the generated class files to see if the
total public/protected interface for the package changed, preventing
recompiles if all you did was change an implementation detail versus a visible
method or attribute.

You could also register various hooks against packages (such as run rmic) if
the package compiled correctly.  We did some funky stuff with autogenerating
WSDL from java source and then generate client classes from the WSDL and
compiling those classes also.  Almost all of that was data driven.  It did
other neat stuff that I don't remember off the top of my head, but I've got all
the source sitting around in various tla/hg repositories.

Nobody uses it any more.  Even me, and I thought about providing some it in an
open source build tool.

For our Windows users, it was faster to nuke the output directories and
recompile everything in one invocation of javac. (Well, 2 invocations since
some of the funky stuff was to use the .class files from the first compile to
generate the wsdl and use *that* wsdl to generate java source as part of the
second bulk compile.) Most of the time difference had to do with the way the
python tool talked to the compile processes by sending the list of file names
through a pipe to the other process and then flushing the pipe.  The rest had
to do with the time spent parsing the files (and I maintained a cache of parsed
file information that contained the last mtime of the source files and would not
process a file whose mtime was the same as what was recorded).

For our Linux users (me, for one), it was not quite a wash to change the
behavior but the delta wasn't enough to worry about.

I'm more than willing to discuss this and also to be proven wrong.

--
Mark A. Flacy
_______________________________________________
Scons-dev mailing list
Scons-dev at scons.org<mailto:Scons-dev at scons.org>
http://two.pairlist.net/mailman/listinfo/scons-dev

_______________________________________________
Scons-dev mailing list
Scons-dev at scons.org<mailto:Scons-dev at scons.org>
http://two.pairlist.net/mailman/listinfo/scons-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://two.pairlist.net/pipermail/scons-dev/attachments/20140801/c722be07/attachment-0001.html>