[Scons-dev] Java Development
Mark A. Flacy
mflacy at verizon.net
Mon Aug 4 14:05:53 EDT 2014
On Friday, August 01, 2014 07:54:16 PM William Blevins wrote:
> The issue is that we are guessing as to what syntax causes the compiler to
> emit an extra class file.
http://docs.oracle.com/javase/specs/jls/se7/html/jls-13.html#jls-13.1 may be
of some use. I've never used a local class, myself, but I see that they are
allowed.
>
> On Fri, Aug 1, 2014 at 7:49 PM, Mark A. Flacy <mflacy at verizon.net> wrote:
> > On Friday, August 01, 2014 05:22:39 PM William Blevins wrote:
> > > SCons Java doesn't need to be that fancy, but I think the root problems
> >
> > can
> >
> > > be solved. The SCons java tool simply doesn't get the love that some of
> >
> > the
> >
> > > other tools get.
> >
> > One common use case is to generate java source from the xjc tool (which
> > converts XML schema to java classes) for use by other java classes.
> >
> > The emitter in that case would need to parse a bindings file as well as
> > the XML
> > schema(s) that the binding file uses. However, the resulting java files
> > should
> > have no dependencies on any external libraries, so they could be compiled
> > as a
> > unit independently from anything using them.
> >
> > And there are Annotations, which can be used to (and *are* used to)
> > generate
> > source. See http://deors.wordpress.com/2011/09/26/annotation-types/ for a
> > discussion of how to do that.
> >
> > > I don't have the experience with the code base to know if my ideas are
> > > workable, so I will need some guidance.
> > >
> > > On Thu, Jul 31, 2014 at 11:04 PM, Mark A. Flacy <mflacy at verizon.net>
> >
> > wrote:
> > > > On Sunday, July 27, 2014 03:10:34 PM William Deegan wrote:
> > > > > William,
> > > > > On July 25, 2014 at 8:27:02 PM, William Blevins (
> >
> > wblevins001 at gmail.com)
> >
> > > > > wrote:
> > > > >
> > > > > Team,
> > > > >
> > > > > I want to get another thread going for SCons Java development.
> > > > >
> > > > > The SCons Java tool has a ton of error reports on Tigris including 7
> > > > > priority 1 issues. At the moment, this tool doesn't stand a chance
> > > >
> > > > against
> > > >
> > > > > other Java competitors, and not because they are great tools. I
> >
> > frankly
> >
> > > > > hate ANT. I have used Java support from SCons and it's seriously
> > > >
> > > > painful;
> > > >
> > > > > nothing like the C++ support. Some other developers have made
> > > > > statements
> > > > > like "No one outside SCons builds Java programs more complicated
> > > > > than
> > > >
> > > > hello
> > > >
> > > > > world." The SCons tool framework is great, and I would really like
> >
> > to
> >
> > > > see
> > > >
> > > > > the Java toolkit see some love. It has potential to be a hidden
> > > > > gem,
> > > >
> > > > and I
> > > >
> > > > > want help out with this, but I don't have the experience to do this
> >
> > on
> >
> > > > > my
> > > > > own, so firstly I'd like to list some of the biggest hurdles to
> > > > > users
> > > >
> > > > SCons
> > > >
> > > > > Java. I'm not gonna try and propose any solutions at the moment. I
> > > > > just
> > > > > want to see if I can get the group thinking about the problems.
> > > > >
> > > > > 1. Adding resource files to a jar causes SCons
> > > > > segfault: http://scons.tigris.org/issues/show_bug.cgi?id=2550
> > > > >
> > > > > I have firsthand experience with this bug. The only way I could
> >
> > figure
> >
> > > > out
> > > >
> > > > > how to workaround it was to make a separate jar just for resources.
> > > > >
> > > > > 2. Java emitter almost never gets the java output correct.
> > > > >
> > > > > One of the many things I hate about ANT is that ANT is stupid. It
> > > > > always
> > > > > executes a build even if code is up-to-date and I usually have to
> > > > > explicitly clean. SCons COULD resolve both problems if the emitters
> > > > > worked. The only way to get remotely consistent working build is to
> > > > > call
> > > > > Jar( 'buildDir' ) when everyone wants to do Jar( [ 'class1', ...
> > > >
> > > > 'classN' ]
> > > >
> > > > > ).
> > > > >
> > > > > 3. Dependencies:
> > > > >
> > > > > SCons does not automatically add classpath items as dependencies.
> >
> > Why
> >
> > > > do I
> > > >
> > > > > need to do this manually? This is what SCons does! It's the heart
> >
> > and
> >
> > > > > soul!
> > > > >
> > > > > I believe this is because of item #6.
> > > > >
> > > > > 4. Consistency:
> > > > >
> > > > > Classpath tokens (among other items) do not behave the same as other
> > > > > builders. Example: I cannot use "#jar/item.jar" in the classpath
> >
> > without
> >
> > > > > expanding via something like File(...).get_path().
> > > > >
> > > > > 5. Interfaces:
> > > > >
> > > > > Java(...) parameters and internal handling aren't intuitive and only
> > > >
> > > > handles
> > > >
> > > > > sources = 'directory' correctly. It doesn't do lists of java files
> >
> > or
> >
> > > > list
> > > >
> > > > > of directories in a sane manner.
> > > > > http://scons.tigris.org/issues/show_bug.cgi?id=1772
> > > > >
> > > > > Personally, I don't think that Java and Jar should be separate
> > > > > functions.
> > > > >
> > > > > How do you get to the classes then? What about Javah! I have an
> >
> > idea,
> >
> > > > > but that's outside the scope of this rant.
> > > > >
> > > > > 6. Performance:
> > > > >
> > > > > The dependency structure for Java exposes class files in a way that
> > > >
> > > > creates
> > > >
> > > > > tons of false positives.
> > > > >
> > > > > SCons current:
> > > > >
> > > > > classes1 = Java(...)
> > > > >
> > > > > classes2 = Java(...)
> > > > >
> > > > > Depends( classes2, classes1 ) # O( N^2 ) dependency graph with tons
> >
> > of
> >
> > > > false
> > > >
> > > > > positives
> > > > >
> > > > > SCons if I have anything useful to say about it
> > > > >
> > > > > jar1 = Jar( classes1 )
> > > > >
> > > > > jar2 = Jar( classes2 )
> > > > >
> > > > > Depends( jar2, jar1 ) # O( 1 ) which obviously fails in parallel
> >
> > builds
> >
> > > > > currently.
> > > > >
> > > > > I am currently data mining a production Java codebase to prove my
> >
> > point.
> >
> > > > > Dirk and I have already discussed this issue somewhat; thanks Dirk
> > > > > :)
> > > > >
> > > > > This actually causes the Task Master thread to get blocked on large
> >
> > jars
> >
> > > > > reducing parallel efficiency in builds to None.
> > > > >
> > > > >
> > > > > The main issue here (if I understand SCons’ internals enough) is
> > > > > that
> > > > > SCons’s doing all dependencies on a per file basis. For many types
> > > > > of
> > > > > builds that works fine. For Java (building jars, and other issues)
> >
> > and
> >
> > > > some
> > > >
> > > > > other types of builds, that’s very inefficient.
> > > > >
> > > > > There’s no “blob” of files where you have N inputs and M outputs,
> > > > > and
> > > >
> > > > thus
> > > >
> > > > > you get the N*M arcs in the graph.
> > > > >
> > > > > Currently the only similar but not really similar enough is the
> > > > > Dir()
> > > >
> > > > Node
> > > >
> > > > > type. But that has it’s own problems, which could be solved by a
> > > > > N*M
> > > >
> > > > type
> > > >
> > > > > node.
> > > > >
> > > > >
> > > > >
> > > > > Other Java issues could likely be resolved building on top of such a
> >
> > new
> >
> > > > > Node type.
> > > > >
> > > > > Though resolving the anonymous and inner classes in a java file
> >
> > creating
> >
> > > > > more than one class file and what it might be named is also still an
> > > >
> > > > issue
> > > >
> > > > > which the scanner and emitter try to solve by parsing the java files
> >
> > and
> >
> > > > > figuring out the proper naming. This of course is not (as I
> >
> > understand
> >
> > > > it)
> > > >
> > > > > formally defined as part of the java language and thus is a per
> >
> > compiler
> >
> > > > > implementation detail.
> > > >
> > > > In my opinion, you are wasting your time.
> > > >
> > > > The way java produces artifacts is so different from the model that
> >
> > SCons
> >
> > > > "expects" that you are attempting to cram an elephant down a shrew's
> > > > throat.
> > > > (or attempting to feed a Great White shark with ants, if you prefer)
> > > > to
> > > > get
> > > > this to work.
> > > >
> > > > SCons works *great* when there's one output per input, especially when
> >
> > you
> >
> > > > can
> > > > deduce the output name from the input name and little else. It also
> >
> > works
> >
> > > > great when it is fairly cheap to determine what files a given file
> >
> > depends
> >
> > > > upon.
> > > >
> > > > Sadly, neither of those conditions exist with java source.
> > > >
> > > >
> > > > Between 2003 and 2010, I developed and maintained a python based build
> > > > tool
> > > > for a Java project that contained ~10K java source files. It would
> > > > analyse the
> > > > java source files to figure out the package-level compile dependencies
> >
> > and
> >
> > > > then
> > > > send lists of files to be compiled as a unit to a persistent compile
> > > > server (in
> > > > fact, you could run multiple compile servers at once to get a large
> >
> > degree
> >
> > > > of
> > > > parallelism). It would then analyze the generated class files to see
> >
> > if
> >
> > > > the
> > > > total public/protected interface for the package changed, preventing
> > > > recompiles if all you did was change an implementation detail versus a
> > > > visible
> > > > method or attribute.
> > > >
> > > > You could also register various hooks against packages (such as run
> >
> > rmic)
> >
> > > > if
> > > > the package compiled correctly. We did some funky stuff with
> > > > autogenerating
> > > > WSDL from java source and then generate client classes from the WSDL
> >
> > and
> >
> > > > compiling those classes also. Almost all of that was data driven. It
> >
> > did
> >
> > > > other neat stuff that I don't remember off the top of my head, but
> > > > I've
> > > > got all
> > > > the source sitting around in various tla/hg repositories.
> > > >
> > > > Nobody uses it any more. Even me, and I thought about providing some
> >
> > it
> >
> > > > in an
> > > > open source build tool.
> > > >
> > > > For our Windows users, it was faster to nuke the output directories
> > > > and
> > > > recompile everything in one invocation of javac. (Well, 2 invocations
> > > > since
> > > > some of the funky stuff was to use the .class files from the first
> >
> > compile
> >
> > > > to
> > > > generate the wsdl and use *that* wsdl to generate java source as part
> >
> > of
> >
> > > > the
> > > > second bulk compile.) Most of the time difference had to do with the
> >
> > way
> >
> > > > the
> > > > python tool talked to the compile processes by sending the list of
> > > > file
> > > > names
> > > > through a pipe to the other process and then flushing the pipe. The
> >
> > rest
> >
> > > > had
> > > > to do with the time spent parsing the files (and I maintained a cache
> >
> > of
> >
> > > > parsed
> > > > file information that contained the last mtime of the source files and
> > > > would not
> > > > process a file whose mtime was the same as what was recorded).
> > > >
> > > > For our Linux users (me, for one), it was not quite a wash to change
> >
> > the
> >
> > > > behavior but the delta wasn't enough to worry about.
> > > >
> > > > I'm more than willing to discuss this and also to be proven wrong.
> > > >
> > > > --
> > > > Mark A. Flacy
> > > > _______________________________________________
> > > > Scons-dev mailing list
> > > > Scons-dev at scons.org
> > > > http://two.pairlist.net/mailman/listinfo/scons-dev
> >
> > --
> > Mark A. Flacy
> > _______________________________________________
> > Scons-dev mailing list
> > Scons-dev at scons.org
> > http://two.pairlist.net/mailman/listinfo/scons-dev
--
Mark A. Flacy
More information about the Scons-dev
mailing list