Hi, i want to connect lisp with java
basicly,
i got a var in java ,maybe int x
i got a funktion in lisp ,maybe (defun x2 (x) (* x
x))
i need the solution of the funktion in java again
What im asking for, is some simple solution of this problem as a
starting point
(I've read quite a lot of stuff 'bout it, but there seem to be so many
ways and i couldn't find one, thank u)
P� Tue, 07 Aug 2007 21:12:52 +0200, skrev <············@gmx.net>:
> Hi, i want to connect lisp with java
>
> basicly,
> i got a var in java ,maybe int x
> i got a funktion in lisp ,maybe (defun x2 (x) (* x
> x))
> i need the solution of the funktion in java again
>
> What im asking for, is some simple solution of this problem as a
> starting point
>
> (I've read quite a lot of stuff 'bout it, but there seem to be so many
> ways and i couldn't find one, thank u)
>
http://www.cliki.net/Java
Pascal Costanza, are you reading this? Back in 2002 you mentioned
that you were developing a JVM that worked (if I understand correctly)
by translating JVM bytecodes to CL and handing the result to the CL
compiler. It has long seemed to me that this would be a useful thing,
allowing Lisp and Java to call each other at function-call speed. Did
you get very far?
-- Scott
Scott Burson wrote:
> Pascal Costanza, are you reading this? Back in 2002 you mentioned
> that you were developing a JVM that worked (if I understand correctly)
> by translating JVM bytecodes to CL and handing the result to the CL
> compiler. It has long seemed to me that this would be a useful thing,
> allowing Lisp and Java to call each other at function-call speed. Did
> you get very far?
No, I have never finished that project due to time restrictions. The
basic JIT compiler worked, but you need to do a lot more to make a JVM work.
If anyone is interested in picking this up, they can have the source
code. But if I recall correctly, there is at least one other project
that does a similar thing and got further than mine.
Pascal
--
My website: http://p-cos.net
Common Lisp Document Repository: http://cdr.eurolisp.org
Closer to MOP & ContextL: http://common-lisp.net/project/closer/
On Aug 7, 11:15 pm, Pascal Costanza <····@p-cos.net> wrote:
> Scott Burson wrote:
> > Pascal Costanza, are you reading this? Back in 2002 you mentioned
> > that you were developing a JVM that worked (if I understand correctly)
> > by translating JVM bytecodes to CL and handing the result to the CL
> > compiler. It has long seemed to me that this would be a useful thing,
> > allowing Lisp and Java to call each other at function-call speed. Did
> > you get very far?
>
> No, I have never finished that project due to time restrictions. The
> basic JIT compiler worked, but you need to do a lot more to make a JVM work.
What would you say are the largest remaining tasks?
> If anyone is interested in picking this up, they can have the source
> code. But if I recall correctly, there is at least one other project
> that does a similar thing and got further than mine.
If it exists, I don't know how to find it.
I'm not sure I'll actually have time to work on this anytime soon, but
it would be interesting to see what you've done. Is the code
someplace I can download it?
-- Scott
Scott Burson wrote:
> On Aug 7, 11:15 pm, Pascal Costanza <····@p-cos.net> wrote:
>> Scott Burson wrote:
>>> Pascal Costanza, are you reading this? Back in 2002 you mentioned
>>> that you were developing a JVM that worked (if I understand correctly)
>>> by translating JVM bytecodes to CL and handing the result to the CL
>>> compiler. It has long seemed to me that this would be a useful thing,
>>> allowing Lisp and Java to call each other at function-call speed. Did
>>> you get very far?
>> No, I have never finished that project due to time restrictions. The
>> basic JIT compiler worked, but you need to do a lot more to make a JVM work.
>
> What would you say are the largest remaining tasks?
It's quicker to say what is already there: What I have built is a JIT
compiler that reads Java class files, transforms them into s-expressions
(basically CLOS classes and methods + invocations of functions and
macros that represent the various Java bytecodes), and then uses compile
to generate executable code on the fly. I needed to hack a little with
user-defined method combinations to get some corner cases of the Java
method calling semantics right. I would probably do the latter
differently now, probably going through the CLOS MOP, but the method
combination approach should be good enough.
This works to the degree that I have convinced myself that the general
approach is sound, and that I have been able to run a few example
programs (of the factorial and fibonacci kind, nothing fancy).
What is missing:
+ There is no support for native calls via Java JNI. This is absolutely
essential to get the Java core API working.
+ There is no support for Java's multithreading model, which has a
number of very specific requirements wrt memory management. It's
probably impossible to implement this portably. It may be possible to
get green threads running, but native threads are probably much harder.
+ I have taken some freedom wrt number representations: Java integers
are Lisp integers (but they should actually be a subrange of integers).
I also don't know how well Java's floats can be matched.
+ I haven't checked for compatibility wrt newer Java class file
specifications.
+ Except for a few toy examples, there is no test suite at all.
What is important is that this was a learning project for me to get to
know Common Lisp. It didn't serve any other purpose, so the quality
issues weren't really important.
I had played with the idea for some time to turn this into something
more usable. Especially, it might have been interesting to use this as a
basis for experimenting with Java language extensions. But this was a
left-over of my time as a Java user, where something like this would
have been very handy. I am now doing my research using Common Lisp and
Scheme, and there is no reason anymore to look back.
For practical purposes, i.e., if you want to connect to Java
applications or access Java libraries from Lisp programs, there are much
better and much more stable solutions available. Executing a Lisp-based
JVM is not really necessary for that, and I don't think it would have
any real advantages.
>> If anyone is interested in picking this up, they can have the source
>> code. But if I recall correctly, there is at least one other project
>> that does a similar thing and got further than mine.
>
> If it exists, I don't know how to find it.
>
> I'm not sure I'll actually have time to work on this anytime soon, but
> it would be interesting to see what you've done. Is the code
> someplace I can download it?
You can download the code at http://p-cos.net/lisp/Gina-JVM.zip with an
MIT/BSD-style license, which means that you can basically do whatever
you want with it.
Pascal
--
My website: http://p-cos.net
Common Lisp Document Repository: http://cdr.eurolisp.org
Closer to MOP & ContextL: http://common-lisp.net/project/closer/
On Aug 9, 11:22 am, Pascal Costanza <····@p-cos.net> wrote:
> Scott Burson wrote:
> > What would you say are the largest remaining tasks?
>
> It's quicker to say what is already there: What I have built is a JIT
> compiler that reads Java class files, transforms them into s-expressions
> (basically CLOS classes and methods + invocations of functions and
> macros that represent the various Java bytecodes), and then uses compile
> to generate executable code on the fly. I needed to hack a little with
> user-defined method combinations to get some corner cases of the Java
> method calling semantics right. I would probably do the latter
> differently now, probably going through the CLOS MOP, but the method
> combination approach should be good enough.
>
> This works to the degree that I have convinced myself that the general
> approach is sound, and that I have been able to run a few example
> programs (of the factorial and fibonacci kind, nothing fancy).
>
Okay, that seems like a good start, and might even suffice for some of
the uses I have in mind (calling Java libraries from Lisp, where the
libraries just do computation -- no threading, I/O, etc.).
> What is missing:
>
> + There is no support for native calls via Java JNI. This is absolutely
> essential to get the Java core API working.
Hmm, I was hoping to put that off a bit, but you're probably right.
> + There is no support for Java's multithreading model, which has a
> number of very specific requirements wrt memory management. It's
> probably impossible to implement this portably. It may be possible to
> get green threads running, but native threads are probably much harder.
If you're talking about the write ordering requirements, my impression
is that those are not difficult for relatively unsophisticated
implementations to meet. It's only highly optimizing compilers (that
want to do things like reordering writes) that need to watch out for
them. I don't know if there even exist any Lisp compilers that
attempt the kinds of optimizations that can cause problems. But, I
haven't studied this issue closely and I may very well be naive.
> + I have taken some freedom wrt number representations: Java integers
> are Lisp integers (but they should actually be a subrange of integers).
My experience back in the 1980s translating C into Lisp suggests that
one can get away with this in almost all cases. I recall only one
line of one program, out of a couple million lines translated, that
actually depended on 32-bit integer overflow.
For this JVM implementation to be product-quality, there would need to
be a way to specify that some particular `int' operation, or perhaps
all of those in some method or class or package, should be limited to
32 bits. But I don't know that this is required for it to handle most
real-world code.
> I also don't know how well Java's floats can be matched.
My guess is, probably not well enough for the most demanding numeric
code, but probably adequately for everything else.
> + I haven't checked for compatibility wrt newer Java class file
> specifications.
>
> + Except for a few toy examples, there is no test suite at all.
Okay, those two don't scare me.
> For practical purposes, i.e., if you want to connect to Java
> applications or access Java libraries from Lisp programs, there are much
> better and much more stable solutions available. Executing a Lisp-based
> JVM is not really necessary for that, and I don't think it would have
> any real advantages.
But as far as I can see, the existing options (JACOL, Allegro JLinker)
are RPC-based. That can suffice for some purposes, but doesn't really
let you mix Java and Lisp code freely. Still, you have a point -- I
need to think carefully whether I really need to do that.
> You can download the code at http://p-cos.net/lisp/Gina-JVM.zip with an
> MIT/BSD-style license, which means that you can basically do whatever
> you want with it.
Merci beaucoup!
-- Scott
Scott Burson wrote:
>> + I have taken some freedom wrt number representations: Java integers
>> are Lisp integers (but they should actually be a subrange of integers).
>
> My experience back in the 1980s translating C into Lisp suggests that
> one can get away with this in almost all cases. I recall only one
> line of one program, out of a couple million lines translated, that
> actually depended on 32-bit integer overflow.
>
> For this JVM implementation to be product-quality, there would need to
> be a way to specify that some particular `int' operation, or perhaps
> all of those in some method or class or package, should be limited to
> 32 bits. But I don't know that this is required for it to handle most
> real-world code.
This is the reason why I didn't worry that much in the beginning.
However, since we are at the bytecode level, and not at the language
level, things may be different.
>> For practical purposes, i.e., if you want to connect to Java
>> applications or access Java libraries from Lisp programs, there are much
>> better and much more stable solutions available. Executing a Lisp-based
>> JVM is not really necessary for that, and I don't think it would have
>> any real advantages.
>
> But as far as I can see, the existing options (JACOL, Allegro JLinker)
> are RPC-based. That can suffice for some purposes, but doesn't really
> let you mix Java and Lisp code freely. Still, you have a point -- I
> need to think carefully whether I really need to do that.
See also http://bc.tech.coop/blog/040930.html
Pascal
--
My website: http://p-cos.net
Common Lisp Document Repository: http://cdr.eurolisp.org
Closer to MOP & ContextL: http://common-lisp.net/project/closer/
On 2007-08-09, Pascal Costanza <··@p-cos.net> wrote:
> For practical purposes, i.e., if you want to connect to Java
> applications or access Java libraries from Lisp programs, there are much
> better and much more stable solutions available. Executing a Lisp-based
> JVM is not really necessary for that, and I don't think it would have
> any real advantages.
Yes, exactly.
But it would be interesting to have a solution that can dump
pre-compiled classes into a lisp image, possibly including the run-time
start of a VM that has already started executing. Such an approach
could save a lot of time on startup. It is really sad to watch a Lisp
server come up in a fraction of a second just to wait for a JVM to start
in the background.
Unfortunately, the code my trivial VM generates has always been much too
large to make it feasible to dump all important java packages into the
lisp core, making dynamic loading of the precompiled code a necessity.
On Aug 9, 2:51 pm, David Lichteblau <···········@lichteblau.com>
wrote:
> On 2007-08-09, Pascal Costanza <····@p-cos.net> wrote:
>
> > For practical purposes, i.e., if you want to connect to Java
> > applications or access Java libraries from Lisp programs, there are much
> > better and much more stable solutions available. Executing a Lisp-based
> > JVM is not really necessary for that, and I don't think it would have
> > any real advantages.
>
> Yes, exactly.
>
> But it would be interesting to have a solution that can dump
> pre-compiled classes into a lisp image, possibly including the run-time
> start of a VM that has already started executing. Such an approach
> could save a lot of time on startup. It is really sad to watch a Lisp
> server come up in a fraction of a second just to wait for a JVM to start
> in the background.
>
> Unfortunately, the code my trivial VM generates has always been much too
> large to make it feasible to dump all important java packages into the
> lisp core, making dynamic loading of the precompiled code a necessity.
Ah, so you are the author of the other JVM that Pascal mentioned.
Okay, I've poked around a little and discovered that your package is
called CLOAK and that it is complete enough to run Eclipse. If that's
true, I think I have an application for it.
Well, it's possible, as Pascal suggests, that my intended application
will work well enough with JACOL. I should look into this.
Nonetheless I would be interested to try CLOAK if you were willing to
make it available.
-- Scott
On 2007-08-09, Scott Burson <········@gmail.com> wrote:
> Ah, so you are the author of the other JVM that Pascal mentioned.
> Okay, I've poked around a little and discovered that your package is
> called CLOAK and that it is complete enough to run Eclipse. If that's
> true, I think I have an application for it.
Not good enough to actually -run- Eclipse.
It can -start- Eclipse. Sometimes.
Some threading issues probably that require someone to think hard and
fix my broken VMThread implementation.
The rest of the issues would be more deterministic and relatively easy
to fix one-by-one for someone who knows Java and Lisp. There are also
still Mauve tests failing, so you could work your way through those.
David Lichteblau wrote:
> On 2007-08-09, Pascal Costanza <··@p-cos.net> wrote:
>> For practical purposes, i.e., if you want to connect to Java
>> applications or access Java libraries from Lisp programs, there are much
>> better and much more stable solutions available. Executing a Lisp-based
>> JVM is not really necessary for that, and I don't think it would have
>> any real advantages.
>
> Yes, exactly.
>
> But it would be interesting to have a solution that can dump
> pre-compiled classes into a lisp image, possibly including the run-time
> start of a VM that has already started executing. Such an approach
> could save a lot of time on startup. It is really sad to watch a Lisp
> server come up in a fraction of a second just to wait for a JVM to start
> in the background.
Hm, aren't there solutions in the Java world for having a JVM be present
all the time, such that an invocation of the java command doesn't take
so much time? I recall Apple having such a solution for its JVM and Sun
incorporating similar features into their JVM later on. But I don't
remember the details...
Pascal
--
My website: http://p-cos.net
Common Lisp Document Repository: http://cdr.eurolisp.org
Closer to MOP & ContextL: http://common-lisp.net/project/closer/
On 2007-08-08, Scott Burson <········@gmail.com> wrote:
>> If anyone is interested in picking this up, they can have the source
>> code. But if I recall correctly, there is at least one other project
>> that does a similar thing and got further than mine.
>
> If it exists, I don't know how to find it.
That is because I never released it.
> I'm not sure I'll actually have time to work on this anytime soon, but
> it would be interesting to see what you've done. Is the code
> someplace I can download it?
Beware, this is source code that I never thought I would release before
rewriting it completely.
Although the VM itself is tiny (less that 9kLOC), you will need matching
versions of GNU Classpath and SBCL, so grab the monster tarball
including full source code and binaries for Classpath, SBCL, binaries of
Eclipse (just for fun, try any other application that you like instead),
and enough precompiled heapfiles to make Eclipse start up in minutes
rather than hours. Linux/x86 is required (because the JNI
implementation is not 64 bit clean or something):
http://common-lisp.net/~dlichteblau/sbcl+classpath+cloak+eclipse.tar.bz2
- Run ./test.sh in that directory to start.
- Wait and watch Eclipse start, then click somewhere and watch it
crash. :-)
- That's if it works. Reports on #lisp indicate that a full startup
happens less often than random crashes during startup.
Change test.lisp to run something other than eclipse. For example,
(cloak::test) runs Hello World. (At least that starts up reliably. :-))
Keep in mind that CLOAK is really unfinished. It does not need users,
it needs hackers, and in particular people willing to rewrite large
parts of it. Here is a list of things that might be fun to do:
* update to current SBCL
* update to current Classpath
* (tedious but not hard) Look for race conditions and other bugs
Eclipse usually starts (but not always, so there are probably
threading-related bugs). Using it crashes soon.
(Of course, Eclipse is a test case chosen for coolness, not
practicality. Pick any other application you like instead.)
Fix these problems as you find them.
* (lots of work) Switch from GNU Classpath to OpenJDK/IcedTea
Ultimately having OpenJDK support is the right thing to do.
* Validation (Java 2-style)
Validation sounds hard if you believe the JVM spec (so ignore it
completely). Read Alessandro Coglio instead:
http://www.kestrel.edu/home/people/coglio/ftjp02.ps
http://www.kestrel.edu/home/people/coglio/ccpe03.pdf
* Validation (Java 6-style)
Trivial.
* Java <-> Lisp Interface
Invent a nice (!) and fast interface for calls from Lisp to Java and back.
Bonus points if you implement the same interface on ABCL.
* JNI/libsbcl
Make SBCL loadable as a shared library to implement the missing JNI
functions that a VM in a previously existing process.
Also implement the AttachThreadSomething functions that allow JNI code to
turn non-Java pthreads into Java threads.
* Port to other platforms
Linux/x86-only right now. Linux/amd64 should be easy.
(Other threading-enabled Unix on x86 might just work.)
* (relatively easy) Get rid of spinlocking on monitor contention.
Currently, cloak uses Bacon's algorithm, which busy waits in some
cases: http://citeseer.ist.psu.edu/283867.html
Implement at least Onodera's algorithm instead:
http://portal.acm.org/citation.cfm?id=320405&coll=portal&dl=ACM
* Fix stack trace disaster
There is an extremely terrible mechanism to translate PCs from
backtraces to their methods, which involves a huge table for all
methods and a GC hook.
Instead, implement Juho Snellman's suggestion of an SBCL extension
that would provide a memory array for long-term pinning of objects,
allowing us to pin all methods referenced from stack traces until user
code asks for them to be resolved. (Keep a table of weak references
to exceptions.)
* (hard to get right) Implement OutOfMemoryError
If you actually want the Lisp and the VM to recover from rigorous
testing of out-of-memory situations, this is tricky.
* (easy) weak references
Weak references are not implemnted yet (IIRC).
The trick could be to hack the compiler so that it recognizes accesses
to the slot of a weak reference and hides a Lisp weak reference
between the slot and its contents.
* finalization
- Find my patch for Java-style finalization in SBCL and use it to
implement finalize().
- Read Bruno Haible's talk to find out what's wrong with my patch.
- Make sure not to finalize objects with a trivial finalize() method.
- Implement PhantomReference and SoftReference.
* Interpreter
A bytecode interpreter would be nice to avoid having to run all
classes through the SBCL compiler.
While a simple and slow interpreter could be written in an afternoon,
we would want an interpreter that is much more optimized.
Have a look how a simple interpreter like JamVM does it, then try to
figure out how to do the same in Lisp. (It is probably harder in Lisp
than in C because you will have endless fun with type declarations and
compiler notes.)
(Alternatively, make the interpreter call specific VOPs instead of
fighting with Lisp types, which might be easy if the compiler has
already been rewritten to do so; see below.)
* Rewrite the "compiler"
The translation from bytecode to Lisp source code is extremely naive.
- (easy) Introduce a BASIC-BLOCK class.
- Compile into IR1 or IR2 instead of going through source code.
o Control exactly which VOPs are used.
o Make compilation fast somehow.
(This amounts to a complete rewrite, but as everyone knows, it is
way easier to rewrite code than write it from scratch, so it might
be easy enough if you know SBCL.)
- (easy) Review binary compatibility (I): Classes are compiled into Lisp
functions and dumped to disk. Right now, we try to generate code
that is independent of other classes, so that each class can be
loaded back from disk even if other classes have changed [in a
binary-compatible way].
- (easy) Review binary compatibility (II): If classes have changed in a way
that breaks binary compatibility, we are required to throw certain
exceptions. Make sure that we actually do that.
- (not too hard and probably fun) Break binary compatibility for
optimization. The generated code can hardly do any optimizations at
all without taking into account the class hierarchy.
Add a mode that optimizes all classes (or perhaps certain jar files)
together:
o All method invocations currently go through a double-dispatch
scheme. In optimized mode, the vtable index could be computed
at compilation time.
o Ditto for slots.
o Inlining of methods!
(Careful, Java needs accurate stack traces, so the effects of
inlining must be recorded exactly so that instructions can be
mapped back to the method they came from at run time.)
o Eliminate unneeded class casts based on type informations
resulting from inlining.
o Eliminate unneeded class initialization checks (for example,
superclasses will already have been initialized when the current
class runs.)
o The big catch: Figure out how to make this scheme safe.
Heapfiles might include cryptographic check sums of all classes
they depend on.
But (hard): Make sure to either keep incremental class loading
working or make our code much more compact so that it's actually
feasible to map entire pre-compiled applications into memory in one
step.
- Either: Propagate information about whether variables can be `null' and
eliminate redundant null checks accordingly. (Ditto for class
initialization.)
- Alternatively, restore the logic that lets null pointer accesses
just happen, and signal the NullPointerException in the segfault
handler. There is a *feature* that might even still work if you set
it. It involves lying to the compiler though and makes things
harder to debug. Worse, it didn't make anything faster back then.
- Hack SBCL to allow untagged arguments across full functions calls.
We do lots of untagged operations, resulting in significant overhead
when the numbers involved are passed to other functions. (And
Java's use of 32 bit and 64 bit arithmetic really makes it
completely unattractive to do the calculations with tagged numbers).
This change could be in the form of a "Java calling convention" that
ordinary Lisp function calls don't use.
- Rewrite parser.lisp, which is currently rather slow and conses too much.
* (hard) Generated much smaller native code.
* (easy) Store the precompiled .heap files in a zip file to save disk space.
Requires loading them into dynamic space using memcpy() instead of mmap().
Should save lots of space.
On Aug 10, 4:36 am, David Lichteblau <···········@lichteblau.com>
wrote:
> Beware, this is source code that I never thought I would release before
> rewriting it completely.
Understood.
Thanks for putting this up for download, and for the excellent to-do
list.
Let me explain what my priorities would be for this project.
Correctly modelling all the little nooks and crannies of JVM behavior,
so that you can download an arbitrary class file off the net and have
it just run -- and so this thing could pass Sun's compatibility test
-- is not very important to me. Sure, it would be nice, but it
doesn't bother me if some porting effort is involved to get any
particular piece of Java code to run on this thing. (If that sounds
implausible, consider the effort required for hand translation.)
So when you talk about stuff like accurate stack backtraces, I have to
wonder how much real code really needs that. Of course, you might
have bad news for me :) but I suspect I could live without this in
most cases. And in the few cases where it does matter, I imagine it
might work to do something simple like maintain a fake stack in a
special variable -- but only along the call paths where it might
actually be accessed. I can even live with it requiring a bit of
manual effort to annotate said call paths (though ultimately, of
course, it would be better for this to be done automatically).
Back in 1984-85 I wrote a C compiler for Lisp Machines. One of my
customers ran it through a C test suite and it was just terrible; it
got all kinds of things wrong. But another customer loved it and said
he had run 1M lines of real-world code through it successfully. I had
evidently figured out, mostly through trial and error, which aspects
of C semantics were the most important to model accurately. There
were still occasional bits of code that didn't run on it unmodified,
but these were rare enough to make the thing useful.
To a first approximation, embedding Java in Lisp ought to be much
easier than embedding C in Lisp (!). The languages are much closer
semantically. So I think there is hope for making something like
CLOAK
useful even if it isn't perfect. Also, I'm more interested in
translating libraries than programs. Indeed, I think this thing could
be of _some_ use even with no threading support at all.
On the other hand, if it can't be made to have halfway decent
performance, then there's no point. Ideally I'd like to get pretty
close to the performance you could get by hand-translating Java to
Lisp; if I can't get even half that for some reason, then this
approach becomes less interesting.
So, where I would be inclined to start would be on the quality of the
generated code. It sounds like there's substantial room for
improvement in that area. (Besides, this is the area I would enjoy
working on the most.)
Anyway, thanks again. I don't think I'll have time to look at it this
weekend, but maybe next weekend.
-- Scott
On 2007-08-11, Scott Burson <········@gmail.com> wrote:
> Let me explain what my priorities would be for this project.
> Correctly modelling all the little nooks and crannies of JVM behavior,
> so that you can download an arbitrary class file off the net and have
> it just run -- and so this thing could pass Sun's compatibility test
> -- is not very important to me. Sure, it would be nice, but it
> doesn't bother me if some porting effort is involved to get any
> particular piece of Java code to run on this thing. (If that sounds
> implausible, consider the effort required for hand translation.)
It used to pass many (but not all) tests in Classpath's Mauve test
suite. Working based on Sun's tests, now that they are available,
should be a nice way of finding out how much really works.
> So when you talk about stuff like accurate stack backtraces, I have to
> wonder how much real code really needs that. Of course, you might
> have bad news for me :) but I suspect I could live without this in
> most cases.
It is essential for security manager purposes to determine the stack of
_classes_ that the current method was called by.
The actual _methods_ in the stack trace are only needed for Exceptions.
And you are quite right, real applications come with bad news here.
To take Eclipse again as an example (and it's really silly that everyone
tests their VM using Eclipse before it is actually finished, but it
certainly is a good test case for lots of weird issues), they actually
ask for stack traces just so that they can do a string comparison on a
stack trace element to figure out where the current method was called
from.
> And in the few cases where it does matter, I imagine it
> might work to do something simple like maintain a fake stack in a
> special variable -- but only along the call paths where it might
> actually be accessed. I can even live with it requiring a bit of
> manual effort to annotate said call paths (though ultimately, of
> course, it would be better for this to be done automatically).
I understand and appreciate your philosophy regarding this.
But since you chose stack traces as an example, please let me expand on
how they work -- it will be just as easy to get rid of the GC hook
instead and do this properly.
This is how stack traces work in CLOAK:
- The first step any Java method takes is to push its class object onto
the stack.
- When an exception is thrown (see the fillInStackTrace method in
cloak/native/VMThrowable.lisp), we walk the stack using SBCL's
X86-CALL-CONTEXT and record two values for each stack frame: The
program counter (let's call it PC) and the first pointer in the
stack frame (which will be the class if the frame belongs to a Java
method; let's call it CLASS).
(This is the step that needs to be fast. Applications like Eclipse
throw thousands of harmless exceptions on startup.)
- When GC has happened (or getStackTrace is called, whichever happens
first) we need to resolve those values into objects, so we take
CLASS and look it up in a hash table that records all classes. (If
there is no entry, it's definitely not a Java stack frame, so we
skip it.) Then we look at each method of that class, and see
whether the recorded PC matches that method's position [actually,
that method's recorded position from -before- the previous GC, which
is the unpleasant part]. If no method was found, it's not a Java
stack frame. If a method was found, we definitely know that class
and method are valid, because Java methods are never on the stack
accidentally.
So that actually works, it would just be nicer to get rid of the big
table of classes and the table of old code component positions.
[...]
> To a first approximation, embedding Java in Lisp ought to be much
> easier than embedding C in Lisp (!). The languages are much closer
> semantically. So I think there is hope for making something like
> CLOAK
> useful even if it isn't perfect. Also, I'm more interested in
> translating libraries than programs. Indeed, I think this thing could
> be of _some_ use even with no threading support at all.
CLOAK might be just fine if you want to run simple code. (Say, to run a
JDBC type 4 driver in the Lisp instead of messing with FFI and the
database's C libraries.)
> On the other hand, if it can't be made to have halfway decent
> performance, then there's no point. Ideally I'd like to get pretty
> close to the performance you could get by hand-translating Java to
> Lisp; if I can't get even half that for some reason, then this
> approach becomes less interesting.
Right now, I am much more worried by the extreme compilation times than
the run time issues, because they make it inconvenient to recompile the
entire test suite after changing something that affects code generation.
(For normal applications, it might not be that bad as long as you
recompile only the classes that actually have changed. But when testing
a compiler change, recompilation of every test is the whole point of
running the test suite again.)
People have already asked me whether CLOAK is fast, and were surprised
that it isn't. (Basically saying: "What? So why did you write it then?")
My answer is that CLOAK is an experiment. I would like to see how fast
a JVM can be that is implemented on top of SBCL. Right now, it is not
optimized and not fast enough. But the experiment isn't over.)
(I just don't have the time to work on it -- it's been a long time since
I touched this code the last time.)
> So, where I would be inclined to start would be on the quality of the
> generated code. It sounds like there's substantial room for
> improvement in that area. (Besides, this is the area I would enjoy
> working on the most.)
That would be very cool.
There are some toy benchmarks that can be started using (cloak::times).
(And using "java microbench.Main" for comparison with a normal VM.)
They are meant to test individual instructions. Of course, real world
performance cannot be captured by this kind of naive code, but the
microbenchmarks are still useful to catch the kind of regression where,
say, arithmetic is not done inline anymore.
d.
On Aug 11, 4:12 am, David Lichteblau <···········@lichteblau.com>
wrote:
> On 2007-08-11, Scott Burson <········@gmail.com> wrote:
>
> It used to pass many (but not all) tests in Classpath's Mauve test
> suite. Working based on Sun's tests, now that they are available,
> should be a nice way of finding out how much really works.
Yes, okay.
> > So when you talk about stuff like accurate stack backtraces, I have to
> > wonder how much real code really needs that. Of course, you might
> > have bad news for me :) but I suspect I could live without this in
> > most cases.
>
> It is essential for security manager purposes to determine the stack of
> _classes_ that the current method was called by.
I don't even know that I care about the security manager.
The thing I'm most interested in running is <a http="http://
www.cs.waikato.ac.nz/~ml/weka/">Weka</a> (I think; I have to admit I
haven't looked at it closely yet). It's a data mining library; it has
no reason (at least, none I can imagine) to be doing weird shit. It's
just doing computation. (Uses a lot of floating point, of course, but
probably not in such a way as to depend on the fine details of the FP
implementation.)
On a different track, yes it would be nice to be able to use this to
develop Swing apps, but it's not strictly necessary and JACOL might
work well enough for that anyway.
> To take Eclipse again as an example (and it's really silly that everyone
> tests their VM using Eclipse before it is actually finished, but it
> certainly is a good test case for lots of weird issues), they actually
> ask for stack traces just so that they can do a string comparison on a
> stack trace element to figure out where the current method was called
> from.
Wild. You know, I've never been that crazy about special variables,
but if people who have to live without them wind up doing weird shit
like that to figure out their dynamic context, I have to say a special
would have been a hell of a lot cleaner.
> But since you chose stack traces as an example, please let me expand on
> how they work -- it will be just as easy to get rid of the GC hook
> instead and do this properly.
Okay, thanks for the explanation.
> So that actually works, it would just be nicer to get rid of the big
> table of classes and the table of old code component positions.
Hey, if it actually works, leave it alone for now :)
> CLOAK might be just fine if you want to run simple code. (Say, to run a
> JDBC type 4 driver in the Lisp instead of messing with FFI and the
> database's C libraries.)
That's certainly the kind of thing I have in mind, yes. I know
someone who might like to run a CORBA interface this way, for another
example.
> Right now, I am much more worried by the extreme compilation times than
> the run time issues
I note that the SBCL compiler has gotten much faster on large routines
since about 0.9.16. There are a couple of large routines in my FSet
package that took minutes to compile in 0.9.16, that go right by in
recent releases. Maybe that will cure this problem?
> (I just don't have the time to work on it -- it's been a long time since
> I touched this code the last time.)
Yeah, I know how that goes :)
> > So, where I would be inclined to start would be on the quality of the
> > generated code. It sounds like there's substantial room for
> > improvement in that area. (Besides, this is the area I would enjoy
> > working on the most.)
>
> That would be very cool.
I can't promise I'll get to it soon, but whenever I do, it will be
great not to be starting from scratch.
If anyone else reading this wants to help out, speak up!
-- Scott
On 2007-08-11, Scott Burson <········@gmail.com> wrote:
> The thing I'm most interested in running is <a http="http://
> www.cs.waikato.ac.nz/~ml/weka/">Weka</a> (I think; I have to admit I
> haven't looked at it closely yet). It's a data mining library; it has
> no reason (at least, none I can imagine) to be doing weird shit. It's
> just doing computation. (Uses a lot of floating point, of course, but
> probably not in such a way as to depend on the fine details of the FP
> implementation.)
>
> On a different track, yes it would be nice to be able to use this to
> develop Swing apps, but it's not strictly necessary and JACOL might
> work well enough for that anyway.
Okay, to start Weka's GUI I needed the following patch relative to the
tarball I posted earlier, fixing a race condition and a reflection bug:
http://www.lichteblau.com/tmp/cloak-weka.diff
To pre-compile Classpath and Weka, run these forms (the second step
takes half an hour on my computer):
(cloak::precompile-zip-file "classpath/lib/glibj.zip")
(cloak::precompile-zip-file "weka-3-4-11/weka.jar")
Then start the jar file using:
(cloak::start-vm nil :jar "weka-3-4-11/weka.jar")
At least the "Explorer" button works, and opens another window. Since I
don't know Weka, I stopped trying at that point.
On Aug 12, 6:24 am, Scott Burson <········@gmail.com> wrote:
>
> The thing I'm most interested in running is <a http="http://www.cs.waikato.ac.nz/~ml/weka/">Weka</a> (I think; I have to admit I
> haven't looked at it closely yet). It's a data mining library; it has
Have you seen this?
https://list.scms.waikato.ac.nz/pipermail/wekalist/2006-September/thread.html#7892
Look at the first thread, ie
https://list.scms.waikato.ac.nz/pipermail/wekalist/2006-September/007890.html
I never got around to using Weka from ABCL, so no comments from me
here, but maybe
it's a viable alternative. The advantage is that you'll be running on
the "real"
JVM. If you go down that path, do provide feedback in case I need to
do it one day.
On Aug 12, 9:23 pm, szergling <···············@gmail.com> wrote:
> I never got around to using Weka from ABCL, so no comments from me
> here, but maybe
> it's a viable alternative.
Hmm, the problem, I expect, is that I don't just want to use CL to
drive Weka -- I also want to do some heavy computation in CL (most of
it, in fact). ABCL doesn't look like it's going to be a good choice
for this.
Thanks for the pointer, though.
-- Scott
On Tue, 7 Aug 2007 ············@gmx.net wrote:
> Hi, i want to connect lisp with java
>
> basicly,
> i got a var in java ,maybe int x
> i got a funktion in lisp ,maybe (defun x2 (x) (* x
> x))
> i need the solution of the funktion in java again
>
> What im asking for, is some simple solution of this problem as a
> starting point
Maybe one of the Lisp implementations for the JVM does the trick for you:
http://www.robert-tolksdorf.de/vmlanguages.html#lispandco
I've tried SISC (it's scheme, not lisp, though), and it's very easy to
integrate Java and Scheme code with it.
- Ville Oikarinen