Hello,
I've been writing a Java program recently that uses
Reflection to call all the methods in all the classes that make
up another Java program, with random values for parameters.
It was lots of fun to write. There's cute things like a method
makeRandomObject( Class c ), which calls itself recursively when
it chooses a constructor for c that requires other objects as
parameters.
It is proving quite useful at the place I work, where it has
already discovered several bugs (the definition of a "bug" is
actually quite a difficult part of the problem, but I've pretty
much settled on some reasonable ways of determining Exceptions
that it is "ok" for methods to throw and ones that it is not).
I'm planning to release the code soon, most likely GPL'd, and
perhaps even try to get a write-up of it printed somewhere...
developerWorks or somewhere like that.
Anyway, to the point of my post. I am, believe it or not, a Lisp
hacker at heart. Actually I'm a more of a Scheme hacker who has
read some interesting things about how CL differs from Scheme.
I'd like to know more about CL though. I'd really like to rewrite
my program in CL, to test other CL programs, because from what
I've read of CL it would be very pretty!
But I know pretty much nothing about the "reflection"
capabilities of CL. I know it would be very hard to write my
program in Scheme, for several reasons. The biggest reason is,
although you can use apply in Scheme to invoke a function, there
is no (standard) way to look at a function's argument list. You
could try giving it random argument lists until one works, but
that would be slow and you'd need to deal with the fact that
error handling in Scheme is not standardized. In my Java program
I put every method and constructor call in its own thread in case
it hangs. In Scheme threads are not part of the language spec so
you'd have to choose one particular Scheme. I'm not going to
write it in Scheme.
But what would it be like, to write the program in CL? Do people
think I should try it? Sorry, I know these are fairly unfocused
questions, but feel free to give unfocused answers in return ;-)
Best wishes,
Bill.
From: Peter Seibel
Subject: Re: Java Reflection compared to Lisp? (quite long - sorry!)
Date:
Message-ID: <m3hdvvry34.fsf@javamonkey.com>
William Bland <····@abstractnonsense.com> writes:
> Hello,
> I've been writing a Java program recently that uses
> Reflection to call all the methods in all the classes that make
> up another Java program, with random values for parameters.
[snip]
> But what would it be like, to write the program in CL? Do people
> think I should try it? Sorry, I know these are fairly unfocused
> questions, but feel free to give unfocused answers in return ;-)
Well, it is unlikely that a literal translation of your Java program
into CL is going to be very successful. (Your program presumably
depends a fair bit on the static typing information that's available
via reflection--since Lisp is dynamically typed, that information is
not available.)
However depending what you're actually trying to do (I'm guessing some
sort of automated testing) there are probably more natural Lispy ways
to do it. In fact given the whole code-is-data-is-code thing in Lisp
and the ability to refer to symbols (i.e. names) as first class
objects, there are lots of options available to you in Common Lisp
that aren't available in Java.
Maybe if you give us some more details of what the purpose of this
program is, someone can point you in a more Lispy direction.
-Peter
--
Peter Seibel ·····@javamonkey.com
Lisp is the red pill. -- John Fraser, comp.lang.lisp
On Thu, 08 Apr 2004 02:40:53 +0000, Peter Seibel wrote:
> William Bland <····@abstractnonsense.com> writes:
>
>> Hello,
>> I've been writing a Java program recently that uses
>> Reflection to call all the methods in all the classes that make
>> up another Java program, with random values for parameters.
>
> [snip]
>
>> But what would it be like, to write the program in CL? Do people
>> think I should try it? Sorry, I know these are fairly unfocused
>> questions, but feel free to give unfocused answers in return ;-)
>
> Well, it is unlikely that a literal translation of your Java program
> into CL is going to be very successful. (Your program presumably
> depends a fair bit on the static typing information that's available
> via reflection--since Lisp is dynamically typed, that information is
> not available.)
>
> However depending what you're actually trying to do (I'm guessing some
> sort of automated testing) there are probably more natural Lispy ways
> to do it. In fact given the whole code-is-data-is-code thing in Lisp
> and the ability to refer to symbols (i.e. names) as first class
> objects, there are lots of options available to you in Common Lisp
> that aren't available in Java.
>
> Maybe if you give us some more details of what the purpose of this
> program is, someone can point you in a more Lispy direction.
>
> -Peter
Sorry, in an effort not to make my original post too long I guess
I left quite a bit of important stuff out. Yes, this is for
automated testing. It's been quite contentious among the people
I've shown my program to, and I'm still quite prepared to admit
that there are probably flaws with the approach, but I do have a
fair amount of empirical evidence now: The fact is, this thing
has caught real bugs, so I'm inclined to believe there is at
least some usefulness to it!
This all started when someone at work gave us a talk on the right
way to write unit tests in Java. My response was basically
"yeah, I suppose I should be more disciplined and write more tests
but they're whole lot of work. I'm going to see how much I can
do with the least amount of work possible". I thought this
wouldn't be very successful and I'd have to knuckle down and
write unit tests instead, but it's been amazingly good at finding
bugs so far.
Basically I'm treating the parameter list of a method as a kind
of promise that, if a method is passed arguments of the given
types, it will either throw an IllegalArgumentException, if any
of them are out of range, or it will attempt to do something
sensible with them. The heuristics I've come up with for what
"sensible" means are quite involved and have lots of special
cases. To give a simple example though, if a function f takes
a single integer as its argument, and we can feed some special
value into f and make it throw an ArithmeticException, e.g.
because of a division by zero, my program logs that as a bug.
I know CL is dynamically typed, but I've seen "declare". Is the
information provided to declare available at run-time? Is
declare actually used very much?
If not, I guess I have to just say each Lisp function takes
some number of objects of arbitrary type as its arguments. Can
I figure out how many, so that at least apply doesn't fail?
What's error handling like in Lisp? Is there a standard way to
say "the arguments you gave me were out of range"? Can I trap
things like (/ 1 0) so I can log them as bugs?
If I don't know anything about the intended types that a function
is expecting I'd guess that I'll be much less likely to tickle
its bugs. For example if a function f has a divide by zero when
called with 42, I not only have to be lucky enough to choose 42
from the universe of all integers, I have to be lucky enough to
choose 42 from the universe of all Lisp objects! Any idea what I
could do about that?
I know nothing at all about CLOS, except that it is CL's object
system... actually I just googled around a little and found some
stuff about generics. It seems to me that my approach could be
successfully applied to generics, since they have parameter type
information in their definitions (or at least the ones I saw did).
Can I retrieve that information at run-time?
By the way, the most troubling thing for the Java version has
been that I don't have a whole lot of control over methods that
I call. A method could decide to spawn hundreds of new threads,
each hammering the cpu and stopping the testing from moving on
to subsequent methods. There seems to be little that the tester
program can do about it because Java threads seem rather hard to
control (and in particular, destroy). What are CL threads like?
Thanks very much for your help.
Best wishes,
Bill.
From: Cameron MacKinnon
Subject: Re: Java Reflection compared to Lisp? (quite long - sorry!)
Date:
Message-ID: <kMCdnbrdF837Rend4p2dnA@golden.net>
William Bland wrote:
> I know CL is dynamically typed, but I've seen "declare". Is the
> information provided to declare available at run-time? Is
> declare actually used very much?
>
> If not, I guess I have to just say each Lisp function takes
> some number of objects of arbitrary type as its arguments. Can
> I figure out how many, so that at least apply doesn't fail?
The interfaces between Emacs and your Lisp (SLIME, e.g.) provide, after
you've typed in a function name, a one line summary showing the number
of arguments, named well enough for a *human* to infer the argument
types. So you could see how SLIME does it as a starting point.
> What's error handling like in Lisp? Is there a standard way to
> say "the arguments you gave me were out of range"? Can I trap
> things like (/ 1 0) so I can log them as bugs?
>
> If I don't know anything about the intended types that a function
> is expecting I'd guess that I'll be much less likely to tickle
> its bugs. For example if a function f has a divide by zero when
> called with 42, I not only have to be lucky enough to choose 42
> from the universe of all integers, I have to be lucky enough to
> choose 42 from the universe of all Lisp objects! Any idea what I
> could do about that?
Note: Automated theorem proving interests me, but I haven't studied it
much, yet.
http://lambda.weblogs.com/discuss/msgReader$2629 will give you an idea
of what ACL2 is about. It could, I'm sure, be adapted to grok a Lisp
function and say "I see that if the third argument is 42, you'll get a
division by zero error." You can even ask it why, and it will explain
its reasoning. It's the sledgehammer to your flyswatter approach (no
offense), but it works by proving, rather than randomly testing.
Might not even need too much adaptation, since it is written in Lisp!
--
Cameron MacKinnon
Toronto, Canada
On Thu, 08 Apr 2004 00:43:17 -0400, Cameron MacKinnon wrote:
> William Bland wrote:
>
> http://lambda.weblogs.com/discuss/msgReader$2629 will give you an idea
> of what ACL2 is about. It could, I'm sure, be adapted to grok a Lisp
> function and say "I see that if the third argument is 42, you'll get a
> division by zero error." You can even ask it why, and it will explain
> its reasoning. It's the sledgehammer to your flyswatter approach (no
> offense), but it works by proving, rather than randomly testing.
>
> Might not even need too much adaptation, since it is written in Lisp!
No offense taken at all - theirs does indeed look like a very
impressive sledgehammer. Mind you, I'm glad they've spent twenty
odd years on their sledgehammer whereas I'm only up to a couple of
weeks on my flyswatter ;-)
Cheers,
Bill.
From: Luke Gorrie
Subject: Re: Java Reflection compared to Lisp? (quite long - sorry!)
Date:
Message-ID: <lhfzbe370n.fsf@dodo.bluetail.com>
William Bland <····@abstractnonsense.com> writes:
> By the way, the most troubling thing for the Java version has
> been that I don't have a whole lot of control over methods that
> I call. A method could decide to spawn hundreds of new threads,
> each hammering the cpu and stopping the testing from moving on
> to subsequent methods.
And I hope your string-argument generator takes care to avoid things
like "-rf /" :-)
On a related note, are you familiar with QuickCheck?
http://www.cs.chalmers.se/~rjmh/QuickCheck/
-Luke
On Thu, 08 Apr 2004 15:59:20 +0200, Luke Gorrie wrote:
> William Bland <····@abstractnonsense.com> writes:
>
>> By the way, the most troubling thing for the Java version has
>> been that I don't have a whole lot of control over methods that
>> I call. A method could decide to spawn hundreds of new threads,
>> each hammering the cpu and stopping the testing from moving on
>> to subsequent methods.
>
> And I hope your string-argument generator takes care to avoid things
> like "-rf /" :-)
>
> On a related note, are you familiar with QuickCheck?
> http://www.cs.chalmers.se/~rjmh/QuickCheck/
>
> -Luke
Hehe. No, my string generator doesn't do stuff like that - I've
just been a lot luckier than I deserve to be... so far ;-)
Thanks for the QuickCheck link. I wasn't aware of that.
Very interesting stuff, but at the moment I'm finding it rather
depressing that everywhere I go someone else has been before
(and most likely done a better job than me). Ah well, that's
life I suppose :-/
Cheers,
Bill.
William Bland <····@abstractnonsense.com> writes:
> I know CL is dynamically typed, but I've seen "declare". Is the
> information provided to declare available at run-time? Is
> declare actually used very much?
No, except for code whose efficiency needs improving by giving the
compiler more information about the types. However, you could
restrict the tool to testing a subset that is similar to Java. In
particular, as you suggest later on:
> It seems to me that my approach could be successfully applied to
> generics, since they have parameter type information in their
> definitions. Can I retrieve that information at run-time?
Yes, using the Metaobject Protocol (MOP) you can extract and examine
the method signatures. That'll be portable but not standard.
> If not, I guess I have to just say each Lisp function takes
> some number of objects of arbitrary type as its arguments. Can
> I figure out how many, so that at least apply doesn't fail?
For non-generic functions, probably not. If the source code is
available from FUNCTION-LAMBDA-EXPRESSION, you can examine that.
> What's error handling like in Lisp? Is there a standard way to
> say "the arguments you gave me were out of range"? Can I trap
> things like (/ 1 0) so I can log them as bugs?
CL error handling is much more expressive than Java (documented in the
Conditions chapter of the standard
<http://www.lispworks.com/reference/HyperSpec/Body/09_.htm>. It's
easy to trap and log errors. Unfortunately, there doesn't happen to
be an error type for argument errors.
> [...] There seems to be little that the tester program can do about
> it because Java threads seem rather hard to control (and in
> particular, destroy). What are CL threads like?
Not standard, and not likely to offer specific protection against
run-away thread creation. Probably easier to destroy, e.g., LispWorks
simply has MP:PROCESS-KILL.
--
Pekka P. Pirinen
Quality control, n.:
Assuring that the quality of a product does not get out of hand
and add to the cost of its manufacture or design.
From: Thomas A. Russ
Subject: Re: Java Reflection compared to Lisp? (quite long - sorry!)
Date:
Message-ID: <ymiwu47r7mq.fsf@sevak.isi.edu>
William Bland <····@abstractnonsense.com> writes:
> I know CL is dynamically typed, but I've seen "declare". Is the
> information provided to declare available at run-time? Is
> declare actually used very much?
There is no standard way to access this information. It is generally
only for the compiler anyway, so even in the absence of a standard, I
don't know of any way of getting this information in any of the Lisp
systems I use.
This is a bit of a lacuna in the meta-level access protocols. In some
ways it is also like the lack of any way of querying the optimization
settings of the code. I've sometimes wanted to write macros of my own
that would be able to emit different code based on the SAFETY settings
during compilation, but no joy.
> If not, I guess I have to just say each Lisp function takes
> some number of objects of arbitrary type as its arguments. Can
> I figure out how many, so that at least apply doesn't fail?
The MOP (MetaObject Protocol) may help with methods, but it won't do
anything for ordinary functions. There are likely to be
implementation-specific ways to find out, but this would not necessarily
be portable.
The closest to a portable solution would be to use the
FUNCTION-LAMBDA-EXPRESSION accessor, but since implementations are not
required to save this information (and, for example, often don't have it
for built-in functions), it is not likely to be useful in practice.
>
> What's error handling like in Lisp? Is there a standard way to
> say "the arguments you gave me were out of range"? Can I trap
> things like (/ 1 0) so I can log them as bugs?
I suppose that if the code were written carefully with type checking,
then it should signal exceptions of type TYPE-ERROR. That is what the
CHECK-TYPE call will do. But there isn't, for example, a standard
index out of range error such as you might get from accessing an array
with too big an index.
> If I don't know anything about the intended types that a function
> is expecting I'd guess that I'll be much less likely to tickle
> its bugs. For example if a function f has a divide by zero when
> called with 42, I not only have to be lucky enough to choose 42
> from the universe of all integers, I have to be lucky enough to
> choose 42 from the universe of all Lisp objects! Any idea what I
> could do about that?
I don't think this approach will work in the absence of writing some
system that maintains more type information than is generally available
in Lisp programs. One would generally not actually expect Lisp
programmers to write code that checks for the type of input arguments.
Rather one would expect that the writer of the caller knows the general
type of argument.
If a function expects an integer as an argument, it generally won't have
a type check to make sure you don't pass a symbol instead. Lisp
programmers really do rely on run-time errors to detect that sort of
completely wrong argument type situatino.
> I know nothing at all about CLOS, except that it is CL's object
> system... actually I just googled around a little and found some
> stuff about generics. It seems to me that my approach could be
> successfully applied to generics, since they have parameter type
> information in their definitions (or at least the ones I saw did).
> Can I retrieve that information at run-time?
Actually, generic functions will normally only have types specified for
those arguments that they expect to use for type dispatch. One of the
interesting consequences of the Lisp model of extensible generic
functions is that one can later add distinctions that are more specific,
but that really only affect some of the methods. They may introduce
distinctions between items that other methods don't care about.
> ..... What are CL threads like?
They don't exist. At least not in the sense that there is any standard
governing threads. Many CL implementations have threads or processes,
but there is no standard interface or anything like Java's specification
of behavior.
--
Thomas A. Russ, USC/Information Sciences Institute
William Bland wrote:
> But I know pretty much nothing about the "reflection"
> capabilities of CL.
First of all, it's important to note that Java has changed the
terminology in significant ways when they incorporated the reflection
API. Usually, reflection encompasses the notions of introspection and
intercession. Introspection means to be able to analyze the structure of
a program, whereas intercession allows you to sneak under the hoods of
your language implementation and change the behaviors of essential
language constructs. So for example, in OOP terms, intercession allows
you to change the semantics of field lookup and method calls (not unlike
the before/after/around advices in CLOS and AspectJ).
Java's reflection API essentially only provides the introspection
capabilities, and only an extremely restricted set of intercession
capabilities. (-> dynamic proxy classes)
An excellent overview of the reflection facilities for CLOS (aka the
Metaobject Protocol) can be found in
http://www-db.stanford.edu/~paepcke/shared-documents/mopintro.ps
An excellent book is "The Art of the Metaobject Protocol" by Gregor
Kiczales, Jim des Rivieres and Daniel Bobrow. Read it!
The MOP specification can be found at http://www.lisp.org/mop/index.html
If you want to use that MOP in Scheme, it's worthwhile to note that it
has been ported under the "Tiny CLOS" moniker, and is available, for
example, as part of the Swindle library at http://www.barzilay.org/Swindle/
Pascal
--
1st European Lisp and Scheme Workshop
June 13 - Oslo, Norway - co-located with ECOOP 2004
http://www.cs.uni-bonn.de/~costanza/lisp-ecoop/
On Thu, 08 Apr 2004 14:37:19 +0200, Pascal Costanza wrote:
>
> William Bland wrote:
>
>> But I know pretty much nothing about the "reflection"
>> capabilities of CL.
>
[snip]
>
> Java's reflection API essentially only provides the introspection
> capabilities, and only an extremely restricted set of intercession
> capabilities. (-> dynamic proxy classes)
>
> An excellent overview of the reflection facilities for CLOS (aka the
> Metaobject Protocol) can be found in
> http://www-db.stanford.edu/~paepcke/shared-documents/mopintro.ps
>
Thanks very much for the information and links. I had assumed
that Java's reflection was a shadow of something better in the
Lisp world - as with so many things it seems. Once I have the
Java version of my tester finished and released (Real Soon Now)
I'm going to learn CLOS and try doing a Lisp version. Perhaps
more interesting than the article I was thinking of writing on
the Java version would be an article comparing both versions
and showing how much easier it is in Lisp?!
Thanks again.
Cheers,
Bill.