I looked (*) for benchmarking results for recent versions of Lisp and
Scheme implementations on modern machines and did not find any.
Has anybody done informal or formal (**) speed or space tests for any
two of the following Lisps, preferably under Solaris (***)?
emacs 19.2x
xlisp-plus with no windows
clisp
allegro 4.2
scheme48 or scsh
scm or guile with slib
stk with no windows
elk with no windows
vscm
Please reply by email if you can. If I get useful results, I will
summarize in tabular form to comp.lang.lisp. I am surprised that I
have not found a web page dedicated to Lisp performance results.
Please note that I am excluding several Lisps (****) and that I am
after effects that are greater than a factor of 2, such as might be
produced by image dumping or compilation. I don't care about
differences in speed less than that, since other factors (in my case,
for the specific purpose of a scripting language (*****), CL
compatibility and the ability to parse with regexps) are more
important.
(*) Various web and ftp sites, comp.benchmarks, and in the Lisp- and
benchmarking-related FAQs.
(**) I would definitely be satisfied with times for the Gabriel Lisp
benchmarks (referenced in the MK FAQ) with as many variables as
possible controlled.
(***) Any other OS would be fine.
(****) I know about bigloo, cmucl, gcl, kcl, akcl, lucid, ilog talk,
jlisp, ylisp, siod, wcl, and harlequin. Please know that they might
all be very good implementations or languages despite my not including
them above. Allegro is included because I use it for other purposes
and therefore can use it as a point of reference if others comment.
Please know that I know about non-Lisp alternatives and reject them
for my purposes. De gustibus.
(*****) The reader might have seen my post on the subject some time
ago. Since then, scsh and guile have come out and I discovered
xlisp-plus. None of those were reviewed in my post. xlisp-plus
interests me the most of these three, despite the features of scsh and
guile, because it attempts slightly more to be compatible with CL.
But speed factors greater than 2x and lack of regexps might veto it.
I am not enough of a Schemer to benchmark Schemes, so I hope that
others will be able to comment. I hope for a CLer response also.
---
···@ptolemy.arc.nasa.gov. AI, multidisciplinary neuroethology, info filtering.
At one time (last May?) I tried to collect a useful benchmark suite
for Common Lisp. I found that most `real-world' programs (as compared
to benchmark-only programs in the gabriel benches) are not portable
enough between CLs, at least not when highly optimized. Not to speak
of a CL <-> Scheme comparison.
The only useful benchmark suite I found myself to be able to collect
would be a collection of Screamer code. Screamer is ported and
optimized on a wide range of platforms and Screamer applications are
not implementation-specific anymore, while they still show different
strengths of given architectures. But this is not a useful benchmark
to predict performance of random CL programs. I could dig up my old
work if you really want such a Screamer-based suite.
To go some deeper, I found that performance is much more
application-dependent for Lisp programs as it is for other languages.
Example: While gcl generally reaches about 1/2 of the speed as my
native compilers (Lispworks and CMU) for my `real work', it needs
about 100 times the runtime for a isolated CLOS-message-call benchmark
I did for inter-language comparison. This is probably because CMU and
Lispworks recognize message calls and implement them in machine
language, while simple PCL implementations have Lisp code running
(don't really know). I bet that gcl can be tuned to be at least 10
times faster by a few days hacking. In a word: Whatever you benchmark,
it is not likely to reflect the quality of the overall implementation.
I have been disappointed by overall CL performance anyway and don't use
Lisp variants for performance-oriented applications anymore. Maybe
that is another reason why I lost interest in collecting a benchmark
suite, I don't need a fast Lisp anymore.
k p c <···@ptolemy.arc.nasa.gov> writes:
>I looked (*) for benchmarking results for recent versions of Lisp and
>Scheme implementations on modern machines and did not find any.
>Has anybody done informal or formal (**) speed or space tests for any
>two of the following Lisps, preferably under Solaris (***)?
In general, I think the big machine-code-generating CL compilers
(Lucid, CMU, Allegro, Lispworks) can blow other implementations away
if you really go into optimization and have code that can be tuned
(much low-level operations). gcl is generally faster than clisp (but
not that much), while clisp is more comfortable. gcl should be easier
to tune for a specific problem.
I didn't use Scheme that much, but I found elk to be faster and more
stable than scm and stk. But performance is not that greatest
difference between them.
>(**) I would definitely be satisfied with times for the Gabriel Lisp
>benchmarks (referenced in the MK FAQ) with as many variables as
>possible controlled.
The garbiel benchmarks are useful to tune an implementation. They
don't give an idea about the performance of real applications.
BTW, you can easily try performance of CL variants out. All but the
commercials are ftp-able and you can get evaluation copies of
Lispworks, Lucid (from Harlequin, too) and Allegro easily.
If you choose a commercial implementation, you can call Support to
help you with implementation-sepcific optimization. That can be more
important than the general quality of the optimizer. CMUCL seems to be
the fastest when it comes to things like processing large
floatig-point vectors.
If I really needed a fast Lisp, I would look at the underlying
technology of the compiler, so that I could make an assumption about
it's potential. And then choose the fastest I can get source for so I
can make it reach it's potential.
--
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
Private email ···············@wavehh.hanse.de Fax +4940 522 8536. No NeXTMail!
From: Ken Whedbee
Subject: Re: Benchmarking results for modern Lisps and Schemes
Date:
Message-ID: <3jhon3$g78@huron.eel.ufl.edu>
In article <·····················@ptolemy-ethernet.arc.nasa.gov>,
k p c <···@ptolemy.arc.nasa.gov> wrote:
>(*****) The reader might have seen my post on the subject some time
>ago. Since then, scsh and guile have come out and I discovered
>xlisp-plus. None of those were reviewed in my post. xlisp-plus
>interests me the most of these three, despite the features of scsh and
>guile, because it attempts slightly more to be compatible with CL.
>But speed factors greater than 2x and lack of regexps might veto it.
>I am not enough of a Schemer to benchmark Schemes, so I hope that
>others will be able to comment. I hope for a CLer response also.
You could load nregex.lsp from the Lisp Repository at CMU to get
regular expressions under lisp.
--
Ken Whedbee
···@hal.eng.ufl.edu
From: Ken Whedbee
Subject: Re: Benchmarking results for modern Lisps and Schemes
Date:
Message-ID: <3jhrmb$h9t@huron.eel.ufl.edu>
In article <·····················@ptolemy-ethernet.arc.nasa.gov>,
k p c <···@ptolemy.arc.nasa.gov> wrote:
>(*****) The reader might have seen my post on the subject some time
>ago. Since then, scsh and guile have come out and I discovered
>xlisp-plus. None of those were reviewed in my post. xlisp-plus
>interests me the most of these three, despite the features of scsh and
>guile, because it attempts slightly more to be compatible with CL.
>But speed factors greater than 2x and lack of regexps might veto it.
>I am not enough of a Schemer to benchmark Schemes, so I hope that
>others will be able to comment. I hope for a CLer response also.
Don't forget xlisp21gbc. This is xlisp21g which Luke Tierney has
added a byte compiler to. It is at the same ftp site as xlisp-stat.
--
Ken Whedbee
···@hal.eng.ufl.edu