From: Matthew D Swank
Subject: clozure-cl and the old standbys
Date: 
Message-ID: <H95Zj.91$Nk4.60@newsfe02.lga>
Copied from #lisp
<akopa> clozure-cl seems faster than cmucl and sbcl on linux x86_64.  Are
        there any comprehensive benchmarks comparing ccl 1.2 to recent 
        sbcls on this platform? I can't access boinkmarks right now, but 
        I don't think ccl is included.

Matt

From: ·······@eurogaran.com
Subject: Re: clozure-cl and the old standbys
Date: 
Message-ID: <1b07aea4-340d-40d9-86a4-3f50c8516cde@79g2000hsk.googlegroups.com>
I have mantained the unofficial debian package of openmcl.
Depending on the concrete type of operations you do, ccl/openmcl will
be slightly faster or slightly slower than sbcl. So consider them of
the same relative speed overall.
From: Raffael Cavallaro
Subject: Re: clozure-cl and the old standbys
Date: 
Message-ID: <1ec7edc4-c59f-48bb-93ee-f87fe8f50dbc@y38g2000hsy.googlegroups.com>
On May 22, 8:12 am, ·······@eurogaran.com wrote:
> I have mantained the unofficial debian package of openmcl.
> Depending on the concrete type of operations you do, ccl/openmcl will
> be slightly faster or slightly slower than sbcl. So consider them of
> the same relative speed overall.

run common lisp bench on both and look at the results. My experience
is that sbcl can be far faster with arrays and floating point
arithmetic than clozure cl, while clozure compiles far more quickly.

<http://www.chez.com/emarsden/downloads/cl-bench.tar.gz>
From: Rainer Joswig
Subject: Re: clozure-cl and the old standbys
Date: 
Message-ID: <joswig-06C5B0.21200922052008@news-europe.giganews.com>
In article 
<····································@y38g2000hsy.googlegroups.com>,
 Raffael Cavallaro <················@gmail.com> wrote:

> On May 22, 8:12�am, ·······@eurogaran.com wrote:
> > I have mantained the unofficial debian package of openmcl.
> > Depending on the concrete type of operations you do, ccl/openmcl will
> > be slightly faster or slightly slower than sbcl. So consider them of
> > the same relative speed overall.
> 
> run common lisp bench on both and look at the results. My experience
> is that sbcl can be far faster with arrays and floating point
> arithmetic than clozure cl, while clozure compiles far more quickly.
> 
> <http://www.chez.com/emarsden/downloads/cl-bench.tar.gz>

CLozure CL should also be faster for a bunch of CLOS operations.

SBCL has the advantage that the speed of the generated code
is more predictable for a wide variety of Lisp operations
(minus CLOS). Using the default settings works fine often
and then optimizing brings a bit more speed. With CCL
higher optimizations can make a huge difference, but it is not
so consistent - there are many areas where the CCL compiler is not
very sophisticated. But for many applications CCL should
deliver a good performance for 'real-world' code (opposed to
benchmark code).

I haven't tested a 64bit SBCL yet. What are the differences there?
With LispWorks I see that the 64bit version is almost twice as fast
as the 32bit version (not for every benchmark, but looking at a mix
of benchmarks). The 64bit version has really good performance.

-- 
http://lispm.dyndns.org/
From: Juho Snellman
Subject: Re: clozure-cl and the old standbys
Date: 
Message-ID: <87abiixgbx.fsf@vasara.proghammer.com>
Rainer Joswig <······@lisp.de> writes:
> CLozure CL should also be faster for a bunch of CLOS operations.

Do you have any concrete details on what that bunch of operations is?
Not that I'm doubting you, but this is the first time I've heard that
claim made, and that's certainly not what the (admittedly flawed) CLOS
benchmarks in cl-bench show.

> SBCL has the advantage that the speed of the generated code
> is more predictable for a wide variety of Lisp operations
> (minus CLOS).

And likewise, it'd be great to hear concrete details on what CLOS
operations you feel SBCL is giving unpredictable speed on.

-- 
Juho Snellman
From: Rainer Joswig
Subject: Re: clozure-cl and the old standbys
Date: 
Message-ID: <joswig-4DB58D.23570222052008@news-europe.giganews.com>
In article <··············@vasara.proghammer.com>,
 Juho Snellman <······@iki.fi> wrote:

> Rainer Joswig <······@lisp.de> writes:
> > CLozure CL should also be faster for a bunch of CLOS operations.
> 
> Do you have any concrete details on what that bunch of operations is?
> Not that I'm doubting you, but this is the first time I've heard that
> claim made, and that's certainly not what the (admittedly flawed) CLOS
> benchmarks in cl-bench show.

I was expecting that the benchmarks are not completely flawed.  I
was under the impression that for some of the CL-BENCH benchmarks
CCL is faster.

> 
> > SBCL has the advantage that the speed of the generated code
> > is more predictable for a wide variety of Lisp operations
> > (minus CLOS).
> 
> And likewise, it'd be great to hear concrete details on what CLOS
> operations you feel SBCL is giving unpredictable speed on.

One example I was thinking of is that SBCL is very fast
for MAKE-INSTANCE with a constant class name. If
MAKE-INSTANCE is invoked on the value of a variable,
then it is suddenly 50 times slower.

-- 
http://lispm.dyndns.org/
From: Juho Snellman
Subject: Re: clozure-cl and the old standbys
Date: 
Message-ID: <8763t5xl3f.fsf@vasara.proghammer.com>
Rainer Joswig <······@lisp.de> writes:
> In article <··············@vasara.proghammer.com>,
>  Juho Snellman <······@iki.fi> wrote:
> > Rainer Joswig <······@lisp.de> writes:
> > > CLozure CL should also be faster for a bunch of CLOS operations.
> > 
> > Do you have any concrete details on what that bunch of operations is?
> > Not that I'm doubting you, but this is the first time I've heard that
> > claim made, and that's certainly not what the (admittedly flawed) CLOS
> > benchmarks in cl-bench show.
> 
> I was expecting that the benchmarks are not completely flawed.  I
> was under the impression that for some of the CL-BENCH benchmarks
> CCL is faster.

Sure, a couple of them are. But that seems like a much less
interesting statement :-) You might as well have written "SBCL should
also be faster for a bunch of CLOS operations", and have been at least
as correct. Here's some results I got yesterday, from reasonably fresh
versions of CCL and SBCL:

                         CCL 1.2   SBCL 1.0.15.41
--------------------------------------------------
CLOS/instantiate         0.69      1
CLOS/simple-instantiate  30.0      1
CLOS/methodcalls         5.15      1
CLOS/method+after        0.67      1
CLOS/complex-methods     1.36      1
EQL-SPECIALIZED-FIB      2.75      1
--------------------------------------------------

(Runtimes of CCL relative to SBCL, smaller is faster)

There are a number of reasons why these benchmark numbers aren't
really that interesting. A big one is that they're completely
arbitrary microbenchmarks. A second one is that they don't often test
what the name implies.

For example on SBCL the cost of method dispatch is so fast that
CLOS/methodcalls is actually spending more time executing bignum
arithmetic in the method bodies than it is spent in anything related
to CLOS. In fact, when Christophe did some data analysis that
clustered the cl-benchmarks according to their historic behaviour,
CLOS/methodcalls fell into a cluster containing bignum tests. So it's
completely possible that method dispatch is as fast in CCL as in SBCL,
and it's just being hidden by those bignum calculations being slower
on CCL. And vice versa, you can't really conclude that :after methods
are slower on SBCL, without actually checking whether that's really
what that benchmark is doing.

So I was kind of hoping there was some more interesting empirical data
beyond your statement than cl-bench. Even though SBCL fares rather
well against all CL implementations on those cl-bench CLOS benchmarks :-)

I expect that the very common operations of creating instances of
known classes, accessing slots and doing "simple" GF dispatch is what
matters the most for real-world applications where CLOS is used
heavily enough for its speed to matter (eg. if the application spends
all its time in the database, obviously the speed of CLOS is
irrelevant).

It could be interesting to try something like running some
deterministic set of Climacs operations using the McCLIM null backend,
since that's some of the CLOS-heaviest openly released code I know of.

> > > SBCL has the advantage that the speed of the generated code is
> > > more predictable for a wide variety of Lisp operations (minus
> > > CLOS).
> > 
> > And likewise, it'd be great to hear concrete details on what CLOS
> > operations you feel SBCL is giving unpredictable speed on.
> 
> One example I was thinking of is that SBCL is very fast
> for MAKE-INSTANCE with a constant class name. If
> MAKE-INSTANCE is invoked on the value of a variable,
> then it is suddenly 50 times slower.

Fair enough. But it's not actually that MAKE-INSTANCE with a
non-constant class name is that slow in SBCL (according to the above,
it's around 30% slower than CCL), it's that MAKE-INSTANCE with a
constant class name and constant initarg names is really really fast
(30 times faster than CCL, if we trust those results). That seems
preferrable to having it be consistently slow.

-- 
Juho Snellman
From: Rainer Joswig
Subject: Re: clozure-cl and the old standbys
Date: 
Message-ID: <joswig-F3F0AF.18544723052008@news-europe.giganews.com>
In article <··············@vasara.proghammer.com>,
 Juho Snellman <······@iki.fi> wrote:

> Rainer Joswig <······@lisp.de> writes:
> > In article <··············@vasara.proghammer.com>,
> >  Juho Snellman <······@iki.fi> wrote:
> > > Rainer Joswig <······@lisp.de> writes:
> > > > CLozure CL should also be faster for a bunch of CLOS operations.
> > > 
> > > Do you have any concrete details on what that bunch of operations is?
> > > Not that I'm doubting you, but this is the first time I've heard that
> > > claim made, and that's certainly not what the (admittedly flawed) CLOS
> > > benchmarks in cl-bench show.
> > 
> > I was expecting that the benchmarks are not completely flawed.  I
> > was under the impression that for some of the CL-BENCH benchmarks
> > CCL is faster.
> 
> Sure, a couple of them are. But that seems like a much less
> interesting statement :-) You might as well have written "SBCL should
> also be faster for a bunch of CLOS operations", and have been at least
> as correct. Here's some results I got yesterday, from reasonably fresh
> versions of CCL and SBCL:
> 
>                          CCL 1.2   SBCL 1.0.15.41
> --------------------------------------------------
> CLOS/instantiate         0.69      1
> CLOS/simple-instantiate  30.0      1
> CLOS/methodcalls         5.15      1
> CLOS/method+after        0.67      1
> CLOS/complex-methods     1.36      1
> EQL-SPECIALIZED-FIB      2.75      1
> --------------------------------------------------
> 
> (Runtimes of CCL relative to SBCL, smaller is faster)

I have run those on a MacBook Pro and the ratios are looking
quite a bit better for the 64bit CCL vs. a 32bit SBCL.
I have rechecked and wrote a few primitive tests and
SBCL looks quite good for those.


> There are a number of reasons why these benchmark numbers aren't
> really that interesting. A big one is that they're completely
> arbitrary microbenchmarks. A second one is that they don't often test
> what the name implies.
> 
> For example on SBCL the cost of method dispatch is so fast that
> CLOS/methodcalls is actually spending more time executing bignum
> arithmetic in the method bodies than it is spent in anything related
> to CLOS. In fact, when Christophe did some data analysis that
> clustered the cl-benchmarks according to their historic behaviour,
> CLOS/methodcalls fell into a cluster containing bignum tests. So it's
> completely possible that method dispatch is as fast in CCL as in SBCL,
> and it's just being hidden by those bignum calculations being slower
> on CCL. And vice versa, you can't really conclude that :after methods
> are slower on SBCL, without actually checking whether that's really
> what that benchmark is doing.
> 
> So I was kind of hoping there was some more interesting empirical data
> beyond your statement than cl-bench. Even though SBCL fares rather
> well against all CL implementations on those cl-bench CLOS benchmarks :-)

I see that LispWorks is quite a bit faster than SBCL on those.
 
> I expect that the very common operations of creating instances of
> known classes, accessing slots and doing "simple" GF dispatch is what
> matters the most for real-world applications where CLOS is used
> heavily enough for its speed to matter (eg. if the application spends
> all its time in the database, obviously the speed of CLOS is
> irrelevant).

I don't think so. Part of the reason to use CLOS with all
its design 'overhead' is that it enables lots of interesting
dynamic behaviour. I would expect that the need to optimize
dynamic CLOS as much as the need of a static CLOS app
(where all classes are known, etc.).

> It could be interesting to try something like running some
> deterministic set of Climacs operations using the McCLIM null backend,
> since that's some of the CLOS-heaviest openly released code I know of.

Makes sense.

Though I think one needs better micro-benchmarks, too.

What would those be?

* define a class with DEFCLASS
* define a class with ENSURE-CLASS

* instance creation
** fixed class, no args
** fixed class, fixed initargs
** fixed class, variable initargs
** variable class, no args
** variable class, fixed args
** variable class, variable args
** small number of slots ... large number of slots
** one class, vs. hierarchy of classes

* method call
** one arg, two args, ... n args
** dispatch on first arg, second arg, ...
** multiple dispatch
** different method combinations
** computed arglist
** simple class, vs. complex class hierarchy
** standard-method-combination
*** :before, :after, :around
** EQL specializer
** method calls after classes have been changed
** calls with objects that have been updated

* slot access
** read, write
** with method for SLOT-VALUE-USING-CLASS
** class slots / instance slots

* accessor
** reader/writer
** simple vs. added methods


> > > > SBCL has the advantage that the speed of the generated code is
> > > > more predictable for a wide variety of Lisp operations (minus
> > > > CLOS).
> > > 
> > > And likewise, it'd be great to hear concrete details on what CLOS
> > > operations you feel SBCL is giving unpredictable speed on.
> > 
> > One example I was thinking of is that SBCL is very fast
> > for MAKE-INSTANCE with a constant class name. If
> > MAKE-INSTANCE is invoked on the value of a variable,
> > then it is suddenly 50 times slower.
> 
> Fair enough. But it's not actually that MAKE-INSTANCE with a
> non-constant class name is that slow in SBCL (according to the above,
> it's around 30% slower than CCL), it's that MAKE-INSTANCE with a
> constant class name and constant initarg names is really really fast
> (30 times faster than CCL, if we trust those results). That seems
> preferrable to having it be consistently slow.

LispWorks is much faster for the non-constant case (I've seen upto
factor 8) and slower (upto factor 2.5) for the constant case.

I was kind of disappointed by the CCL speed when I wrote my own
tests. SBCL looks extremely good for the constant case, but
I was also disappointed by its speed for the non-constant case.

I find the non-constant case of MAKE-INSTANCE quite common. Like:

(defun make-button (&key (class *standard-button-class*))
   (make-instance class ...))

(let ((*standard-button-class* *gtk-button-class*))
  (make-button ...))

And so on. I also expect lots of code to be generated,
where the class is also chosen sometime later based
on runtime information.

With most other datastructures one chooses them before compile-time.
CLOS is then especially useful when applications need to be
configured at runtime and the actual classes/hierarchies and
sets of methods might change.

-- 
http://lispm.dyndns.org/
From: Waldek Hebisch
Subject: Re: clozure-cl and the old standbys
Date: 
Message-ID: <g14fsp$c2f$1@z-news.pwr.wroc.pl>
Matthew D Swank <··················@gmail.com> wrote:
> Copied from #lisp
> <akopa> clozure-cl seems faster than cmucl and sbcl on linux x86_64.  Are
>         there any comprehensive benchmarks comparing ccl 1.2 to recent 
>         sbcls on this platform? I can't access boinkmarks right now, but 
>         I don't think ccl is included.
> 
> Matt

One data point: FriCAS test suite on 2.66 GHz Pentium D (measured
using Linux time command):

       Closure CL 1.2-rc1             sbcl-1.0.16
real    13m17.471s                    11m17.416s
user    12m44.236s                    10m41.280s
sys      0m15.209s                     0m36.610s


Test suite uses only little floating point.  Main operations
are function calls, fixnum arithmetic, list and vector traversal.
10-20% is bignum arithmetic.  Both for Closure CL and sbcl
I use default optimize/safety settings.  In my experience
sbcl default checks are quite effective finding problems
with moderate impact on speed.  I yesterday downloaded Closure CL
1.2-rc1, so had only little time to play with it, but
IIRC OpenMcl used to include by default some runtime checks,
so the comparison should be fair.

Using older versions (older FriCAS version, older sbcl and
older openmcl snapshot) on 2GHz AMD64 machine I had:

openmcl             sbcl

15m38s             11m22s

I am not sure if Closure CL 1.2 significantly improved speed
compared to snapshots or just differences are due to machine
architecture.

-- 
                              Waldek Hebisch
·······@math.uni.wroc.pl