From: Sean Doran
Subject: CMUCL compiler bug? (old lazy-list data not gc-ed)
Date:
Message-ID: <52g1qknu50.fsf@sean.ebone.net>
I'm using the following CMUCL system:
CMU Common Lisp 18a.97.08.24, running on sean.ebone.net
Send bug reports and questions to your local CMU CL maintainer, or to
··········@cs.cmu.edu.
Loaded subsystems:
Python 1.0, target Intel x86
CLOS based on PCL version: September 16 92 PCL (f)
CLX X Library MIT R5.02
Motif toolkit and graphical debugger 1.0
Hemlock 3.5
(and yes I did try ··········@cs.cmu.edu :-) )
and the following source code file:
; from graham, p. 211
(defconstant unforced (gensym))
(defstruct delay forced closure)
(defmacro delay (expr)
(let ((self (gensym)))
`(let ((,self (make-delay :forced unforced)))
(setf (delay-closure ,self)
#'(lambda ()
(setf (delay-forced ,self) ,expr)))
,self)))
(defun force (x)
(if (delay-p x)
(if (eq (delay-forced x) unforced)
(funcall (delay-closure x))
(delay-forced x))
x))
; me
(defun search-ll (pred ll)
"search a lazy list"
(let ((lis (force ll)))
(if (funcall pred (car lis)) lis
(search-ll pred (cdr lis)))))
(defun counter (i)
(delay (cons i (counter (+ i 1)))))
and when I load this and invoke:
time (search-ll (lambda (n) (= n 5000)) (counter 0)))
Compiling LAMBDA NIL:
Compiling Top-Level Form:
[GC threshold exceeded with 2,000,384 bytes in use. Commencing GC.]
[GC completed with 176,128 bytes retained and 1,824,256 bytes freed.]
[GC will next occur when at least 2,176,128 bytes are in use.]
[GC threshold exceeded with 2,176,512 bytes in use. Commencing GC.]
[GC completed with 374,784 bytes retained and 1,801,728 bytes freed.]
[GC will next occur when at least 2,374,784 bytes are in use.]
Evaluation took:
10.4 seconds of real time
8.843696 seconds of user run time
0.251432 seconds of system run time
[Run times include 0.77 seconds GC run time]
1580 page faults and
5493248 bytes consed.
(5000
. #S(DELAY
:FORCED #:G415
:CLOSURE #<Interpreted Function "LAMBDA (I)" {91BECD1}>))
(I see a nice essentially linear curve as I adjust the value of n to
test for; for n=50000, for example:
Evaluation took:
115.39 seconds of real time
96.352486 seconds of user run time
2.173703 seconds of system run time
[Run times include 14.62 seconds GC run time]
19427 page faults and
54946304 bytes consed.)
This is really slow so I compile and load the resulting
.x86f file.
* (time (search-ll (lambda (n) (= n 50000)) (counter 0)))
Compiling LAMBDA NIL:
Compiling Top-Level Form:
[GC threshold exceeded with 2,348,032 bytes in use. Commencing GC.]
[GC completed with 2,067,968 bytes retained and 280,064 bytes freed.]
[GC will next occur when at least 4,067,968 bytes are in use.]
Evaluation took:
1.42 seconds of real time
0.982011 seconds of user run time
0.059257 seconds of system run time
[Run times include 0.8 seconds GC run time]
955 page faults and
2078720 bytes consed.
(50000
. #S(DELAY
:FORCED #:G423
:CLOSURE #<Closure Over Function "DEFUN COUNTER" {9045661}>))
which is nice nut if I try for 500000, I see that the
garbage collector is not recovering memory:
* (time (search-ll (lambda (n) (= n 500000)) (counter 0)))
Compiling LAMBDA NIL:
Compiling Top-Level Form:
[GC threshold exceeded with 3,899,392 bytes in use. Commencing GC.]
[GC completed with 1,711,616 bytes retained and 2,187,776 bytes freed.]
[GC will next occur when at least 3,711,616 bytes are in use.]
[GC threshold exceeded with 3,712,000 bytes in use. Commencing GC.]
[GC completed with 3,708,928 bytes retained and 3,072 bytes freed.]
[GC will next occur when at least 5,708,928 bytes are in use.]
[GC threshold exceeded with 5,709,312 bytes in use. Commencing GC.]
[GC completed with 5,707,776 bytes retained and 1,536 bytes freed.]
[GC will next occur when at least 7,707,776 bytes are in use.]
[GC threshold exceeded with 7,708,160 bytes in use. Commencing GC.]
[GC completed with 7,703,552 bytes retained and 4,608 bytes freed.]
[GC will next occur when at least 9,703,552 bytes are in use.]
[GC threshold exceeded with 9,703,936 bytes in use. Commencing GC.]
[GC completed with 9,700,352 bytes retained and 3,584 bytes freed.]
[GC will next occur when at least 11,700,352 bytes are in use.]
[GC threshold exceeded with 11,700,736 bytes in use. Commencing GC.]
[GC completed with 11,698,176 bytes retained and 2,560 bytes freed.]
[GC will next occur when at least 13,698,176 bytes are in use.]
[GC threshold exceeded with 13,698,560 bytes in use. Commencing GC.]
[GC completed with 13,695,488 bytes retained and 3,072 bytes freed.]
[GC will next occur when at least 15,695,488 bytes are in use.]
[GC threshold exceeded with 15,695,872 bytes in use. Commencing GC.]
[GC completed with 15,692,800 bytes retained and 3,072 bytes freed.]
[GC will next occur when at least 17,692,800 bytes are in use.]
[GC threshold exceeded with 17,693,184 bytes in use. Commencing GC.]
[GC completed with 17,690,112 bytes retained and 3,072 bytes freed.]
[GC will next occur when at least 19,690,112 bytes are in use.]
[GC threshold exceeded with 19,690,496 bytes in use. Commencing GC.]
[GC completed with 19,686,912 bytes retained and 3,584 bytes freed.]
[GC will next occur when at least 21,686,912 bytes are in use.]
Evaluation took:
86.39 seconds of real time
29.462076 seconds of user run time
10.088135 seconds of system run time
[Run times include 36.97 seconds GC run time]
133262 page faults and
20787712 bytes consed.
(500000
. #S(DELAY
:FORCED #:G423
:CLOSURE #<Closure Over Function "DEFUN COUNTER" {91452D9}>))
This is obviously bad behaviour when processing very long
lazy-lists.
Again, I don't see this behaviour at all with the
uncompiled version. (I ran an = n 500000 on the .lisp
file and it behaved properly).
Is this expected behaviour with Python or is this a bug?
Sean.
From: Raymond Toy
Subject: Re: CMUCL compiler bug? (old lazy-list data not gc-ed)
Date:
Message-ID: <4nsoukkv3w.fsf@rtp.ericsson.se>
>>>>> "Sean" == Sean Doran <···@sean.ebone.net> writes:
Sean> I'm using the following CMUCL system:
Sean> CMU Common Lisp 18a.97.08.24, running on sean.ebone.net Send
Sean> bug reports and questions to your local CMU CL maintainer,
Sean> or to ··········@cs.cmu.edu. Loaded subsystems:
Sean> Python 1.0, target Intel x86 CLOS based on PCL version:
Sean> September 16 92 PCL (f) CLX X Library MIT R5.02 Motif
Sean> toolkit and graphical debugger 1.0 Hemlock 3.5
Sean> (and yes I did try ··········@cs.cmu.edu :-) )
You may have better luck sending to ·········@cons.org or
··········@cons.org.
I've forwarded this message to ·········@cons.org.
[source code deleted]
Sean> Again, I don't see this behaviour at all with the uncompiled
Sean> version. (I ran an = n 500000 on the .lisp file and it
Sean> behaved properly).
Sean> Is this expected behaviour with Python or is this a bug?
It appears to be a bug in the x86 version. I don't see this on the
Sparc version. n = 500000 runs very quickly and the GC messages show
memory being reclaimed.
Ray
From: Sean Doran
Subject: Re: CMUCL compiler bug? (old lazy-list data not gc-ed)
Date:
Message-ID: <5290w7yaeg.fsf@sean.ebone.net>
Raymond Toy <···@rtp.ericsson.se> writes:
> It appears to be a bug in the x86 version. I don't see this on the
> Sparc version. n = 500000 runs very quickly and the GC messages show
> memory being reclaimed.
I have been able to test this out too. I get proper
behaviour with 18a running on a SPARC running SunOS 4.1.4,
and I get bad behaviour with 17f running on IRIX 5.3.
I tried GCL on a lark, and GCL also keels over dead on the
x86 architechture. I haven't checked other architechtures
yet. With GCL unless you bump up the stacksize rlimit,
you get a core dump which shows endless calls to garbage
collection marking routines.
The bright side is that with a sufficiently large stack
size, memory apparently isn't eaten as badly as quickly as
with CMUCL.
I'm trying to subscribe to the cmucl-imp list (no response
yet from ·················@cons.org or from
··························@cons.org), and when I do I
shall have some other questions about the .x86 port.
Finally, some people sent me some explanations for the
CMUCL behaviour which I will digest when I have a moment.
Sean.