Hi all,
I'm trying to understand how to reclaim space in CMUCL after creating a
huge array. Can anyone point out what's wrong here:
* (progn (defparameter a (make-array 100000000)) nil)
nil
* (funcall (compile nil (lambda () (loop for i from 0 to 10000000 do
(setf (aref a i) "fixed string")))))
; Compiling lambda nil:
; Compiling Top-Level Form:
nil
So I'm now using a few MB of memory:
* (gc :full t)
nil
* (room)
Dynamic Space Usage: 400,074,936 bytes.
Read-Only Space Usage: 19,419,824 bytes.
Static Space Usage: 2,686,400 bytes.
Control Stack Usage: 548 bytes.
Binding Stack Usage: 96 bytes.
The current dynamic space is 0.
Garbage collection is currently enabled.
Breakdown for dynamic space:
400,007,576 bytes for 48 simple-vector-type objects.
482,920 bytes for 53,743 other objects.
400,490,496 bytes for 53,791 dynamic objects (space total.)
So I now decide to (setf a nil). As there is now no pointer to the array I
was expecting the array to be garbage collected:
* (setf a nil)
nil
* (gc :full t)
nil
* (room)
Dynamic Space Usage: 400,073,800 bytes.
Read-Only Space Usage: 19,419,824 bytes.
Static Space Usage: 2,686,400 bytes.
Control Stack Usage: 548 bytes.
Binding Stack Usage: 96 bytes.
The current dynamic space is 0.
Garbage collection is currently enabled.
Breakdown for dynamic space:
400,007,576 bytes for 48 simple-vector-type objects.
482,920 bytes for 53,782 other objects.
400,490,496 bytes for 53,830 dynamic objects (space total.)
But I'm still using just as much space. It doesn't even help if I
unintern A. What's the appropriate way to reclaim the space?
Thanks,
Adam
"Adam Warner" <······@consulting.net.nz> writes:
> So I now decide to (setf a nil). As there is now no pointer to the
> array I was expecting the array to be garbage collected:
Maybe it's a bug in the version of CMUCL you're using. It seems to
work OK in 18e:
snapdragon:man1 > lisp -dynamic-space-size 500
; Loading #p"/export/u1/homes/gilham/init.lisp".
; Compiling LAMBDA (VAR):
; Compiling Top-Level Form:
;; Loading pathname translations from /export/u1/homes/gilham/cmucl/lib/unix.translations
;; Loading #p"/export/u1/homes/gilham/cmucl/lib/subsystems/defsystem-library.x86f".
;; Loading #p"/export/u1/homes/gilham/cmucl/lib/subsystems/inspect-library.x86f".
Warning: INSPECT also exports the following symbols:
(INSPECT:REMOVE-OBJECT-DISPLAY INSPECT:REMOVE-ALL-DISPLAYS)
;; Loading local initializations (if any).
CMU Common Lisp 18e, running on snapdragon.csl.sri.com
With core: /export/u1/homes/gilham/cmucl/lib/lisp.core
Dumped on: Fri, 2003-04-04 12:53:56-08:00 on snapdragon.csl.sri.com
Send questions and bug reports to gilham,
or see <URL:http://www.cons.org/cmucl/support.html>.
Loaded subsystems:
Python 1.1, target Intel x86
CLOS 18e (based on PCL September 16 92 PCL (f))
Gray Streams Protocol Support
CLX X Library MIT R5.02
* (progn (defparameter a (make-array 100000000)) nil)
NIL
* (gc :full t)
NIL
* (room)
Dynamic Space Usage: 400,912,168 bytes.
Read-Only Space Usage: 20,876,776 bytes.
Static Space Usage: 2,932,864 bytes.
Control Stack Usage: 484 bytes.
Binding Stack Usage: 96 bytes.
The current dynamic space is 0.
Garbage collection is currently enabled.
Breakdown for dynamic space:
400,127,776 bytes for 1,455 simple-vector-type objects.
4,282,592 bytes for 468,252 other objects.
404,410,368 bytes for 469,707 dynamic objects (space total.)
* (setf a nil)
NIL
* (gc :full t)
NIL
* (room)
Dynamic Space Usage: 915,208 bytes.
Read-Only Space Usage: 20,876,776 bytes.
Static Space Usage: 2,932,864 bytes.
Control Stack Usage: 484 bytes.
Binding Stack Usage: 96 bytes.
The current dynamic space is 0.
Garbage collection is currently enabled.
Breakdown for dynamic space:
3,639,368 bytes for 454,921 cons objects.
751,544 bytes for 12,063 other objects.
4,390,912 bytes for 466,984 dynamic objects (space total.)
*
--
Fred Gilham ······@csl.sri.com
The amazing thing is, back when I was a C++ programmer, I thought that
[Design Patterns] was SUCH a great book. And I guess it was, in a
way: it made it possible to use C++ and get something done. In the
long run, of course, that may have been a disservice... - Alain Picard
Hi Fred Gilham,
>> So I now decide to (setf a nil). As there is now no pointer to the
>> array I was expecting the array to be garbage collected:
>
> Maybe it's a bug in the version of CMUCL you're using. It seems to work
> OK in 18e:
It also does in CMU Common Lisp release x86-linux 3.1.7 18d+ 18 January
2003 build 4523, since you skipped a step (I set 10% of the array values
to literal strings before the setf).
Regards,
Adam
Adam Warner wrote:
> Hi Fred Gilham,
>
>
>>>So I now decide to (setf a nil). As there is now no pointer to the
>>>array I was expecting the array to be garbage collected:
>>
>>Maybe it's a bug in the version of CMUCL you're using. It seems to work
>>OK in 18e:
>
>
> It also does in CMU Common Lisp release x86-linux 3.1.7 18d+ 18 January
> 2003 build 4523, since you skipped a step (I set 10% of the array values
> to literal strings before the setf).
>
> Regards,
> Adam
Pardon a question from the peanut gallery, but do you see the same
behavior if you run the same exercise inside a function, instead of in
the REPL? Just wondering if the repl logic somehow ended up with a copy
of the reference to the big array.
--
kenny tilton
clinisys, inc
http://www.tilton-technology.com/
---------------------------------------------------------------
"Everything is a cell." -- Alan Kay
In article <················@nyc.rr.com>,
Kenny Tilton <·······@nyc.rr.com> wrote:
>Pardon a question from the peanut gallery, but do you see the same
>behavior if you run the same exercise inside a function, instead of in
>the REPL? Just wondering if the repl logic somehow ended up with a copy
>of the reference to the big array.
If you reread the original post, you'll see that he was careful to wrap
things in PROGN's that returned NIL, rather than the array, so that none of
the REPL's variables would capture references to the array.
--
Barry Margolin, ··············@level3.com
Genuity Managed Services, a Level(3) Company, Woburn, MA
*** DON'T SEND TECHNICAL QUESTIONS DIRECTLY TO ME, post them to newsgroups.
Please DON'T copy followups to me -- I'll assume it wasn't posted to the group.
Barry Margolin <··············@level3.com> wrote in message news:<··············@paloalto-snr1.gtei.net>...
> In article <················@nyc.rr.com>,
> Kenny Tilton <·······@nyc.rr.com> wrote:
> >Pardon a question from the peanut gallery, but do you see the same
> >behavior if you run the same exercise inside a function, instead of in
> >the REPL? Just wondering if the repl logic somehow ended up with a copy
> >of the reference to the big array.
>
> If you reread the original post, you'll see that he was careful to wrap
> things in PROGN's that returned NIL...
yeah, i saw that, that is why i said "if the repl logic somehow ended
up with a copy". ie, i was not blaming his code (youse guys would have
spotted any flaw), i was guessing at where a bug or explanation might
be. the compilation suggestion was meant to just get the interpreter
issues and repl issues out of the list of suspects.
glad it worked. :)
kt
In article <····························@posting.google.com>,
Kenny Tilton <·······@nyc.rr.com> wrote:
>Barry Margolin <··············@level3.com> wrote in message
>news:<··············@paloalto-snr1.gtei.net>...
>> In article <················@nyc.rr.com>,
>> Kenny Tilton <·······@nyc.rr.com> wrote:
>> >Pardon a question from the peanut gallery, but do you see the same
>> >behavior if you run the same exercise inside a function, instead of in
>> >the REPL? Just wondering if the repl logic somehow ended up with a copy
>> >of the reference to the big array.
>>
>> If you reread the original post, you'll see that he was careful to wrap
>> things in PROGN's that returned NIL...
>
>yeah, i saw that, that is why i said "if the repl logic somehow ended
>up with a copy". ie, i was not blaming his code (youse guys would have
>spotted any flaw), i was guessing at where a bug or explanation might
>be. the compilation suggestion was meant to just get the interpreter
>issues and repl issues out of the list of suspects.
The REPL is not some magical, mystical piece of software that grabs random
objects in memory. The only things it sees are the expressions that it
reads and prints, so those are the only things it should ever end up with
pointers to.
--
Barry Margolin, ··············@level3.com
Genuity Managed Services, a Level(3) Company, Woburn, MA
*** DON'T SEND TECHNICAL QUESTIONS DIRECTLY TO ME, post them to newsgroups.
Please DON'T copy followups to me -- I'll assume it wasn't posted to the group.
Barry Margolin wrote:
> In article <····························@posting.google.com>,
> Kenny Tilton <·······@nyc.rr.com> wrote:
>
>>Barry Margolin <··············@level3.com> wrote in message
>>news:<··············@paloalto-snr1.gtei.net>...
>>
>>>In article <················@nyc.rr.com>,
>>>Kenny Tilton <·······@nyc.rr.com> wrote:
>>>
>>>>Pardon a question from the peanut gallery, but do you see the same
>>>>behavior if you run the same exercise inside a function, instead of in
>>>>the REPL? Just wondering if the repl logic somehow ended up with a copy
>>>>of the reference to the big array.
>>>
>>>If you reread the original post, you'll see that he was careful to wrap
>>>things in PROGN's that returned NIL...
>>
>>yeah, i saw that, that is why i said "if the repl logic somehow ended
>>up with a copy". ie, i was not blaming his code (youse guys would have
>>spotted any flaw), i was guessing at where a bug or explanation might
>>be. the compilation suggestion was meant to just get the interpreter
>>issues and repl issues out of the list of suspects.
>
>
> The REPL is not some magical, mystical piece of software that grabs random
> objects in memory. The only things it sees are the expressions that it
> reads and prints, so those are the only things it should ever end up with
> pointers to.
>
Do you relly think that is a REPL you are breathing?
There is no REPL, there are only wadges of code which generate the
illusion of one. code which might have a bug. We were looking for a bug,
so why the lectures on how perfect code works?
Lemme ask you this: what if the "E" involves an interpreter? I have no
idea if CMU works that way, but if a repl does, are you as confident
about the sanctity of all the interpreter code? Could the interpreter,
if it were written in Lisp, end up with a reference to an intermediate
result?
The biggest problem here is that you did not give sufficient weight to
my "from the peanut gallery" disclaimer. I was just lobbing an idea onto
the field to get the group unstuck.
Glad it worked.
:)
--
kenny tilton
clinisys, inc
http://www.tilton-technology.com/
---------------------------------------------------------------
"Everything is a cell." -- Alan Kay
From: Michael J. Ferrador
Subject: Re: Reclaiming space with CMUCL
Date:
Message-ID: <3E95D2C5.DD5EDC0F@orn.com>
Barry Margolin wrote:
> In article <················@nyc.rr.com>,
> Kenny Tilton <·······@nyc.rr.com> wrote:
> >Pardon a question from the peanut gallery, but do you see the same
> >behavior if you run the same exercise inside a function, instead of in
> >the REPL? Just wondering if the repl logic somehow ended up with a copy
> >of the reference to the big array.
>
> If you reread the original post, you'll see that he was careful to wrap
> things in PROGN's that returned NIL, rather than the array, so that none of
> the REPL's variables would capture references to the array.
These (CLHS 25.1.1) variables? -,+,/,*,++, etc...
Hi Kenny Tilton,
>> It also does in CMU Common Lisp release x86-linux 3.1.7 18d+ 18 January
>> 2003 build 4523, since you skipped a step (I set 10% of the array values
>> to literal strings before the setf).
>>
>> Regards,
>> Adam
>
> Pardon a question from the peanut gallery, but do you see the same
> behavior if you run the same exercise inside a function, instead of in
> the REPL? Just wondering if the repl logic somehow ended up with a copy
> of the reference to the big array.
18 Jan 2003 build with the same code saved in mem.lisp:
(progn (defparameter a (make-array 100000000)) nil)
(funcall (compile nil (lambda () (loop for i from 0 to 10000000 do
(setf (aref a i) "fixed string")))))
(gc :full t)
(room)
(setf a nil)
(gc :full t)
(room)
(unintern 'a)
(gc :full t)
(room)
Result is the same:
$ /usr/bin/lisp -noinit -load mem.lisp
; Loading #p"/home/adam/t/mem.lisp".
; [GC threshold exceeded with 400,159,184 bytes in use. Commencing GC.]
; [GC completed with 400,049,112 bytes retained and 110,072 bytes freed.]
; [GC will next occur when at least 402,049,112 bytes are in use.]
; Compiling LAMBDA NIL:
; Compiling Top-Level Form:
; [GC threshold exceeded with 400,249,016 bytes in use. Commencing GC.]
; [GC completed with 400,080,240 bytes retained and 168,776 bytes freed.]
; [GC will next occur when at least 402,080,240 bytes are in use.]
Dynamic Space Usage: 400,082,872 bytes.
Read-Only Space Usage: 19,419,824 bytes.
Static Space Usage: 2,686,400 bytes.
Control Stack Usage: 1,572 bytes.
Binding Stack Usage: 88 bytes.
The current dynamic space is 0.
Garbage collection is currently enabled.
Breakdown for dynamic space:
400,006,208 bytes for 35 simple-vector-type objects.
254,912 bytes for 25,813 other objects.
400,261,120 bytes for 25,848 dynamic objects (space total.)
; [GC threshold exceeded with 400,101,592 bytes in use. Commencing GC.]
; [GC completed with 400,080,984 bytes retained and 20,608 bytes freed.]
; [GC will next occur when at least 402,080,984 bytes are in use.]
Dynamic Space Usage: 400,083,616 bytes.
Read-Only Space Usage: 19,419,824 bytes.
Static Space Usage: 2,686,400 bytes.
Control Stack Usage: 1,572 bytes.
Binding Stack Usage: 88 bytes.
The current dynamic space is 0.
Garbage collection is currently enabled.
Breakdown for dynamic space:
400,005,600 bytes for 31 simple-vector-type objects.
169,504 bytes for 14,890 other objects.
400,175,104 bytes for 14,921 dynamic objects (space total.)
; [GC threshold exceeded with 400,101,832 bytes in use. Commencing GC.]
; [GC completed with 400,076,368 bytes retained and 25,464 bytes freed.]
; [GC will next occur when at least 402,076,368 bytes are in use.]
Dynamic Space Usage: 400,079,000 bytes.
Read-Only Space Usage: 19,419,824 bytes.
Static Space Usage: 2,686,400 bytes.
Control Stack Usage: 1,572 bytes.
Binding Stack Usage: 88 bytes.
The current dynamic space is 0.
Garbage collection is currently enabled.
Breakdown for dynamic space:
400,005,600 bytes for 31 simple-vector-type objects.
169,504 bytes for 15,335 other objects.
400,175,104 bytes for 15,366 dynamic objects (space total.)
Completely different result if the code mem2.lisp below is compiled (maybe
it works uncompiled. It was taking so long that I halted it):
(defparameter a (make-array 100000000))
(defun create-strings ()
(loop for i from 0 to 10000000 do
(setf (aref a i) "fixed string")))
(create-strings)
(gc :full t)
(room)
(setf a nil)
(gc :full t)
(room)
(unintern 'a)
(gc :full t)
(room)
$ /usr/bin/lisp -noinit -load mem2.x86f
; Loading #p"/home/adam/t/mem2.x86f".
; [GC threshold exceeded with 400,148,376 bytes in use. Commencing GC.]
; [GC completed with 400,050,560 bytes retained and 97,816 bytes freed.]
; [GC will next occur when at least 402,050,560 bytes are in use.]
Dynamic Space Usage: 400,053,160 bytes.
Read-Only Space Usage: 19,419,824 bytes.
Static Space Usage: 2,686,400 bytes.
Control Stack Usage: 1,860 bytes.
Binding Stack Usage: 128 bytes.
The current dynamic space is 0.
Garbage collection is currently enabled.
Breakdown for dynamic space:
400,004,848 bytes for 36 simple-vector-type objects.
149,776 bytes for 14,701 other objects.
400,154,624 bytes for 14,737 dynamic objects (space total.)
; [GC threshold exceeded with 400,071,568 bytes in use. Commencing GC.]
; [GC completed with 53,104 bytes retained and 400,018,464 bytes freed.]
; [GC will next occur when at least 2,053,104 bytes are in use.]
Dynamic Space Usage: 55,696 bytes.
Read-Only Space Usage: 19,419,824 bytes.
Static Space Usage: 2,686,400 bytes.
Control Stack Usage: 1,860 bytes.
Binding Stack Usage: 128 bytes.
The current dynamic space is 0.
Garbage collection is currently enabled.
Breakdown for dynamic space:
101,992 bytes for 12,749 cons objects.
18,632 bytes for 412 simple-string-type objects.
10,504 bytes for 204 instance objects.
20,424 bytes for 764 other objects.
151,552 bytes for 14,129 dynamic objects (space total.)
; [GC threshold exceeded with 79,264 bytes in use. Commencing GC.]
; [GC completed with 58,272 bytes retained and 20,992 bytes freed.]
; [GC will next occur when at least 2,058,272 bytes are in use.]
Dynamic Space Usage: 60,848 bytes.
Read-Only Space Usage: 19,419,824 bytes.
Static Space Usage: 2,686,400 bytes.
Control Stack Usage: 1,860 bytes.
Binding Stack Usage: 128 bytes.
The current dynamic space is 0.
Garbage collection is currently enabled.
Breakdown for dynamic space:
97,352 bytes for 12,169 cons objects.
21,696 bytes for 451 simple-string-type objects.
11,896 bytes for 232 instance objects.
20,608 bytes for 780 other objects.
151,552 bytes for 13,632 dynamic objects (space total.)
The difference seems to be my use of local compilation in the first test.
Regards,
Adam
"Adam Warner" <······@consulting.net.nz> writes:
> Hi Fred Gilham,
>
> >> So I now decide to (setf a nil). As there is now no pointer to the
> >> array I was expecting the array to be garbage collected:
> >
> > Maybe it's a bug in the version of CMUCL you're using. It seems to work
> > OK in 18e:
>
> It also does in CMU Common Lisp release x86-linux 3.1.7 18d+ 18 January
> 2003 build 4523, since you skipped a step (I set 10% of the array values
> to literal strings before the setf).
I've been busy the last few days and haven't followed this thread as
closely as I otherwise would have -- did you ever figure out what's
going on here? This in particular makes me suspect that it's junk on
the stack confusing the GC.
--
/|_ .-----------------------.
,' .\ / | No to Imperialist war |
,--' _,' | Wage class war! |
/ / `-----------------------'
( -. |
| ) |
(`-. '--.)
`. )----'
Hi Thomas F. Burdick,
> I've been busy the last few days and haven't followed this thread as
> closely as I otherwise would have -- did you ever figure out what's
> going on here?
No. The only thing I know is that these two approaches caused different
results [with (defparameter a (make-array 100000000))]:
;;only form locally compiled in mem.lisp
(funcall (compile nil (lambda () (loop for i from 0 to 10000000 do
(setf (aref a i) "fixed string")))))
;;part of the compiled program mem2.x86f
(defun create-strings ()
(loop for i from 0 to 10000000 do
(setf (aref a i) "fixed string")))
(create-strings)
Refer <http://groups.google.co.nz/groups?selm=pan.2003.04.10.05.08.47.287095%40consulting.net.nz>
Regards,
Adam
>>>>> "aw" == Adam Warner <······@consulting.net.nz> writes:
aw> I'm trying to understand how to reclaim space in CMUCL after
aw> creating a huge array. Can anyone point out what's wrong here:
aw>
aw> * (progn (defparameter a (make-array 100000000)) nil)
aw> * (setf a nil)
aw> * (gc :full t)
aw> * (room)
aw>
aw> Dynamic Space Usage: 400,073,800 bytes.
aw> Read-Only Space Usage: 19,419,824 bytes.
aw> Static Space Usage: 2,686,400 bytes.
aw> Control Stack Usage: 548 bytes.
aw> Binding Stack Usage: 96 bytes.
you're running into a limitation of the conservative garbage collector
used on x86 platforms (this doesn't happen on the SPARC/Solaris port
of CMUCL, for instance). The problem is that you haven't specified an
initial element for the array, so its elements aren't initialized, and
are full of random junk. This confuses the garbage collector, since
some of the junk looks like it might be a reference to a valid memory
location. The array therefore potentially contains live objects, and
isn't collected.
If you use :initial-element, or declare the element-type, or fill all
the positions in the array during program execution, you won't run
into this problem.
You can (to some extent) diagnose this type of problem by setting the
alien variable gencgc_verbose. In the following transcript you can
watch the generational collector moving the array from the youngest
generations to the oldest, where it will remain uncollected. You can
see that the array is being handled by special "large-object" code.
USER> (alien:def-alien-variable ("gencgc_verbose" gencgc-verbose) c-call::int)
#<alien::heap-alien-info (system:foreign-symbol-address "gencgc_verbose" :flavor :data) (alien:signed 32)>
USER> (setf gencgc-verbose 2)
2
USER> (setq *a* nil)
nil
USER> (gc :full t)
Generation Boxed Unboxed LB LUB Alloc Waste Trig WP GCs Mem-age
0: 1471 0 0 0 6022632 2584 10000000 0 0 0.0000
1: 254 70 97657 0 401296280 33896 411252792 97843 1 2.9998
2: 0 0 0 0 0 0 2000000 0 0 0.0000
3: 0 0 0 0 0 0 2000000 0 0 0.0000
4: 0 0 0 0 0 0 2000000 0 0 0.0000
5: 0 0 0 0 0 0 2000000 0 0 0.0000
Total bytes alloc=407318912
Starting GC of generation 0 with raise=1 alloc=6022632 trig=10000000 GCs=0
*W control stack vector length 0
Non-movable pages due to conservative pointers = 1, 4096 bytes
Scavenge static space: 2364848 bytes
GC of generation 0 finished:
Generation Boxed Unboxed LB LUB Alloc Waste Trig WP GCs Mem-age
0: 0 0 0 0 0 0 10000000 0 0 0.0000
1: 261 71 97657 0 401324584 38360 411252792 97843 1 3.9995
2: 0 0 0 0 0 0 2000000 0 0 0.0000
3: 0 0 0 0 0 0 2000000 0 0 0.0000
4: 0 0 0 0 0 0 2000000 0 0 0.0000
5: 0 0 0 0 0 0 2000000 0 0 0.0000
Total bytes alloc=401324584
Starting GC of generation 1 with raise=1 alloc=401324584 trig=411252792 GCs=1
*W control stack vector length 0
Non-movable pages due to conservative pointers = 5, 20480 bytes
Scavenge static space: 2364848 bytes
** copy_large_object: 400000008
GC of generation 1 finished:
Generation Boxed Unboxed LB LUB Alloc Waste Trig WP GCs Mem-age
0: 0 0 0 0 0 0 10000000 0 0 0.0000
1: 0 0 0 0 0 0 10000000 0 0 0.0000
2: 228 71 97657 0 401214056 13720 2000000 0 0 0.0000
3: 0 0 0 0 0 0 2000000 0 0 0.0000
4: 0 0 0 0 0 0 2000000 0 0 0.0000
5: 0 0 0 0 0 0 2000000 0 0 0.0000
Total bytes alloc=401214056
Starting GC of generation 2 with raise=1 alloc=401214056 trig=2000000 GCs=0
*W control stack vector length 0
Non-movable pages due to conservative pointers = 5, 20480 bytes
Scavenge static space: 2364848 bytes
** copy_large_object: 400000008
GC of generation 2 finished:
Generation Boxed Unboxed LB LUB Alloc Waste Trig WP GCs Mem-age
0: 0 0 0 0 0 0 10000000 0 0 0.0000
1: 0 0 0 0 0 0 10000000 0 0 0.0000
2: 0 0 0 0 0 0 10000000 0 0 0.0000
3: 228 71 97657 0 401214056 13720 2000000 0 0 0.0000
4: 0 0 0 0 0 0 2000000 0 0 0.0000
5: 0 0 0 0 0 0 2000000 0 0 0.0000
Total bytes alloc=401214056
Starting GC of generation 3 with raise=1 alloc=401214056 trig=2000000 GCs=0
*W control stack vector length 0
Non-movable pages due to conservative pointers = 5, 20480 bytes
Scavenge static space: 2364848 bytes
** copy_large_object: 400000008
GC of generation 3 finished:
Generation Boxed Unboxed LB LUB Alloc Waste Trig WP GCs Mem-age
0: 0 0 0 0 0 0 10000000 0 0 0.0000
1: 0 0 0 0 0 0 10000000 0 0 0.0000
2: 0 0 0 0 0 0 10000000 0 0 0.0000
3: 0 0 0 0 0 0 10000000 0 0 0.0000
4: 228 71 97657 0 401214056 13720 2000000 0 0 0.0000
5: 0 0 0 0 0 0 2000000 0 0 0.0000
Total bytes alloc=401214056
Starting GC of generation 4 with raise=1 alloc=401214056 trig=2000000 GCs=0
*W control stack vector length 0
Non-movable pages due to conservative pointers = 5, 20480 bytes
Scavenge static space: 2364848 bytes
** copy_large_object: 400000008
GC of generation 4 finished:
Generation Boxed Unboxed LB LUB Alloc Waste Trig WP GCs Mem-age
0: 0 0 0 0 0 0 10000000 0 0 0.0000
1: 0 0 0 0 0 0 10000000 0 0 0.0000
2: 0 0 0 0 0 0 10000000 0 0 0.0000
3: 0 0 0 0 0 0 10000000 0 0 0.0000
4: 0 0 0 0 0 0 10000000 0 0 0.0000
5: 228 71 97657 0 401214056 13720 2000000 0 0 0.0000
Total bytes alloc=401214056
Starting GC of generation 5 with raise=0 alloc=401214056 trig=2000000 GCs=0
*W control stack vector length 0
Non-movable pages due to conservative pointers = 5, 20480 bytes
Scavenge static space: 2364848 bytes
** copy_large_object: 400000008
GC of generation 5 finished:
Generation Boxed Unboxed LB LUB Alloc Waste Trig WP GCs Mem-age
0: 0 0 0 0 0 0 10000000 0 0 0.0000
1: 0 0 0 0 0 0 10000000 0 0 0.0000
2: 0 0 0 0 0 0 10000000 0 0 0.0000
3: 0 0 0 0 0 0 10000000 0 0 0.0000
4: 0 0 0 0 0 0 10000000 0 0 0.0000
5: 228 71 97657 0 401214056 13720 411214056 0 1 0.0000
Total bytes alloc=401214056
Write protected 97885 of 97956 pages in generation 5.
--
Eric Marsden <URL:http://www.laas.fr/~emarsden/>
From: Joe Marshall
Subject: Re: Reclaiming space with CMUCL
Date:
Message-ID: <wui3yeh3.fsf@ccs.neu.edu>
Eric Marsden <········@laas.fr> writes:
> >>>>> "aw" == Adam Warner <······@consulting.net.nz> writes:
>
> aw> I'm trying to understand how to reclaim space in CMUCL after
> aw> creating a huge array. Can anyone point out what's wrong here:
> aw>
> aw> * (progn (defparameter a (make-array 100000000)) nil)
>
> aw> * (setf a nil)
> aw> * (gc :full t)
> aw> * (room)
> aw>
> aw> Dynamic Space Usage: 400,073,800 bytes.
> aw> Read-Only Space Usage: 19,419,824 bytes.
> aw> Static Space Usage: 2,686,400 bytes.
> aw> Control Stack Usage: 548 bytes.
> aw> Binding Stack Usage: 96 bytes.
>
> you're running into a limitation of the conservative garbage collector
> used on x86 platforms (this doesn't happen on the SPARC/Solaris port
> of CMUCL, for instance). The problem is that you haven't specified an
> initial element for the array, so its elements aren't initialized, and
> are full of random junk. This confuses the garbage collector, since
> some of the junk looks like it might be a reference to a valid memory
> location. The array therefore potentially contains live objects, and
> isn't collected.
It wouldn't matter if the array referred to live objects if nothing
referred to the array. It would be far more likely with an array of
this size that some pattern of bits somewhere in memory contains the
address of part of the array. It is even *more* likely that memory
management special cases `large' objects.
> You can see that the array is being handled by special
> "large-object" code.
Apparently that is what is happening.
Eric Marsden <········@laas.fr> wrote in message news:<···············@melbourne.laas.fr>...
> you're running into a limitation of the conservative garbage collector
> used on x86 platforms (this doesn't happen on the SPARC/Solaris port
> of CMUCL, for instance). The problem is that you haven't specified an
> initial element for the array, so its elements aren't initialized, and
> are full of random junk. This confuses the garbage collector, since
> some of the junk looks like it might be a reference to a valid memory
> location. The array therefore potentially contains live objects, and
> isn't collected.
What? The conditions for collecting an object is that no no objects
point to it, not that it doesn't point to other objects! Some of the
garbage might look like the address of the array itself, but in that
case that's a circular reference that shouldn't stand in the way of
collection.
If this is in fact the problem, surely it must be that there is some
other junk-containing object that contains a false reference to Adam's
array.
In article <···············@melbourne.laas.fr>,
Eric Marsden <········@laas.fr> wrote:
>you're running into a limitation of the conservative garbage collector
>used on x86 platforms (this doesn't happen on the SPARC/Solaris port
>of CMUCL, for instance). The problem is that you haven't specified an
>initial element for the array, so its elements aren't initialized, and
>are full of random junk. This confuses the garbage collector, since
>some of the junk looks like it might be a reference to a valid memory
>location. The array therefore potentially contains live objects, and
>isn't collected.
Huh? What about the part of his code where he looped through the array,
setting every element to "some string"?
In any case, a GC (conservative or not) doesn't care what the huge array
references. What matters is whether any other objects reference it.
--
Barry Margolin, ··············@level3.com
Genuity Managed Services, a Level(3) Company, Woburn, MA
*** DON'T SEND TECHNICAL QUESTIONS DIRECTLY TO ME, post them to newsgroups.
Please DON'T copy followups to me -- I'll assume it wasn't posted to the group.
Hi Eric Marsden,
> you're running into a limitation of the conservative garbage collector
> used on x86 platforms (this doesn't happen on the SPARC/Solaris port of
> CMUCL, for instance). The problem is that you haven't specified an
> initial element for the array, so its elements aren't initialized, and
> are full of random junk. This confuses the garbage collector, since some
> of the junk looks like it might be a reference to a valid memory
> location. The array therefore potentially contains live objects, and
> isn't collected.
>
> If you use :initial-element, or declare the element-type, or fill all
> the positions in the array during program execution, you won't run into
> this problem.
Thanks for the explanation Eric! It's a great lesson to remember for my
coding. But I'm working on a safe arbitrary code execution environment and
discovering that I can't clean up certain memory usage is unfortunate. Not
a huge deal though because I'll be killing the image every time
computations take too long (it is my understanding that CMUCL has
cooperative-style multitasking and I can't achieve this without killing
the Unix process or merely solving the Halting Problem).
Regards,
Adam
"Adam Warner" <······@consulting.net.nz> writes:
> Thanks for the explanation Eric! It's a great lesson to remember for my
> coding. But I'm working on a safe arbitrary code execution environment and
> discovering that I can't clean up certain memory usage is unfortunate. Not
> a huge deal though because I'll be killing the image every time
> computations take too long (it is my understanding that CMUCL has
> cooperative-style multitasking and I can't achieve this without killing
> the Unix process or merely solving the Halting Problem).
I think another possibility is to use a different gc (there is at
least one more in the sources). This probably involves recompiling,
though.
Mario.
Mario S. Mommer <········@yahoo.com> writes:
> "Adam Warner" <······@consulting.net.nz> writes:
>> Thanks for the explanation Eric! It's a great lesson to remember for my
>> coding. But I'm working on a safe arbitrary code execution environment and
>> discovering that I can't clean up certain memory usage is unfortunate. Not
>> a huge deal though because I'll be killing the image every time
>> computations take too long (it is my understanding that CMUCL has
>> cooperative-style multitasking and I can't achieve this without killing
>> the Unix process or merely solving the Halting Problem).
>
> I think another possibility is to use a different gc (there is at
> least one more in the sources). This probably involves recompiling,
> though.
The only other GC in the sources that'll work on x86 is CGC, which is
also conservative. That said, I don't think this is exactly the
problem anyway. SBCL (with essentially the same GENCGC as CMUCL has,
modulo code cleanups) has no trouble reclaiming the memory after (setf
a nil).
[ Plug: Current CVS SBCL also has native threads, on Linux x86, so you
can do (sb-unix::with-timeout 30 (arbitrary-computation)) and you'll
get a TIMEOUT condition signalled _to the correct thread_ after 30s.
This will probably move into a more sensible package before it's
released ]
-dan
--
http://www.cliki.net/ - Link farm for free CL-on-Unix resources
Hi Daniel Barlow,
> The only other GC in the sources that'll work on x86 is CGC, which is
> also conservative. That said, I don't think this is exactly the problem
> anyway. SBCL (with essentially the same GENCGC as CMUCL has, modulo
> code cleanups) has no trouble reclaiming the memory after (setf a nil).
>
> [ Plug: Current CVS SBCL also has native threads, on Linux x86, so you
> can do (sb-unix::with-timeout 30 (arbitrary-computation)) and you'll get
> a TIMEOUT condition signalled _to the correct thread_ after 30s. This
> will probably move into a more sensible package before it's released ]
This is amazing news Daniel. Congratulations! Native threading is finally
available for a free ANSI Common Lisp implementation.
Regards,
Adam
>>>>> "ecm" == Eric Marsden <········@laas.fr> wrote various silly things:
ecm> The problem is that you haven't specified an initial element
ecm> for the array, so its elements aren't initialized, and are full
ecm> of random junk.
in fact, when you start up CMUCL, an array that isn't explicitly
initialized will have its elements default to zero. This is because
the memory where the array is allocated has been reserved using
mmap(), and mmap zeros pages as the application touches them. So in
this case the array shouldn't contain random junk.
ecm> This confuses the garbage collector, since some of the junk
ecm> looks like it might be a reference to a valid memory location.
ecm> The array therefore potentially contains live objects, and
ecm> isn't collected.
this was foolish, as other people pointed out: if the array isn't
reachable from a root (such as a global variable), it should be
collected, even if it references live objects.
Given that this problem doesn't seem to appear when using the
file-level entry point into CMUCL's compiler, I suspect that something
in the form-level entry point to the compiler is causing a problem
for the conservative collector. Executing a form like
(dotimes (i 10) (funcall (compile nil (lambda () t))))
seems to be sufficient to make the array collectable.
ecm> If you use :initial-element, or declare the element-type, or
ecm> fill all the positions in the array during program execution,
ecm> you won't run into this problem.
this seems to be true, though I'm not sure why.
--
Eric Marsden <URL:http://www.laas.fr/~emarsden/>