I often find myself wanting a tool that would allow me to find all the
nongarbage that Lisp is refusing to collect. I have been using
Harlequin Common Lisp lately, and I find that on big problems it often
mushrooms to about 50M even though I'm pretty sure I'm eliminating all
pointers to the large structures my program generates for temporary
use. What I'd like is a tool that will tell me what the 50 MBytes
consists of and why it can't be collected. It seems to me this could be
written in a portable way (starting at all symbols accessible
anywhere). A portable tool wouldn't find the storage that Harlequin has
lost in some special way all its own, but it would allow me to convince
them that it exists (or exonerate them by showing a trail to something I
thought I deleted all the references to).
Does anyone know of any such package?
-- Drew McDermott
In article <················@yale.edu>,
Drew McDermott <··············@yale.edu> wrote:
>I often find myself wanting a tool that would allow me to find all the
>nongarbage that Lisp is refusing to collect.
Pardon me if I'm oversimplying things but....
Is it uncollected garbage or does the heap space not reduced/resized after
expansion?
I can't recall how informative the invocation of
(room t )
is in Lispworks but it presumambly will tell you where this allocation
(if any) is in the "generations". A categorization of the type of stuff may
not be present. There is no standard requirement for the utility of
this function's output.
>use. What I'd like is a tool that will tell me what the 50 MBytes
>consists of and why it can't be collected.
Not sure if applicable but sometimes forgotten....
If the result of some expression is huge and is printed to the
Listener, then the Listener "history" symbols ( * , ** , etc. ) will hold
onto it (directly or indirectly ) for a while even though nothing in
your "program" is refering to the data. Four or five NILs subsequently
entered at the Listener solves that problem, eventually.
> It seems to me this could be
>written in a portable way (starting at all symbols accessible
>anywhere).
I'm not so sure this can be portable... I would imagine that
most garbage collectors are written in terms of implementation
specific primatives. Doing this without stepping on the "toes"
of the garbage collector, implementation internal allocation data
structures, and being portable? That seems like a tall order.
Wrapping a common "interface" to implementation specific code seems
more tractable. Seems like this is something ROOM should/could
optionally report though.
It seems as though "magic" would be required to do a measurement
without preturbing what is being observed. The implementation
specific primatives would provide this.
--
Lyman S. Taylor "I'm a Doctor! Not a commando."
(·····@cc.gatech.edu) The enhanced EMH Doctor in a ST:Voyager epidsode.
Drew McDermott wrote:
>
> I often find myself wanting a tool that would allow me to find all the
> nongarbage that Lisp is refusing to collect. I have been using
I'm not aware of such a tool, but something related which will tell
you if indeed there is some unclaimed "garbage" is to dump out all
your objects into a file, and then in another fresh lisp restore
from the dump and then compare the memory useage.
For ACL, see excl:fasl-write and excl:fasl-read, Harlequin may
have something similiar. the SAVE-OBJECT ascii representation
code is a more portable solution.
-Kelly Murray ···@intellimarket.com
still offering $500 for an unrestricted CL JPEG encode/decode util.
Drew McDermott <··············@yale.edu> writes:
> I often find myself wanting a tool that would allow me to find all the
> nongarbage that Lisp is refusing to collect. I have been using
> Harlequin Common Lisp lately, and I find that on big problems it often
> mushrooms to about 50M even though I'm pretty sure I'm eliminating all
> pointers to the large structures my program generates for temporary
> use. What I'd like is a tool that will tell me what the 50 MBytes
> consists of and why it can't be collected. It seems to me this could be
> written in a portable way (starting at all symbols accessible
> anywhere). A portable tool wouldn't find the storage that Harlequin has
> lost in some special way all its own, but it would allow me to convince
> them that it exists (or exonerate them by showing a trail to something I
> thought I deleted all the references to).
>
> Does anyone know of any such package?
I'll risk a blatant plug by saying that Allegro CL has such a function,
though unexported and undocumented, called excl::get-references, which
accepts a lisp object and returns a vector holding all of the objects in
the heap which point to the object in question. The reason why it is
not exported or documented is because it gets messy fairly quickly; if
you realize that every returned vector is itself a new pointer to the
object in question, a chain of calls to excl::get-references can get
fairly cluttered with vector objects that were not part of the original
heap. Nevertheless, it has been a useful function, if you are willing
to put up with its droppings.
Perhaps Harlequin also has such a function.
--
Duane Rettig Franz Inc. http://www.franz.com/ (www)
1995 University Ave Suite 275 Berkeley, CA 94704
Phone: (510) 548-3600; FAX: (510) 548-8253 ·····@Franz.COM (internet)
Drew McDermott <··············@yale.edu> wrote in message
·····················@yale.edu...
>I often find myself wanting a tool that would allow me to find all the
>nongarbage that Lisp is refusing to collect.
I wrote one of these a few years ago at Lucid. Here is how it worked:
First, it treats the collectable heap as a directed graph. Then it finds
all the `bridges' in the graph, i.e. those pointers that hold a unique
reference to an object. Then, it weighs those pointers by determining
how much total storage would be freed if the pointer didn't exist. Finally,
it produced a report that noted which symbols contained such pointers,
and which datatypes such pointers pointed at. The former would tell
you where the large amounts of data lived, the latter would tell you what
the
large structures were.
It turned out to be quite a useful tool, but I think it has since been lost.
The algorithm for finding bridges in the heap was pretty bad, and I'm
sure someone more versed in graph theory could do better.
~jrm
Drew McDermott <··············@yale.edu> writes:
> I often find myself wanting a tool that would allow me to find all the
> nongarbage that Lisp is refusing to collect. I have been using
> Harlequin Common Lisp lately, and I find that on big problems it often
> mushrooms to about 50M even though I'm pretty sure I'm eliminating all
> pointers to the large structures my program generates for temporary
> use. What I'd like is a tool that will tell me what the 50 MBytes
> consists of and why it can't be collected. It seems to me this could be
> written in a portable way (starting at all symbols accessible
> anywhere). A portable tool wouldn't find the storage that Harlequin has
> lost in some special way all its own, but it would allow me to convince
> them that it exists (or exonerate them by showing a trail to something I
> thought I deleted all the references to).
>
> Does anyone know of any such package?
>
> -- Drew McDermott
In general, once an application allocates RAM from the system, there
is no way for the OS to reclaim that memory until the application's
process is terminated. There are "relocating allocators" that some
applications use that allow them to return unused memory to the
OS, but usage of these is far from pervasive as of yet. For example,
XEmacs allows this compile-time option:
The `--rel-alloc' option can be used to either enable or disable use
of the relocating allocator. Turning on --rel-alloc will allow XEmacs
to return unused memory to the operating system, thereby reducing its
memory footprint. However, it may make XEmacs runs more slowly,
especially if your system's `mmap' implemntation is missing or
inefficient.
Neither ACL5 nor CMUCL have such a capability, nor does (most
unfortunately) Netscape. I would never, ever restart any of these
applications if they were capable of letting the OS reclaim unused
memory, because Netscape can swell to 140MB after a few days and then
it's back down to 25MB after a restart.
Christopher
Christopher R. Barry wrote:
> In general, once an application allocates RAM from the system, there
> is no way for the OS to reclaim that memory until the application's
> process is terminated. There are "relocating allocators" that some
[snip]
>
> Neither ACL5 nor CMUCL have such a capability, nor does (most
> unfortunately) Netscape. I would never, ever restart any of these
> applications if they were capable of letting the OS reclaim unused
> memory, because Netscape can swell to 140MB after a few days and then
I think you are mistaken about CMUCL. I've seen CMUCL grow and shrink
in size as an application runs, at least ps and top say so. Both the
total size and the RSS change.
Ray
--
---------------------------------------------------------------------------
----> Raymond Toy ····@mindspring.com
http://www.mindspring.com/~rtoy
"R. Toy" <····@mindspring.com> writes:
> Christopher R. Barry wrote:
> > In general, once an application allocates RAM from the system, there
> > is no way for the OS to reclaim that memory until the application's
> > process is terminated. There are "relocating allocators" that some
> [snip]
> >
> > Neither ACL5 nor CMUCL have such a capability, nor does (most
> > unfortunately) Netscape. I would never, ever restart any of these
> > applications if they were capable of letting the OS reclaim unused
> > memory, because Netscape can swell to 140MB after a few days and then
>
> I think you are mistaken about CMUCL. I've seen CMUCL grow and shrink
> in size as an application runs, at least ps and top say so. Both the
> total size and the RSS change.
After playing around a little, I can see that it does this a little bit,
but I guess I had never noticed it before because it could be a lot
better. Here are the numbers:
Fresh launch of CMUCL:
SIZE RSS SHARE
____ ___ _____
9928 cbarry 0 0 8108 8108 4360 S 1104 0.0 12.7 0:00 lisp
After doing a (fact 15000):
9928 cbarry 0 0 90988 42M 2616 S 14M 0.0 69.3 0:23 lisp
A global GC after that:
9928 cbarry 0 0 75868 27M 2876 S 0 0.0 44.1 0:23 lisp
And basically, CMUCL will never use less memory than this for the
duration of it's usage. I have a WindowMaker dock app that shows me my
real-time memory and swap usage and after doing that (fact 15000)
there is nothing in the world that is going to make my swap usage back
down to 0 until I shut down and restart CMUCL. CMUCL does grow and
shrink somewhat, but it grows unstoppably over time, as does ACL5 and
most other apps out there that at times use a lot of memory.
Christopher
>>>>> "Christopher" == Christopher R Barry <······@2xtreme.net> writes:
Christopher> After playing around a little, I can see that it does this a little bit,
Christopher> but I guess I had never noticed it before because it could be a lot
Christopher> better. Here are the numbers:
Christopher> Fresh launch of CMUCL:
Christopher> SIZE RSS SHARE
Christopher> ____ ___ _____
Christopher> 9928 cbarry 0 0 8108 8108 4360 S 1104 0.0 12.7 0:00 lisp
Christopher> After doing a (fact 15000):
Christopher> 9928 cbarry 0 0 90988 42M 2616 S 14M 0.0 69.3 0:23 lisp
Christopher> A global GC after that:
Christopher> 9928 cbarry 0 0 75868 27M 2876 S 0 0.0 44.1 0:23 lisp
Christopher> And basically, CMUCL will never use less memory than this for the
Christopher> duration of it's usage. I have a WindowMaker dock app that shows me my
This is interesting. Is this Linux running 18a?
On my sparc, top shows a size and RSS of about 23M and 8.9M at start
up. After running (fact 15000) for a while (I stopped it after
sucking about 100M of memory), the size and RSS had grown to about
100M each. After (gc), the size and RES are 27M and 7480K. This is a
post 18b binary on Sparc. Perhaps the conservative generational GC on
Linux is leaking some memory?
Ray
Raymond Toy <···@rtp.ericsson.se> writes:
[...]
> This is interesting. Is this Linux running 18a?
Debian running Peter's 2.4.8 from December 17th. There is a 2.4.9 now,
but I doubt it's different.
Christopher
* "R. Toy" <····@mindspring.com>
| I think you are mistaken about CMUCL. I've seen CMUCL grow and shrink in
| size as an application runs, at least ps and top say so. Both the total
| size and the RSS change.
this may be misleading. under Linux, the VM size of the Allegro CL
process does not decrease according to /proc/self/status even when the
working set is greatly reduced by using SYSTEM:RESIZE-AREAS. I was a
little surprised by this, since ps and top and other tools report a
decrease. I guess it's time to trust the kernel and not various "smart"
tools that try to provide user-friendly information.
#:Erik
Erik Naggum wrote:
>
> * "R. Toy" <····@mindspring.com>
> | I think you are mistaken about CMUCL. I've seen CMUCL grow and shrink in
> | size as an application runs, at least ps and top say so. Both the total
> | size and the RSS change.
>
> this may be misleading. under Linux, the VM size of the Allegro CL
> process does not decrease according to /proc/self/status even when the
> working set is greatly reduced by using SYSTEM:RESIZE-AREAS. I was a
> little surprised by this, since ps and top and other tools report a
> decrease. I guess it's time to trust the kernel and not various "smart"
> tools that try to provide user-friendly information.
Hmm, I wasn't aware of this "feature" of Linux. However, I think the
same (decreased memory usage according to ps and top) occurs on Solaris
for CMUCL.
Maybe Solaris version of ps and top has the same "feature"?
Ray
--
---------------------------------------------------------------------------
----> Raymond Toy ····@mindspring.com
http://www.mindspring.com/~rtoy
Erik Naggum <····@naggum.no> writes:
> this may be misleading. under Linux, the VM size of the Allegro CL
> process does not decrease according to /proc/self/status even when the
> working set is greatly reduced by using SYSTEM:RESIZE-AREAS.
Note that the Linux VmSize measures all mapped address space
regardless of whether a page exists to back it. One trick in Linux to
avoid races with other pieces of code that are allocating address
space is to pre-map a VM arena and allocate out of that. Deallocation
consists of creating a new mapping on top of the address space you
want to free up. That way, other code doing non-fixed mappings can't
get its mitts on your address space in between figuring out where to
put it and actually grabbing it. The unused parts of the arena jack
up VmSize but doesn't use any resources other than page table entries.
I don't know if Allegro does that or something like it, but it shows
that VmSize is not a good measure of "memory" usage.
--David Gadbois
* David Gadbois <·······@lagavulin.cyc.com>
| I don't know if Allegro does that or something like it, but it shows
| that VmSize is not a good measure of "memory" usage.
thanks. now I don't know what would be a good measure, anymore. do you
have any suggestions?
#:Erik
Erik Naggum <····@naggum.no> writes:
> * David Gadbois <·······@lagavulin.cyc.com>
> | I don't know if Allegro does that or something like it, but it shows
> | that VmSize is not a good measure of "memory" usage.
>
> thanks. now I don't know what would be a good measure, anymore. do you
> have any suggestions?
Q: How do you tell if a toadstool is poisonous?
A: Eat it; if you died, it was poisonous.
Sorry for the tasteless joke, but it does illustrate one way to
measure the measuring tools. Some of our customers that need
multi-hundred-megabyte processes and which do not trust the tools
that measure memory usage check these tools out by pushing the
limits of their systems. Which tools to trust starts to become
clear as more experience with the failures (actually the
_exact_ _point_ of the failures) is seen. Sometimes the point-of-failure
experiments involve only swap space, so the experiments are relatively
easy. Other times, the experiments involve virtual address availability,
which is sometimes much harder unless your system is beefed up with
ram and/or swap on the order of gigabytes.
--
Duane Rettig Franz Inc. http://www.franz.com/ (www)
1995 University Ave Suite 275 Berkeley, CA 94704
Phone: (510) 548-3600; FAX: (510) 548-8253 ·····@Franz.COM (internet)
Erik Naggum <····@naggum.no> writes:
> * David Gadbois <·······@lagavulin.cyc.com>
> | I don't know if Allegro does that or something like it, but it
> | shows that VmSize is not a good measure of "memory" usage.
>
> thanks. now I don't know what would be a good measure, anymore.
> do you have any suggestions?
Alas, there does not seem to be a general way of doing this in Linux.
I use ROOM and Lisp implementation-specific elaborations thereof.
One kernel hacking project I have been thinking about is adding some
way of getting information on what pages have been dirtied at some
time since their initialization. Such a facility, with a suitable
procfs interface, would be just the ticket for get a better handle on
memory usage. Even better, it would make for a really cheap write
barrier for a generational garbage collector: The VM hardware already
sets dirty bits, and the kernel just has to save the info somewhere
when it clears the bits. It would be much superior to other write
barrier implementations like trapping SIGSEGVs, checking address
bounds, or blinding marking a card.
Anyone have any advice on how to go about such a project?
--David Gadbois
>* David Gadbois <·······@lagavulin.cyc.com>
>| I don't know if Allegro does that or something like it, but it shows
>| that VmSize is not a good measure of "memory" usage.
>
> thanks. now I don't know what would be a good measure, anymore. do you
> have any suggestions?
When you start hearing the disk drive whir, you've allocated too much.
-- Harley
PS Just another reason why thin clients are a bad idea!
* "Harley Davis" <··············@spamless_ilog.com>
| When you start hearing the disk drive whir, you've allocated too much.
well, with 512M RAM, no swap space, and the machine in a separate room,
if I start hearing the disks, I may well be in serious trouble, but it
won't be because of memory allocation.
#:Erik
Date: 06 Jan 1999 08:12:07 +0000
From: Erik Naggum <····@naggum.no>
* "Harley Davis" <··············@spamless_ilog.com>
| When you start hearing the disk drive whir, you've allocated too much.
well, with 512M RAM, no swap space, and the machine in a separate room,
if I start hearing the disks, I may well be in serious trouble, but it
won't be because of memory allocation.
I believe I remember someone hacking something up on a TI Explorer that
would play disk-like sounds through the speaker. He'd just moved to a
site that kept all the system units in a machine room. Run lights are
nice, but that audible feedback really got your attention.
Paul Fuqua
Texas Instruments, Dallas, Texas ··@hc.ti.com
[off-topic alert...]
Paul Fuqua <··@elissa.hc.ti.com> wrote:
+---------------
| I believe I remember someone hacking something up on a TI Explorer that
| would play disk-like sounds through the speaker. He'd just moved to a
| site that kept all the system units in a machine room. Run lights are
| nice, but that audible feedback really got your attention.
+---------------
Circa 1965, I connected a speaker to the "Zero Balance" light[*] on an
IBM 1410, which let us hear the non-linear regressions converging towards
a solution -- the "chomping" noises got faster and faster as it converged.
If it was still taking ~10 seconds or so per "chomp", you knew you had
plenty of time to go get a cup of coffee and still get back before it was
done. If they were coming about a second apart, you knew it was time to go
over and put the printer on-line, since it was about done. [We left the
printer off-line when not actually in use because IBM charged us for service
based on how long each piece of peripheral equipment was on-line!!! No joke!]
This was actually quite useful, since you often didn't know until you had
run one of those things for a while whether it was going to run for only
ten minutes or for ten hours (or more)!!
After the speaker had been connected for a while, the machine room operators
discovered that they too could hear "how things were going", and know whether
it was safe to sneak down the hall to take a nap. In particular, if the card
reader went empty or something was ready to print and the operating system
started busy-waiting on the device, the speaker would let out a *loud* constant
"HOOONNNNNNK!" sound... which got 'em on their feet in a hurry (especially
since that particular sound penetrated up through the ceiling to the manager's
office, and if it went on too long he knew it meant somebody was goofing off!).
-Rob
[*] Obligatory Lisp content: On a 1410, the "Zero Balance" light
came on whenever (zerop *most-recent-arithmetic-result-or-comparison*).
-----
Rob Warnock, 8L-855 ····@sgi.com
Applied Networking http://reality.sgi.com/rpw3/
Silicon Graphics, Inc. Phone: 650-933-1673
2011 N. Shoreline Blvd. FAX: 650-964-0811
Mountain View, CA 94043 PP-ASEL-IA
From: ········@totient.demon.co.uk
Subject: The Sound Of Computation (Was Re: Garbage hunting)
Date:
Message-ID: <77222k$f17$1@nnrp1.dejanews.com>
In article <············@fido.engr.sgi.com>,
····@rigden.engr.sgi.com (Rob Warnock) wrote:
> [off-topic alert...]
> Circa 1965, I connected a speaker to the "Zero Balance" light[*] on an
> IBM 1410, which let us hear the non-linear regressions converging towards
(with-geek-mode-on Perhaps we should integrate Common Lisp Music and
Screamer to produce sounds as it non-deterministically searches and
backtracks for solutions.....
We could have The Sound of The Travelling Salesman Problem or maybe a
strange cover version "(I can't get no) Constraint Satisfaction" for those NP
problems. )
;-)
Jon
-----------== Posted via Deja News, The Discussion Network ==----------
http://www.dejanews.com/ Search, Read, Discuss, or Start Your Own
······@2xtreme.net (Christopher R. Barry) writes:
> Drew McDermott <··············@yale.edu> writes:
>
> > I often find myself wanting a tool that would allow me to find all the
> > nongarbage that Lisp is refusing to collect. I have been using
> > Harlequin Common Lisp lately, and I find that on big problems it often
> > mushrooms to about 50M even though I'm pretty sure I'm eliminating all
> > pointers to the large structures my program generates for temporary
> > use. What I'd like is a tool that will tell me what the 50 MBytes
> > consists of and why it can't be collected. It seems to me this could be
> > written in a portable way (starting at all symbols accessible
> > anywhere). A portable tool wouldn't find the storage that Harlequin has
> > lost in some special way all its own, but it would allow me to convince
> > them that it exists (or exonerate them by showing a trail to something I
> > thought I deleted all the references to).
> >
> > Does anyone know of any such package?
> >
> > -- Drew McDermott
>
> In general, once an application allocates RAM from the system, there
> is no way for the OS to reclaim that memory until the application's
> process is terminated.
I think that the original requestor is more concerned about individual
objects than about blocks of memory, but at any rate your statement is
not necessarily true; it depends on how the memory was allocated
originally. It also depends on the operating system as to how
mmap/munmap (and VirtualAlloc/VirtualFree on Windows) will respond to
such requests, and if there are no bugs in these allocation functions,
as the paragraph you supplied below implies:
> There are "relocating allocators" that some
> applications use that allow them to return unused memory to the
> OS, but usage of these is far from pervasive as of yet. For example,
> XEmacs allows this compile-time option:
>
> The `--rel-alloc' option can be used to either enable or disable use
> of the relocating allocator. Turning on --rel-alloc will allow XEmacs
> to return unused memory to the operating system, thereby reducing its
> memory footprint. However, it may make XEmacs runs more slowly,
> especially if your system's `mmap' implemntation is missing or
> inefficient.
>
> Neither ACL5 nor CMUCL have such a capability, nor does (most
> unfortunately) Netscape. I would never, ever restart any of these
> applications if they were capable of letting the OS reclaim unused
> memory, because Netscape can swell to 140MB after a few days and then
> it's back down to 25MB after a restart.
ACL5 certainly does have this capability. The sys:resize-areas function
can reduce the virtual memory utilization, as long as there are no
"Gaps" in the heap (sections of memory not owned by the lisp allocator).
I did notice when I ran an experiment on ACL5 on NT that even after the
memory was relinquished by the lisp, the operating system didn't actually
take it back until it needed the physical pages (i.e. I started some other
programs that would require virtual memory allocation). At that point,
Task manager did record the shedding of the pages, and even a global gc
did not bring them back, though it touches most of the allocated pages.
--
Duane Rettig Franz Inc. http://www.franz.com/ (www)
1995 University Ave Suite 275 Berkeley, CA 94704
Phone: (510) 548-3600; FAX: (510) 548-8253 ·····@Franz.COM (internet)
> In response to my query about a tool for finding Lisp garbage,
> Christopher R. Barry wrote:
> In general, once an application allocates RAM from the system, there
> is no way for the OS to reclaim that memory until the application's
> process is terminated. ...
Perhaps I should be a bit clearer. I am willing to allow Lisp to keep the
memory indefinitely. However, what I observe is that it seems to expand and
then *lose* the memory in question. In other words, suppose I run program 1,
kill Lisp, and then run program 2, and observe that in each case Lisp expands
to 50 Mb. Now if I run program 1 and then program 2 without killing Lisp, it
grows to 50Mb after program 1, and then *keeps growing* to accommodate program
2. I have 128Mb of primary memory on this computer, so the system runs just
fine during program 1, but then starts thrashing during program 2, runs slower
and slower, and eventually the Lisp dies completely, occasionally bringing
down the entire computer, which has to be restarted.
So: I would really like to know what's hanging around after program 1
finishes, especially since I go out of my way to destroy every global
reference I can find to the huge data structures created during its execution.
-- Drew McDermott
Drew McDermott <··············@yale.edu> writes:
> Perhaps I should be a bit clearer. I am willing to allow Lisp to keep the
> memory indefinitely. However, what I observe is that it seems to expand and
> then *lose* the memory in question. In other words, suppose I run program 1,
> kill Lisp, and then run program 2, and observe that in each case Lisp expands
> to 50 Mb. Now if I run program 1 and then program 2 without killing Lisp, it
> grows to 50Mb after program 1, and then *keeps growing* to accommodate program
> 2. I have 128Mb of primary memory on this computer, so the system runs just
> fine during program 1, but then starts thrashing during program 2, runs slower
> and slower, and eventually the Lisp dies completely, occasionally bringing
> down the entire computer, which has to be restarted.
>
> So: I would really like to know what's hanging around after program 1
> finishes, especially since I go out of my way to destroy every global
> reference I can find to the huge data structures created during its execution.
>
> -- Drew McDermott
AFAIK, things that will hang around:
1. Code. This turned out to be a fairly non-trivial problem in an old
rule-based system I was working with. Each time I compiled the rule file,
each rule would expand into I think four new functions.
2. Symbols.
3. Constants?
One thing worth trying might be to load up program 1, check it's usage, run
program 1, and check it's usage again, just to get a feel for how much of
the heap program 1 has eaten up. If after releasing all your allocated
storage and manually invoking a GC you end up with higher usage, you really
do have a problem. You might consider getting in touch with Harlequin
support, they have been very helpful to me.
Sunil
>>>>> "sm" == Sunil Mishra <·······@whizzy.cc.gatech.edu> writes:
> AFAIK, things that will hang around:
> 1. Code. This turned out to be a fairly non-trivial problem in an
> old rule-based system I was working with. Each time I compiled the
> rule file, each rule would expand into I think four new functions.
Code, like other stuff, should be collected when it is no longer
referenced. Most often, code is referenced (through function objects)
from the function cell of a symbol. When you (SETF SYMBOL-FUNCTION)
to a different value or use FMAKUNBOUND or get rid of the symbol (see
below), the code can be garbage collected. If it isn't (and the only
reference had really been from that symbol), than that may be a bug of
your Lisp system. Some systems may also have parts of the code in
"read-only" space on the assumption that it won't ever be modified or
garbage collected, but that shouldn't be the case for normally loaded
user code.
> 2. Symbols.
Symbols are usually referenced from packages (although they can be
referenced by many many other things of course). If you unintern a
symbol, or remove the package, the symbol should be garbage collected
unless other references exist.
> 3. Constants?
It's not clear to me what you mean by that. Constants as defined by
DEFCONSTANT should go away when the corresponding symbol goes away.
Constant objects such as quoted structure or strings should be
garbage-collectible unless they have been allocated in a "read-only"
area as mentioned below.
--
Simon Leinen ·····@babar.switch.ch
SWITCH http://www.switch.ch/misc/leinen/
Who is General Failure & why's he reading my disk?
Simon Leinen <·····@limmat.switch.ch> writes:
> >>>>> "sm" == Sunil Mishra <·······@whizzy.cc.gatech.edu> writes:
> > AFAIK, things that will hang around:
>
> > 1. Code. This turned out to be a fairly non-trivial problem in an
> > old rule-based system I was working with. Each time I compiled the
> > rule file, each rule would expand into I think four new functions.
>
> Code, like other stuff, should be collected when it is no longer
> referenced. Most often, code is referenced (through function objects)
> from the function cell of a symbol. When you (SETF SYMBOL-FUNCTION)
> to a different value or use FMAKUNBOUND or get rid of the symbol (see
> below), the code can be garbage collected. If it isn't (and the only
> reference had really been from that symbol), than that may be a bug of
> your Lisp system. Some systems may also have parts of the code in
> "read-only" space on the assumption that it won't ever be modified or
> garbage collected, but that shouldn't be the case for normally loaded
> user code.
Yes, it was a serious problem with the system, and was eventually fixed as
it became obvious that the situation was untenable. (The advocated method
of development at that point was to develop rules interpreted, in which
case the patterns weren't compiled into Lisp, and then to compile when
things were sufficiently debugged.)
> > 2. Symbols.
>
> Symbols are usually referenced from packages (although they can be
> referenced by many many other things of course). If you unintern a
> symbol, or remove the package, the symbol should be garbage collected
> unless other references exist.
Thanks for the clarification, I thought this was what ought to happen, but
I was never sure. Drew's problem though was that program 1 (and the package
it was in) hung around. It was not clear if he *needed* the package for
program 2, and if not killing the package itself might be a good way out
for him.
>
> > 3. Constants?
>
> It's not clear to me what you mean by that. Constants as defined by
> DEFCONSTANT should go away when the corresponding symbol goes away.
> Constant objects such as quoted structure or strings should be
> garbage-collectible unless they have been allocated in a "read-only"
> area as mentioned below.
You interpreted the statement as I had intended.
The symbol will not necessarily go away, for the reasons in #2, and if the
constant points to something like a hash table or other structure, then
ensuring it is empty might take a little effort. The reason for the "?" is
that I would *never* attach a data structure like that to a constant, but
I've seen stranger things happen.
Sunil
Drew McDermott <··············@yale.edu> writes:
> > In response to my query about a tool for finding Lisp garbage,
>
> > Christopher R. Barry wrote:
>
> > In general, once an application allocates RAM from the system, there
> > is no way for the OS to reclaim that memory until the application's
> > process is terminated. ...
>
> Perhaps I should be a bit clearer. I am willing to allow Lisp to keep the
> memory indefinitely. However, what I observe is that it seems to expand and
> then *lose* the memory in question. In other words, suppose I run program 1,
> kill Lisp, and then run program 2, and observe that in each case Lisp expands
> to 50 Mb. Now if I run program 1 and then program 2 without killing Lisp, it
> grows to 50Mb after program 1, and then *keeps growing* to accommodate program
> 2. I have 128Mb of primary memory on this computer, so the system runs just
> fine during program 1, but then starts thrashing during program 2, runs slower
> and slower, and eventually the Lisp dies completely, occasionally bringing
> down the entire computer, which has to be restarted.
>
> So: I would really like to know what's hanging around after program 1
> finishes, especially since I go out of my way to destroy every global
> reference I can find to the huge data structures created during its execution.
Heh, well, I do remember hearing some figure from Microsoft that
Windows NT 5.0 will fix something like 160 memory leaks and 200
crashing bugs in Windows NT 4.0. :-)
Christopher
In article <··············@2xtreme.net>
······@2xtreme.net (Christopher R. Barry) writes:
>Neither ACL5 nor CMUCL have such a capability, nor does (most
>unfortunately) Netscape. I would never, ever restart any of these
>applications if they were capable of letting the OS reclaim unused
>memory, because Netscape can swell to 140MB after a few days and then
>it's back down to 25MB after a restart.
The problem with Netscape in particular is not that it cannot give
back memory that was once allocated due and is no more due to the
system's design but because it is horribly buggy and infested with
memory leaks. Garbage collection would certainly pay here. :)
On many systems, the malloc(3) memory allocation facility, a C library
function which is usually used by applications, X Window and X toolkits
(XtAlloc), has the ability to return memory to the system by lowering
the brk (traditionally the top of a process' data segment under Unix,
without mmapped stuff), if a contiguous block just under the brk is free,
and indeed they do so often (examples are the malloc implementations of
Digital Unix, Free/OpenBSD, Gnu/Linux and probably others).
It is true however, that traditional Unix malloc implementations have
not been able to do that (both for speed and pseudo-philosophical reasons
like "a program should always be able to get memory it once had
allocated" etc.)
With ACL5 or CMUCL I would suspect that they do not use malloc at the
lower levels but have their own brk-pusher under the hood, which may
or may not implement lowering the brk if there's enough room free at
the top of the heap.
--
Matthias K. Buelow * Long live the Thing King! *
On 06 Jan 1999 05:55:23 +0100, Matthias Buelow wrote:
>With ACL5 or CMUCL I would suspect that they do not use malloc at the
>lower levels but have their own brk-pusher under the hood, which may
>or may not implement lowering the brk if there's enough room free at
>the top of the heap.
CMUCL at least uses mmap to get memory and to load the core file, and we
use munmap to release it, for example after doing a GC we free the
oldspace. This is clearly visible if you use xosview to watch the memory
consumption.
Groetjes, Peter
--
It's logic Jim, but not as we know it. | ········@debian.org for pleasure,
"God, root, what is difference?",Pitr | ········@inthan.be for more pleasure!
In article <···············@utopia.informatik.uni-wuerzburg.de>,
Matthias Buelow <·····@cip.informatik.uni-wuerzburg.de> wrote:
>It is true however, that traditional Unix malloc implementations have
>not been able to do that (both for speed and pseudo-philosophical reasons
>like "a program should always be able to get memory it once had
>allocated" etc.)
In fact, I believe that early free() man pages specifically said that you
could safely access the freed memory until the next call to malloc() or
realloc().
>With ACL5 or CMUCL I would suspect that they do not use malloc at the
>lower levels but have their own brk-pusher under the hood, which may
>or may not implement lowering the brk if there's enough room free at
>the top of the heap.
Actually, I suspect that Unix Lisp implementations that are able to return
memory don't use brk() or malloc() at all to implement Lisp heap. They
probably use mmap() with /dev/zero or shmget(), as these allow random
chunks of address space to be allocated and freed. They work especially
well with generational copying GC's -- when you move everything from
oldspace to copyspace, you can then release the memory segment that
contained oldspace. The brk() mechanism is especially *unsuited* for this,
since it will be pretty rare that the space you've just emptied happened to
be the one adjacent to the break.
--
Barry Margolin, ······@bbnplanet.com
GTE Internetworking, Powered by BBN, Burlington, MA
*** DON'T SEND TECHNICAL QUESTIONS DIRECTLY TO ME, post them to newsgroups.
Don't bother cc'ing followups to me.
In article <················@yale.edu>, Drew McDermott <··············@yale.edu> writes:
>I often find myself wanting a tool that would allow me to find all the
>nongarbage that Lisp is refusing to collect. I have been using
>Harlequin Common Lisp lately, and I find that on big problems it often
>mushrooms to about 50M even though I'm pretty sure I'm eliminating all
>pointers to the large structures my program generates for temporary
>use. What I'd like is a tool that will tell me what the 50 MBytes
>consists of and why it can't be collected. It seems to me this could be
>written in a portable way (starting at all symbols accessible
>anywhere). A portable tool wouldn't find the storage that Harlequin has
>lost in some special way all its own, but it would allow me to convince
>them that it exists (or exonerate them by showing a trail to something I
>thought I deleted all the references to).
>
>Does anyone know of any such package?
I think that there is a slight semantic problem here. If there is any
nongarbage that HCL is refusing to collect so it can be reused then
there is definitely a bug in the collector. This does not mean that
HCL returns all free memory, from its point of view, to the OS.
Depending on what type of collector it is the free memory might be in
chunks which it cannot return to the OS, or it might feel that it
might as well hang on to it seeing it will probably want to use it
again.
However a tool which shows how the memory is being used can be very
useful, especially when debugging the collector. :-)
--
Robert Virding Tel: +46 (0)8 719 95 28
Computer Science Laboratory Email: ··@erix.ericsson.se
Ericsson Telecom AB WWW: http://www.ericsson.se/cslab/~rv
S-126 25 �LVSJ�, SWEDEN OSE: http://www.erlang.org/
"Folk s�ger att jag inte bryr mig om n�gonting, men det skiter jag i".
Drew McDermott wrote:
> I often find myself wanting a tool that would allow me to find all the
> nongarbage that Lisp is refusing to collect. I have been using
> Harlequin Common Lisp lately, and I find that on big problems it often
> mushrooms to about 50M even though I'm pretty sure I'm eliminating all
> pointers to the large structures my program generates for temporary
> use. What I'd like is a tool that will tell me what the 50 MBytes
> consists of and why it can't be collected.
Let's tackle the first question first: what makes up this 50MB bloat?
You don't say what release / platform you're working on here. If you're on
Windows, you can try this:
(require "heap-analysis")
(delivery::analyse-heap)
[Note: the above is for LWW 4.1; in LWW 4.0.x the symbol was
mm::analyse-heap.]
By default, ANALYSE-HEAP counts every object in the image, performs a
garbage collection (mark-and-sweep 2), and then repeats the count,
displaying the numbers for both count. It displays numbers of object
(nbefore, nafter) and totals in bytes (tbefore, tafter), broken down by
type. Example follows.
That should tell you what the 50Mb consists of. Why it cannot be collected
is a much more difficult question and there is no generic solution. If you
mail the output of ANALYSE-HEAP to ············@harlequin.com, they probably
can probably give you some advice.
- nick
CL-USER 31 > (DV::ANALYSE-HEAP)
heap by subtag
--------------
Subtag Totals nbefore nafter tbefore tafter
CONS 453997 436420 3631976 3491360
ENTRY-TABLE 7 7 403200 403200
STACK-GROUP-HEADER 7 7 1568 1568
STACK-GROUP-BODY 7 7 449528 449528
INSTANCE 9657 9105 401080 387072
ARRAY 33050 31191 3632528 2103568
BASE-CHAR 20442 20362 2109112 887792
SIMPLE-CHAR 2385 2342 344464 332304
CHARACTER 3 3 1608 1608
(UNSIGNED-BYTE 1) 28 17 34056 33304
(UNSIGNED-BYTE 8) 478 478 35176 35176
(UNSIGNED-BYTE 16) 18 18 18864 18864
(UNSIGNED-BYTE 32) 3 3 432 432
(SIGNED-BYTE 32) 2 2 6000 6000
T 9691 7966 1082816 788088
SYMBOL 51124 51118 2498952 2498704
BIGNUM 749 725 15744 15288
RATIO 60 60 960 960
DOUBLE-FLOAT 10247 10235 163952 163760
COMPLEX 13 13 208 208
COMPILED-CODE 30133 29478 8766088 8678128
CFO closure 746 745 204872 204720
FUNCALLABLE-INSTANCE 4783 4783 153056 153056
FIXED-INSTANCE 24320 24290 389120 388640
RECORD 110006 104902 3559352 3433384
DISPATCHER 4838 4838 77408 77408
CLOSURE 22687 18677 544488 448248
INTERPRETED-FUNCTION 13 13 416 416
NIL
CL-USER 32 >
I have some code, that runs on Win/Harlequin, but explodes on
'Nix <Solaris/HP-UX> because it can't handle what it thinks is
a string:
*** - MAKE-STRING-INPUT-STREAM: argument #<END OF FILE> is not a string
How can I make clisp IGNORE this?:
(defun processFile (filename)
(let ( (cmdlist '())
(cmd '()))
(with-open-file (line-stream filename :direction :input )
(do ((line (read-line line-stream nil )
(read-line line-stream nil)))
((not line) (return (processCommandList (reverse cmdlist))))
(setf cmd (getListFromLine line))
(if (not (eq cmd ()))(setf cmdlist (cons cmd cmdlist)))
)
)
)
)
(thanks)
In article <·······································@polaris.umuc.edu>,
Sanchita Nag <····@polaris.umuc.edu> wrote:
> I have some code, that runs on Win/Harlequin, but explodes on
> 'Nix <Solaris/HP-UX> because it can't handle what it thinks is
> a string:
>
> *** - MAKE-STRING-INPUT-STREAM: argument #<END OF FILE> is not a string
There must be more to the problem than the processFile function you
included. It doesn't have any string streams. Perhaps you should look at
a backtrace in the debugger and see where you're really dying.
P.S. Rhetorical question: Why did you post this as a followup to the
"Garbage Hunting" message? It makes things very confusing for those of us
who use threaded newsreaders when you put an unrelated message in a thread.
Don't use the followup/reply command when you want to start a new
discussion.
--
Barry Margolin, ······@bbnplanet.com
GTE Internetworking, Powered by BBN, Burlington, MA
*** DON'T SEND TECHNICAL QUESTIONS DIRECTLY TO ME, post them to newsgroups.
Don't bother cc'ing followups to me.
Barry Margolin <······@bbnplanet.com> writes:
> In article <·······································@polaris.umuc.edu>,
> Sanchita Nag <····@polaris.umuc.edu> wrote:
> > I have some code, that runs on Win/Harlequin, but explodes on
> > 'Nix <Solaris/HP-UX> because it can't handle what it thinks is
> > a string:
> >
> > *** - MAKE-STRING-INPUT-STREAM: argument #<END OF FILE> is not a string
>
> There must be more to the problem than the processFile function you
> included. It doesn't have any string streams. Perhaps you should look at
> a backtrace in the debugger and see where you're really dying.
>
> P.S. Rhetorical question: Why did you post this as a followup to the
> "Garbage Hunting" message? It makes things very confusing for those of us
> who use threaded newsreaders when you put an unrelated message in a thread.
> Don't use the followup/reply command when you want to start a new
> discussion.
Did you see it attached to the garbage hunting thread? I didn't...
Sunil
Sunil Mishra <·······@whizzy.cc.gatech.edu> writes:
> Barry Margolin <······@bbnplanet.com> writes:
>
> > In article <·······································@polaris.umuc.edu>,
> > Sanchita Nag <····@polaris.umuc.edu> wrote:
> > > I have some code, that runs on Win/Harlequin, but explodes on
> > > 'Nix <Solaris/HP-UX> because it can't handle what it thinks is
> > > a string:
> > >
> > > *** - MAKE-STRING-INPUT-STREAM: argument #<END OF FILE> is not a string
> >
> > There must be more to the problem than the processFile function you
> > included. It doesn't have any string streams. Perhaps you should look at
> > a backtrace in the debugger and see where you're really dying.
> >
> > P.S. Rhetorical question: Why did you post this as a followup to the
> > "Garbage Hunting" message? It makes things very confusing for those of us
> > who use threaded newsreaders when you put an unrelated message in a thread.
> > Don't use the followup/reply command when you want to start a new
> > discussion.
>
> Did you see it attached to the garbage hunting thread? I didn't...
>
> Sunil
In gnus use the command "A T" to fetch the current thread. You'll get
the huge garbage hunting thread.
Christopher
······@2xtreme.net (Christopher R. Barry) writes:
> In gnus use the command "A T" to fetch the current thread. You'll get
> the huge garbage hunting thread.
>
> Christopher
I use GNUS all the time, and just tried the command you suggested. Two
things happend:
* shift-a shift-t showed up in the mode line as "A t" (rather than "A T")
* emacs beeped.
Looks like my setup is different/broken...
Sunil
Sunil Mishra <·······@whizzy.cc.gatech.edu> writes:
> ······@2xtreme.net (Christopher R. Barry) writes:
>
> > In gnus use the command "A T" to fetch the current thread. You'll get
> > the huge garbage hunting thread.
> >
> > Christopher
>
> I use GNUS all the time, and just tried the command you suggested. Two
> things happend:
>
> * shift-a shift-t showed up in the mode line as "A t" (rather than "A T")
> * emacs beeped.
>
> Looks like my setup is different/broken...
>
> Sunil
If you have a menubar at the top of your Emacs go to Article -> `Fetch
current thread'. This is bound by default to "A T" (at least with
every version I've ever used) and the function is
`gnus-summary-refer-thread'.
Christopher
······@2xtreme.net (Christopher R. Barry) writes:
> Sunil Mishra <·······@whizzy.cc.gatech.edu> writes:
>
> > ······@2xtreme.net (Christopher R. Barry) writes:
> >
> > > In gnus use the command "A T" to fetch the current thread. You'll get
> > > the huge garbage hunting thread.
> > >
> > > Christopher
> >
> > I use GNUS all the time, and just tried the command you suggested. Two
> > things happend:
> >
> > * shift-a shift-t showed up in the mode line as "A t" (rather than "A T")
> > * emacs beeped.
> >
> > Looks like my setup is different/broken...
> >
> > Sunil
>
> If you have a menubar at the top of your Emacs go to Article -> `Fetch
> current thread'. This is bound by default to "A T" (at least with
> every version I've ever used) and the function is
> `gnus-summary-refer-thread'.
>
> Christopher
Hmmm, gnus 5.5 on emacs 20.2, as present on our school machines, does not
appear to have this command. I'll have to look into this...
Sunil
Sunil Mishra wrote:
> Did you see it attached to the garbage hunting thread? I didn't...
Then your newsreader isn't doing threading based on the
References: fields of articles. Or it is, but splits off
things with different subjects. Or something.
In any case: the "Newbie question" article had a References:
field pointing to the "Garbage hunting" article. It shouldn't
have done.
--
Gareth McCaughan Dept. of Pure Mathematics & Mathematical Statistics,
·····@dpmms.cam.ac.uk Cambridge University, England.