I've used C++ system is the past where the allocation operators are
changed to gather more information. For example, instead of
Foo *foo = new Foo();
You would use an allocator that has a category and a name:
Foo *foo = new(FOO_PACKAGE) Foo();
The same can be done for malloc and friends. The primary usage of
this is memory budget tracking, different parts of the system are
assigned budgets and you can analyze usage to see where your
application is blowing out.
How would one implement this kind of change in Lisp?
Off the top of my head I'd firstly say that it is probably
implementation specific, and secondly that the best way to do it might
be to override the functions that allocate memory (cons, make-array,
make-hash-table, etc) & employ a dynamic variable or callstack
information to track where the allocation occurred.
Any more ideas? Or is this just something that large Lisp apps don't
worry about?
Cheers,
Brad
In AllegroCL (www.franz.com) if you type this: (room t)
It would give you something like this:
CG-USER(1): (room t)
area area address(bytes) cons other bytes
# type 8 bytes each
(free:used) (free:used)
Top #x20f30000
New #x20e20000(1114112) 704:51265 223904:420664
New #x20d10000(1114112) ----- -----
2 Old #x20b10000(2097152) 665:20734 1447216:476704
1 Old #x20a10000(1048576) 0:18342 0:900432
0 Old #x20000a90(10548592) 0:350536 0:7725216
Tot (Old Areas) 665:389612 1447216:9102352
Root pages: 257
Lisp heap: #x20000000 pos: #x20f30000 resrve: #x2f000000
C heap: #x54000000 pos: #x54028000 resrve: #x55000000
Pure space: #x1f310000 end: #x1fff0f38
code type items bytes
112: (SIMPLE-ARRAY T) 86956 4419104 34.1%
1: CONS 437191 3497528 27.0%
8: FUNCTION 28627 1713264 13.2%
7: SYMBOL 47707 1144968 8.8%
117: (SIMPLE-ARRAY CHARACTER) 11883 605840 4.7%
108: (SHORT-SIMPLE-ARRAY CODE) 1017 394208 3.0%
12: STANDARD-INSTANCE 22439 359024 2.8%
9: CLOSURE 16016 278032 2.1%
15: STRUCTURE 4037 151808 1.2%
111: (SHORT-SIMPLE-ARRAY FOREIGN) 7649 128376 1.0%
118: (SIMPLE-ARRAY BIT) 320 110944 0.9%
125: (SIMPLE-ARRAY (UNSIGNED-BYTE 8)) 249 86336 0.7%
10: HASH-TABLE 436 17440 0.1%
18: BIGNUM 1029 10448 0.1%
127: (SIMPLE-ARRAY (UNSIGNED-BYTE 32)) 8 7984 0.1%
120: (SIMPLE-ARRAY FIXNUM) 5 5168 0.0%
93: (ARRAY (UNSIGNED-BYTE 8)) 180 4320 0.0%
126: (SIMPLE-ARRAY (UNSIGNED-BYTE 16)) 7 3640 0.0%
16: SINGLE-FLOAT 324 2592 0.0%
17: DOUBLE-FLOAT 137 2192 0.0%
130: MV-VECTOR 9 792 0.0%
86: (ARRAY BIT) 29 696 0.0%
80: (ARRAY T) 13 312 0.0%
20: COMPLEX 11 176 0.0%
11: READTABLE 8 128 0.0%
85: (ARRAY CHARACTER) 5 120 0.0%
96: (SHORT-SIMPLE-ARRAY T) 6 96 0.0%
19: RATIO 6 96 0.0%
123: (SIMPLE-ARRAY (SIGNED-BYTE 32)) 1 88 0.0%
75: SHORT-ARRAY 2 48 0.0%
total bytes = 12945768
aclmalloc arena:
max size free bytes used bytes total
16 1584 2512 4096
48 5856 6432 12288
112 3248 336 3584
240 3120 720 3840
496 2480 1488 3968
1008 4032 0 4032
2032 2032 38608 40640
4080 0 24480 24480
5104 20416 0 20416
9200 9200 27600 36800
total bytes: 51968 102176 154144
Other implementations might have other tools for doing that.
bradb wrote:
> I've used C++ system is the past where the allocation operators are
> changed to gather more information. For example, instead of
> Foo *foo = new Foo();
> You would use an allocator that has a category and a name:
> Foo *foo = new(FOO_PACKAGE) Foo();
>
> The same can be done for malloc and friends. The primary usage of
> this is memory budget tracking, different parts of the system are
> assigned budgets and you can analyze usage to see where your
> application is blowing out.
> How would one implement this kind of change in Lisp?
> Off the top of my head I'd firstly say that it is probably
> implementation specific, and secondly that the best way to do it might
> be to override the functions that allocate memory (cons, make-array,
> make-hash-table, etc) & employ a dynamic variable or callstack
> information to track where the allocation occurred.
>
> Any more ideas? Or is this just something that large Lisp apps don't
> worry about?
>
> Cheers,
> Brad
>
In article <·······················@l22g2000prc.googlegroups.com>,
bradb <··············@gmail.com> wrote:
> I've used C++ system is the past where the allocation operators are
> changed to gather more information. For example, instead of
> Foo *foo = new Foo();
> You would use an allocator that has a category and a name:
> Foo *foo = new(FOO_PACKAGE) Foo();
>
> The same can be done for malloc and friends. The primary usage of
> this is memory budget tracking, different parts of the system are
> assigned budgets and you can analyze usage to see where your
> application is blowing out.
> How would one implement this kind of change in Lisp?
> Off the top of my head I'd firstly say that it is probably
> implementation specific, and secondly that the best way to do it might
> be to override the functions that allocate memory (cons, make-array,
> make-hash-table, etc) & employ a dynamic variable or callstack
> information to track where the allocation occurred.
>
> Any more ideas? Or is this just something that large Lisp apps don't
> worry about?
On Lisp Machines, all the memory allocation functions (CONS, MAKE-ARRAY,
etc.) take optional arguments specifying an "area", which defaults to
the value of something like *DEFAULT-CONS-AREA*. The main use of this
is that areas may have different properties (for instance, you can tell
the GC not to scan an area, or set an area to be read-only), you can
discard an area to deallocate everything in it in one fell swoop (this
tends to go along with the no-GC attribute mentioned), and diagnostics
can display memory use by area.
--
Barry Margolin, ······@alum.mit.edu
Arlington, MA
*** PLEASE post questions in newsgroups, not directly to me ***
*** PLEASE don't copy me on replies, I'll read them in the group ***
In article <····························@comcast.dca.giganews.com>,
Barry Margolin <······@alum.mit.edu> wrote:
> In article <·······················@l22g2000prc.googlegroups.com>,
> bradb <··············@gmail.com> wrote:
>
> > I've used C++ system is the past where the allocation operators are
> > changed to gather more information. For example, instead of
> > Foo *foo = new Foo();
> > You would use an allocator that has a category and a name:
> > Foo *foo = new(FOO_PACKAGE) Foo();
> >
> > The same can be done for malloc and friends. The primary usage of
> > this is memory budget tracking, different parts of the system are
> > assigned budgets and you can analyze usage to see where your
> > application is blowing out.
> > How would one implement this kind of change in Lisp?
> > Off the top of my head I'd firstly say that it is probably
> > implementation specific, and secondly that the best way to do it might
> > be to override the functions that allocate memory (cons, make-array,
> > make-hash-table, etc) & employ a dynamic variable or callstack
> > information to track where the allocation occurred.
> >
> > Any more ideas? Or is this just something that large Lisp apps don't
> > worry about?
>
> On Lisp Machines, all the memory allocation functions (CONS, MAKE-ARRAY,
> etc.) take optional arguments specifying an "area", which defaults to
> the value of something like *DEFAULT-CONS-AREA*. The main use of this
> is that areas may have different properties (for instance, you can tell
> the GC not to scan an area, or set an area to be read-only), you can
> discard an area to deallocate everything in it in one fell swoop (this
> tends to go along with the no-GC attribute mentioned), and diagnostics
> can display memory use by area.
Have a peek:
http://lispm.dyndns.org/lisp/pics/peek-area.jpg
--
http://lispm.dyndns.org
Barry Margolin <······@alum.mit.edu> writes:
> On Lisp Machines, all the memory allocation functions (CONS, MAKE-ARRAY,
> etc.) take optional arguments specifying an "area", which defaults to
> the value of something like *DEFAULT-CONS-AREA*. The main use of this
> is that areas may have different properties (for instance, you can tell
> the GC not to scan an area, or set an area to be read-only), you can
> discard an area to deallocate everything in it in one fell swoop (this
> tends to go along with the no-GC attribute mentioned), and diagnostics
> can display memory use by area.
Some additional color, since all the interesting aspects of this were in
the use/policy issues, not in the technical design...
I seem to recall that *DEFAULT-CONS-AREA* could be rebound, but it
turned out to be a bad thing to do in the cases I saw. For example,
someone made the compiler, for a while, have macro consing go into its
own area and people were surprised how much permanent heap storage
happened in macros, including trace output that ended up on dynamic
windows panes and was deadly to move the mouse over later after the
area had been reset. As I recall, they had to turn off the "feature"
binding *DEFAULT-CONS-AREA* during compilation because it caused so
many problems.
I think probably better luck was had if one never re-bound
*DEFAULT-CONS-AREA* and always had to give an explicit arg, as Barry
suggests, treating *DEFAULT-CONS-AREA* as a constant, not a changeable
variable. By only passing explicit args to the allocation functions,
one really couldn't create the problem by accident in which something
was consed in the wrong place.
And I think using this may have sometimes improved locality.
Although...
I'm not sure the EGC didn't do a better job on its own... I don't
recall people much talking about area consing in the latter days of
the LispM when I'd have thought it more mature; I think there was a
heavier leaning on the regular GC to naturally consolidate things used
together anyway. I wonder if the EGC obviated the need for area
consing. (It might or might not have... I'm not sure I ever knew. I
imagine this is something that was written up somewhere, so take this
and most of my other remarks above mostly as pointers to things one
might want to check on rather than just assume that because the
feature existed, it was a total success.)
Not that the EGC would have had the ability to do "category tracking" as
mentioned in the OP's subject line.
In article <·············@nhplace.com>,
Kent M Pitman <······@nhplace.com> wrote:
> I'm not sure the EGC didn't do a better job on its own... I don't
> recall people much talking about area consing in the latter days of
> the LispM when I'd have thought it more mature; I think there was a
> heavier leaning on the regular GC to naturally consolidate things used
> together anyway. I wonder if the EGC obviated the need for area
> consing. (It might or might not have... I'm not sure I ever knew. I
> imagine this is something that was written up somewhere, so take this
> and most of my other remarks above mostly as pointers to things one
> might want to check on rather than just assume that because the
> feature existed, it was a total success.)
I think I recall that DEFRESOURCE allowed you to associate an area for
the resource objects. That seemed like a good match, since resource
objects were typically not GCed (they were saved for high-volume reuse).
But for more typical consing, I have a feeling that you're right.
--
Barry Margolin, ······@alum.mit.edu
Arlington, MA
*** PLEASE post questions in newsgroups, not directly to me ***
*** PLEASE don't copy me on replies, I'll read them in the group ***
bradb wrote:
> The same can be done for malloc and friends. The primary usage of
> this is memory budget tracking, different parts of the system are
> assigned budgets and you can analyze usage to see where your
> application is blowing out.
I'd use a low-level backend instrumentation for memory profiling... in
today's impoverished Gnu/Linux world I can't think of anything right now
but back in the days when life was better, I was using the 3rd Degree
Atom Tools that came with DEC Unix for such stuff (C programs in
particular but any object file/executable could be instrumented,
including a Lisp system, if it could be recompiled for use with the tools.)