I am developing a small web application using Hunchentoot. It is
currently running in a VMWare instance with Ubuntu as the guest OS and
Windows XP as host. Because it runs on my machine I have given the
instance 300 MB memory.
Now I am thinking of logging the latest X requests in a global
variable, to make debugging easy if error reports from users comes
in. I was thinking of maybe saving the last 100 or so, but due to the
limited memory I have, I would like to know if there is a way for me
to calculate or measure how much memory these 100 request objects will
consume, to see if this is feasible. Is there? Other than running
`top' and
watching the memory consumption increase, that is.
I am using the latest SBCL available in Ubuntu 8.04.
Thanks!
/Mathias
PS. To keep only the last X requests stored, I wrote a couple of
"ring" functions. If anyone have suggestions on a better data
structure to save this in, please let me know:
(defun make-ring (len)
(cons 0 (make-array len)))
(defun ring-push (ring item)
(let ((vec (cdr ring))
(idx (car ring)))
(setf (aref vec idx) item)
(setf (car ring) (mod (1+ idx) (length vec)))))
(defun ring-pop (ring)
(let ((vec (cdr ring))
(idx (car ring)))
(setf (car ring) (mod (- (+ idx (length vec)) 1) (length vec)))
(aref vec (car ring))))
(defvar *requests*)
(setf *requests* (make-ring 10))
And I call it like this:
(ring-push *requests* *request*)
From: Edi Weitz
Subject: Re: Can I measure how much memory an object consumes?
Date:
Message-ID: <uk5h6ygos.fsf@agharta.de>
On Tue, 3 Jun 2008 00:38:32 -0700 (PDT), Mathias Dahl <············@gmail.com> wrote:
> I am developing a small web application using Hunchentoot. It is
> currently running in a VMWare instance with Ubuntu as the guest OS
> and Windows XP as host. Because it runs on my machine I have given
> the instance 300 MB memory.
>
> Now I am thinking of logging the latest X requests in a global
> variable, to make debugging easy if error reports from users comes
> in. I was thinking of maybe saving the last 100 or so, but due to
> the limited memory I have, I would like to know if there is a way
> for me to calculate or measure how much memory these 100 request
> objects will consume, to see if this is feasible. Is there? Other
> than running `top' and watching the memory consumption increase,
> that is.
That's implementation-dependent. The question has been asked before,
not too long ago, so you'll find answers for different implementations
if you search comp.lang.lisp a bit. Here's something for LispWorks:
http://www.lispworks.com/documentation/lw51/LWRM/html/lwref-211.htm
Edi.
--
Lisp is not dead, it just smells funny.
Real email: (replace (subseq ·········@agharta.de" 5) "edi")
On 2008-06-03, Mathias Dahl <············@gmail.com> wrote:
> Now I am thinking of logging the latest X requests in a global
> variable, to make debugging easy if error reports from users comes
> in. I was thinking of maybe saving the last 100 or so, but due to the
> limited memory I have, I would like to know if there is a way for me
> to calculate or measure how much memory these 100 request objects will
> consume, to see if this is feasible. Is there? Other than running
> `top' and watching the memory consumption increase, that is.
You could use graph.lisp (linked below), which is an SBCL-specific
answer that I wrote some time ago. (It's a quick copy&paste hack based
on other code, so don't expect its source code to be particularly nice.)
Here is why I think graph.lisp is a helpful approach to this:
When someone asks the kind of question you posted, the usual
recommendation is to allocate a large number of objects instead and
observe the effects on memory allocation, e.g. using ROOM and TIME.
How does that approach differ from your original question about the
exact memory requirements of a particular number of objects?
One commonly cited difficulty is the definition of "object". For
example, there's a good chance that all your CLOS objects have constant
size: At least in SBCL, the actual data is stored in a separate vector,
which the "CLOS object" only points to.
But when you asked about the size of those objects, you probably wanted
to include the size of that vector.
Once you realize that the total size of the "object" includes the sizes
of other "objects" referenced from it, you are actually measuring the
size of a graph of objects, and the question becomes: "How deep into
which parts of the graph do you want to recurse?".
There is no answer to that question that will always be helpful -- it
depends on what you'd like to achieve (just as with the similar problems
of equality or copying of objects).
But there are some heuristics which yield good results. In your case:
- you'd like to follow the CLOS instance to its data vector
- you don't want to include the class object, which is referenced from
the CLOS instance (in SBCL's case, the LAYOUT)
- you probably don't want to include symbols, function, packages and
similar objects either
You'll also have to think about shared structure, so the size of the
graph starting at that list of 100 requests is possibly smaller than the
sum of the sizes of 100 one-element lists of one request each.
Based on all this, my approach is to draw a graph of the objects, and
let the user discover the difficult questions himself.
So for the objects you asked about, we would just draw the CLOS instance
and its data vector separately, exposing this implementation detail to
the user instead of covering it up.
We still need the heuristics mentioned above to keep the graph
reasonably small, since we could easily plot the entire lisp image if we
weren't careful here.
It's usually hopeless to try and plot even a single package with all its
symbols. But just as the traditional box&pointer diagrams for lists,
there pictures help explain the subject matter quite well, I think, and
once you've got an idea what the Lisp does under the hood, you'll also
feel more comfortable with the numbers ROOM and TIME give.
Here are some examples (object size in bytes):
* http://www.lichteblau.com/tmp/dot/pathname.png
A pathname, illustrating that we follow neither the layout to the
structure class nor the symbols.
* http://www.lichteblau.com/tmp/dot/defstruct-vs-defclass.png
An illustration of the difference between the implementation of
structure classes and standard classes.
* http://www.lichteblau.com/tmp/dot/package-empty.png
An empty package (as mentioned above, anything larger wouldn't be
very readable).
* http://www.lichteblau.com/tmp/dot/function-print.png
A larger example, showing the code component that #'print is a part
of.
The code is here:
http://www.lichteblau.com/tmp/dot/graph.lisp
http://www.lichteblau.com/tmp/dot/test.lisp
An installation of graphviz is required.
> I am using the latest SBCL available in Ubuntu 8.04.
Don't know whether it still works with that version, but there's a good
chance that it does.
d.
From: Thomas A. Russ
Subject: Re: Can I measure how much memory an object consumes?
Date:
Message-ID: <ymi8wxebgal.fsf@blackcat.isi.edu>
Mathias Dahl <············@gmail.com> writes:
> PS. To keep only the last X requests stored, I wrote a couple of
> "ring" functions. If anyone have suggestions on a better data
> structure to save this in, please let me know:
Well, this is obviously a challenge that few readers of c.l.l can
resist. ;)
One interesting solution would involve the use of a circular list data
structure. This solution, though fun, keeps things perhaps a bit
backwards, since it keeps a pointer to start of the data and accesses
things going from earliest to latest rather than vice versa.
(setq *print-circle* t) ; Or you'll regret it!
(defstruct ring
data
current
next)
(defun create-ring (len)
(let ((circ (make-list len)))
(setf (cdr (last circ)) circ)
(make-ring :data circ :current circ :next circ)))
(defun ring-push (ring item)
(setf (car (ring-next ring)) item)
(when (eq (ring-next ring) (ring-current ring))
(setf (ring-current ring) (cdr (ring-current ring))))
(setf (ring-next ring) (cdr (ring-next ring)))
ring)
(defun ring-pop (ring)
(prog1 (car (ring-current ring))
(setf (car (ring-current ring)) nil)
(setf (ring-current ring) (cdr (ring-current ring)))))
(defun ring-head (ring)
(car (ring-current ring)))
--
Thomas A. Russ, USC/Information Sciences Institute