I am getting the following message in my *inferior-lisp*:
* Heap exhausted during allocation: 1540096 bytes available, 8323088
requested.
debugger invoked on a SB-KERNEL::HEAP-EXHAUSTED-ERROR in thread
#<THREAD "control-thread" {AB38F99}>: Heap exhausted: 1540096 bytes
available, 8323088 requested. PROCEED WITH CAUTION!
No backtrace shows up in SLIME, and I have to kill the sbcl process
(version 1.0.12.0) manually from the command line.
Using format's in my code, I managed to locate the part that leads to
this error: it is a foreign function call (to UMFPACK's solver). I
would like to analyze and solve this problem, but the fact that SBCL
fails to respond after it happens makes this difficult.
Questions:
1. could this be SBCL detecting that UMFPACK exhausted the heap, or is
it some process inside SBCL that exhausted the heap (sorry for the
silly question, I am not familiar with the intricacies of memory
management).
2. Could it be that some setting of --dynamic-space-size would solve
the problem? The SBCL manual says that the default is platform-
dependent, how can I find out what it is?
Thanks,
Tamas
On 2008-01-12, Tamas <······@gmail.com> wrote:
> 1. could this be SBCL detecting that UMFPACK exhausted the heap, or is
> it some process inside SBCL that exhausted the heap (sorry for the
> silly question, I am not familiar with the intricacies of memory
> management).
Detecting this situation is easy, but handling it gracefully is
difficult. A hard limit on the amount of memory allocated (which cannot
be exceeded under any circumstances) would be very hard to get right.
But in my recent work on incremental allocation, I have added a "soft"
limit, which provides protection against out of memory situations, but
can (and will) be exceeded while the condition is being handled.
Please try my SBCL branch called "relocation" on repo.or.cz if you are
interested in testing this soft limit.[1] Please note that incremental
allocation doesn't work on all platforms yet.
As for a hard limit, here is what unpatched SBCL does, and my thoughts
on why a better implementation of this mechanism is unlikely to arrive
any time soon:
SBCL attempts to signal a STORAGE-CONDITION when memory runs out, but
that mechanism is not very useful for two reasons:
- an attempt to handle that condition can fail badly when it needs
even more memory
- in particular, if GC itself runs out of memory, it is too late for a
STORAGE-CONDITION and SBCL just gives up.
Your question often comes up, presumably because some languages and
their implementations (Java in particular) have relatively reliable
support for out of memory situations where an exception is thrown and
can be caught.
This it isn't easy in any implementation. In most Java VMs the
exception is pre-allocated, so that throwing the exception does not lead
to another allocation attempt. Then the exception propagates down the
stack. Exception handlers can try to allocate new memory in this
situation, and the worst thing that can happen is that another
OutOfMemoryError is thrown, which will cause the stack to be unwound
even further.
SBCL attempts to signal a condition just like this, but there are
several complications if the goal is an absolutely safe mechanism for
storage conditions:
- with Lisp's condition handling architecture, which runs handler
functions -before- unwinding the stack, there is the possibility
that a handler will try to allocate more memory and cause yet
another STORAGE-CONDITION before unwinding the stack. But as long
as the stack hasn't been unwound, no memory will have been freed
either.
- the use of a copying GC means that in a worst case scenario, twice
as much memory will be needed during GC before any memory can
actually be freed. To guarantee being able to do this GC, SBCL
would have to limit the amount of memory available to the allocator
to half of the user-specified limit, which would not be very useful.
- even that limit isn't exact because pinning of pages by the (partly)
conservative GC algorithm on x86 and x86-64 can waste additional
memory
> 2. Could it be that some setting of --dynamic-space-size would solve
Yes. Just experiment with different values to this argument.
Again, when using my incremental allocation branch you would not have
this problem, because the page table is resized automatically and
--dynamic-space-size only specifies the soft limit.
> the problem? The SBCL manual says that the default is platform-
> dependent, how can I find out what it is?
You can calculate the default dynamic space size using
(- SB-VM:DYNAMIC-SPACE-END SB-VM:DYNAMIC-SPACE-START)
(Note that DYNAMIC-SPACE-END is currently misnamed -- it refers to the
the default value, not the actual run-time value specified by
--dynamic-space-size. A better name would be DEFAULT-DYNAMIC-SPACE-END.)
David
--------
[1] http://article.gmane.org/gmane.lisp.steel-bank.devel/10506/