Based on some discussions I've had about SBCL heap limitations on
ppc-darwin, I came up with the idea to do some practical testing of
heap allocation. By practical, I mean use malloc to allocate as much
memory in a single call as I can.
But this got me thinking further. On both Linux and OS X, malloc is
willing to overcommit memory. That is, you can ask for a large amount
of memory and you will very quickly get back a pointer to the memory
region if the call succeeds. No actual RAM has been used at this
point (except for whatever memory is used to keep track of mallocs so
that free can free them). The returned pointer is simply to a region
of the application's logical address space. Both SBCL and CMUCL also
do overcommit. On OS X, SBCL does not give you large contiguous
regions of memory (yet). It does on Linux. So my further thought
was, what if a Lisp implementation allocates all the heap it can for
its own uses?
My thought was that perhaps with the heap exhausted, you couldn't do a
foreign function call that causes heap space to be allocated. There
won't be any left. So I have this question. Am I right? How do Lisp
implementations deal with FFI calls possibly needing their own heap
space? It seems like you have to leave some heap available and malloc
has to be able to use that bit of heap.
I wrote a very simple C program to see just how much heap I can malloc
under OS X and Linux. I have OS X 10.3.3 on my Mac PowerBook G4. On
my Linux box I'm running the 2.6.5 kernel. The Mac has only 640MB of
RAM while the Linux box has 1GB. The results surprised me. Vega is
my Linux box, Interloper is my Mac:
·····@vega:~/usr/src/mallocator
$ ./mallocator
I was able to malloc a maximum block of 1758666731 bytes.
That's about 1717447 Kb, 1677 Mb, or 1 Gb.
·····@interloper:~/usr/src/mallocator
$ ./mallocator 2>/dev/null
I was able to malloc a maximum block of 2380234752 bytes.
That's about 2324448 Kb, 2269 Mb, or 2 Gb.
And here is the C code (It isn't optimized to make me look smart.
It's just a quick hack and I've already thought of a more concise way
to do this but I don't feel like writing it now.):
// Program to see just how large a chunk of memory can be allocated with
// a single call to malloc(3). This will be done by calling malloc and
// then free(3) with larger and larger values until malloc fails. Then
// a sort of binary search on the size will be used to find the boundry
// between the point where malloc succeeds and fails.
//
// Note that the memory may not actually be available, ie overcommit.
// We don't care about this because we will immediately free the memory.
// This is just an excercise in VM heap size.
#include <stdlib.h>
#include <stdio.h>
int main () {
size_t size = 1024, last_size = 0, temp;
void *memory = NULL;
do {
free(memory);
memory = malloc(size);
if (memory) {
last_size = size; // remember
size <<= 1;
}
} while (memory && size);
if (memory) {
free(memory);
memory = NULL;
}
// This may look weird, but size_t is generally an unsigned type and
// we may have wrapped when we doubled size the last time. This will
// make size a proper number that is less than twice last_size. Also,
// we will never want to have size smaller than last_size.
size = last_size - 1024 + last_size;
while (size - last_size > 1) {
memory = malloc(size);
if (memory) {
free(memory);
memory = NULL;
temp = size - last_size + size;
last_size = size;
size = temp;
} else {
size -= (size - last_size) / 2;
}
}
printf ("\n\n\nI was able to malloc a maximum block of %u bytes.\n", last_size);
printf ("That's about %u Kb, ", last_size = last_size / 1024);
printf ("%u Mb, ", last_size = last_size / 1024);
printf ("or %u Gb.\n", last_size / 1024);
return 0;
}
It would be a crude, and perhaps unreliable means of reverse
engineering the logical address space to get the value of the pointer
and assume the heap grows 'up' to the pointer value + number of bytes
allocated. It's no substitute for looking in the OS documentation to
find out where the heap space goes.
On OS X, malloc spits out a bunch of errors from vm_allocate when I
ask for too much. I gather that is a Mach ABI function that malloc
uses. If that is the case, then as long as vm_allocate and its
counterpart are used, FFI would work so long as Lisp didn't take all
the heap. The question would be, how much to take?
Perhaps Lisp can take all of it. If the Lisp system was clever, it
might be able to provide its own versions of malloc and free to the
foreign functions that it uses. I don't know how durable that is. It
would also give a foreign function carte blanc to overwrite all your
heap. That would not be fun.
--
I wouldn't mind the rat race so much if it wasn't for all the damn cats.
Following up my own post here. If conciseness equals clarity, then I
should post better code:
// Program to see just how large a chunk of memory can be allocated with
// a single call to malloc(3). This will be done by calling malloc and
// then free(3) with size values that represent a binary search of the
// size_t number field.
//
// Note that the memory may not actually be available, ie overcommit.
// We don't care about this because we will immediately free the memory.
// This is just an excercise in VM heap size.
#include <stdlib.h>
#include <stdio.h>
int main () {
// size_t is an unsigned type. by assigning -1, we say give it the
// largest available value without having to check with ctypes.h.
// we do not want size to ever be smaller than last_size.
size_t size = -1, last_size = 0, temp;
void *memory = NULL;
while (size - last_size > 1) {
memory = malloc(size);
if (memory) {
free(memory);
memory = NULL;
temp = size + (size - last_size) / 2;
last_size = size;
size = temp;
} else {
size -= (size - last_size) / 2;
}
}
printf ("I was able to malloc a maximum block of %u bytes.\n", last_size);
printf ("That's about %u Kb, ", last_size = last_size / 1024);
printf ("%u Mb, ", last_size = last_size / 1024);
printf ("or %u Gb.\n", last_size / 1024);
return 0;
}
--
I wouldn't mind the rat race so much if it wasn't for all the damn cats.
David Steuber wrote:
> But this got me thinking further. On both Linux and OS X, malloc is
> willing to overcommit memory. That is, you can ask for a large amount
> of memory and you will very quickly get back a pointer to the memory
> region if the call succeeds.
Well, only if you haven't disabled overcommit on linux, which is something
many of us do on scientific workstations where you really might be accessing
every page you ask for.
David Steuber <·····@david-steuber.com> wrote in message news:<··············@david-steuber.com>...
> But this got me thinking further. On both Linux and OS X, malloc is
> willing to overcommit memory. That is, you can ask for a large amount
> of memory and you will very quickly get back a pointer to the memory
> region if the call succeeds. No actual RAM has been used at this
> point (except for whatever memory is used to keep track of mallocs so
> that free can free them). The returned pointer is simply to a region
> of the application's logical address space. Both SBCL and CMUCL also
> do overcommit. On OS X, SBCL does not give you large contiguous
> regions of memory (yet). It does on Linux. So my further thought
> was, what if a Lisp implementation allocates all the heap it can for
> its own uses?
>
I think it deoends on what you mean. `allocating all the heap it can'
would presumably mean `allocating all the address space there is',
since you can always add more swap space to back any requirement
later. On a 32 bit machine this is probably really easy to do, on a
64bit machine, well, if you ask for *that* much memory, you probably
are going to get what you deserve...
··········@tfeb.org (Tim Bradshaw) writes:
> David Steuber <·····@david-steuber.com> wrote in message news:<··············@david-steuber.com>...
>
> > But this got me thinking further. On both Linux and OS X, malloc is
> > willing to overcommit memory. That is, you can ask for a large amount
> > of memory and you will very quickly get back a pointer to the memory
> > region if the call succeeds. No actual RAM has been used at this
> > point (except for whatever memory is used to keep track of mallocs so
> > that free can free them). The returned pointer is simply to a region
> > of the application's logical address space. Both SBCL and CMUCL also
> > do overcommit. On OS X, SBCL does not give you large contiguous
> > regions of memory (yet). It does on Linux. So my further thought
> > was, what if a Lisp implementation allocates all the heap it can for
> > its own uses?
> >
>
> I think it deoends on what you mean. `allocating all the heap it can'
> would presumably mean `allocating all the address space there is',
> since you can always add more swap space to back any requirement
> later. On a 32 bit machine this is probably really easy to do, on a
> 64bit machine, well, if you ask for *that* much memory, you probably
> are going to get what you deserve...
I mean in the sense of reserving all the available address space for
heap. From the way I gather that malloc works on some systems, you
can ask for the memory and get a pointer to it. So far, so good.
Once you start to use that memory, the OS then tries its darndest to
give it to you. Bad things happen if there is not enough physical
memory to back the virtual/logical memory. (I hate it when terms are
used imprecisely. I no longer know what is being talked about.).
When 2104 rolls around, I expect to have enough RAM in my computer to
back a 64bit address space. There should be plenty of swap available
on the disk (or whatever the permenant storage is then).
If I may recast my question for clarity, how does a Lisp
implementation deal with the fact that ffi calls may require sharing
the application's heap space with those pesky foreigners?
--
I wouldn't mind the rat race so much if it wasn't for all the damn cats.
David Steuber <·····@david-steuber.com> wrote in message news:<··············@david-steuber.com>...
> I mean in the sense of reserving all the available address space for
> heap. From the way I gather that malloc works on some systems, you
> can ask for the memory and get a pointer to it. So far, so good.
> Once you start to use that memory, the OS then tries its darndest to
> give it to you. Bad things happen if there is not enough physical
> memory to back the virtual/logical memory. (I hate it when terms are
> used imprecisely. I no longer know what is being talked about.).
I think this is right. You get all the address space, but it will not
be mapped to anything until you actually ask for it. This is pretty
much always the right thing to do, I think as it allows tricks like
mapping enormous spaces which you expect to use only sparsely (for
instance, the Solaris filesystem snapshot system memory maps a file
the size of the filesystem it is snapshotting, and then fills it
sparsely). It would be good to have an `and if I actually try and
touch all this memory now, what will happen?' call. On Symbolics
machines, you could run out of space, get into a break loop, then add
swap space and continue, and there's no real reason why systems can't
do that now, I think.
> If I may recast my question for clarity, how does a Lisp
> implementation deal with the fact that ffi calls may require sharing
> the application's heap space with those pesky foreigners?
I think grabbing the entire address space isn't likely to be a good
idea. Instead grab just as much as you might need, and allow other
people to have their own space.
--tim
··········@tfeb.org (Tim Bradshaw) writes:
> David Steuber <·····@david-steuber.com> wrote in message news:<··············@david-steuber.com>...
>
> > I mean in the sense of reserving all the available address space for
> > heap. From the way I gather that malloc works on some systems, you
> > can ask for the memory and get a pointer to it. So far, so good.
> > Once you start to use that memory, the OS then tries its darndest to
> > give it to you. Bad things happen if there is not enough physical
> > memory to back the virtual/logical memory. (I hate it when terms are
> > used imprecisely. I no longer know what is being talked about.).
>
> I think this is right.
No, no, no! This is correct in a descriptive sense, but this line
of pursuit is absolutely wrong for most applications that do
any significant allocation (i.e. any app that grows), and will likely
cause unfathomable deaths when swap space gets low or is exhausted.
And, of course, Lisp will be blamed.
> You get all the address space, but it will not
> be mapped to anything until you actually ask for it.
But the asking for it is critical. You already have asked for it, and
have been promised it, but then when you go to get it, the operating system
renegs, and you're left with a cryptic error message that is completely
undebuggable. See below.
This is pretty
> much always the right thing to do, I think as it allows tricks like
> mapping enormous spaces which you expect to use only sparsely (for
> instance, the Solaris filesystem snapshot system memory maps a file
> the size of the filesystem it is snapshotting, and then fills it
> sparsely). It would be good to have an `and if I actually try and
> touch all this memory now, what will happen?' call.
The way to get that to occur is to insist that all memory allocated be
backed by swap (not necessarily swap disk space - memory can also act as
swap backing). See below.
> On Symbolics
> machines, you could run out of space, get into a break loop, then add
> swap space and continue, and there's no real reason why systems can't
> do that now, I think.
It's probably because there is some reserve memory to print out errors
and potentially to recover. But many systems don't reserve such space;
it depends on the operating system.
Let me first define terms that I will use, and then I'll use them in
explanation below of our experiences with this issue:
- Physical memory: memory that has physical addresses; e.g. address zero
has all address lines at binary zero, and resides at precisely one location
in Random Access Memory.
- Virtual memory: memory that is not physical, but which instead resides
somewhere on storage not necessarily at the physical location. A process
might define address 0 for itself, which (depending on whether it is
shared or not) may not be the same physical address as another process's
address 0. If the storage representing the address is disk space, then the
virtual memory is said to be "swapped out".
- Virtual swap (an SGI term): memory that is not backed up by actual
physical storage. A system can have a mixture of physical swap (consisting
of RAM and disk space) and virtual swap (which is just a number managed
by the operating system).
Examples of virtual swap:
1. SGI: "man swapctl" will describe virtual swap and how it can be
controlled. Irix did away with the distinction between fork() and vfork()
and instead allowed all forks to operate, even if there was not enough
swap space to perform a copy of the entire address space, as long as the
virtual swap requirements were met. This was done so that a huge
program (say, a program which consumed more than half the swap space,
such as a lisp :-) could fork itself for the purpose of exec-ing a much
smaller program, such as pwd.
2. Solaris: mmap has the MAP_NORESERVE attribute, which provides for a
mapping which does not grab actual swap space until it was actually
accessed.
3. AIX: AIX also allows virtual paging space, but also provides a SIGDANGER
signal, which says that the paging space is getting low. The default action
of SIGDANGER is to terminate the application.
We first encountered this problem of virtual swap on NeXT boxes, which
would _always_ allocate virtually, but when there was no more paging space,
the system would hang until the space became available, If most programs
were not going to terminate, and some programs needed yet more swap,
then this would cause deadlock situations. I don't remember what we did
to work around this problem; I didn't do the NeXT port of Allegro CL,
but I do remember it being a problem.
The next time we saw this problem was on AIX. Interestingly, AIX had
accounted for this problem, and had provided SIGDANGER, which we
call signal() on to catch (so our lisp doesn't go away). However,
ironically, the X11 that was provided with AIX was not compiled with
any calls to handle SIGDANGER, so although our lisp did not terminate
when swap space got low, X itself would terminate. The percentage of
memory that triggers a SIGDANGER is configurable (and the amount of
physical blocks is very easily configured, as well) but it is hard
to know how much memory one needs to recover from an ou-of-swap situation.
Finally, we have had customers with huge applications on both SGI and
on Solaris (which eventually pressed us to do 64-bit versions of
Allegro CL on many platforms) who also noted that when their physical
swap space was recduced and virtual swap was increased (due to them
needing the extra disk space for file storage purposes, or wanting to do
vforks, etc) the number of incidents rose in which unexplainable deaths
of the lisp _and_ the operating systems would occur. These deaths
could _never_ be reproduced when the virtual swap was zero (i.e. all
virtual memory was backed up by physical swap), but when we dug in to
debug these situations, we found that inevitably what had happened was
that the program snarfing up more physical pages (by touching them) was
then grabbing the last physical page, and then when it couldn't get any
more (and, meanwhile, other system daemons would try to allocate memory
and fail, thus terminating citical operating system programs) the first
touch of virtual space which failed would cause SIGBUS or SIGSEGV, but
an error message couldn't always be printed out (because such a call to
printf required a buffer for the first time, and memory for such a buffer
wasn't available!)
In short, don't go the route of allocating virtual swap; it will
inevitably lead to disaster. The rub is, that disaster will likely
happen on a lisp, because it will be one of the few systems that can
take an application to such large memory requirements, and so thus Lisp
will be accused for the failure.
I think that what David is / should be really looking for is
virtual _address_ allocation, rather than virtual swap (or overcommit,
or whatever else you want to call it). It is ok to reserve address
ranges, and to adjust its permissions to No Access so that stray pointers
don't accidentally commit pages that weren't asked for by the heap
manager. Such stray accesses will cause Segvs, as is appropriate, and
when the heap allocator wants to turn portions of the reserved address
ranges into usable heap space, it can remap that range of memory (but
not virtually) and adjust the permissions again to cover exactly that
new memory that is desired. This also guarantees that any allocation
errors that will occur will occur at heap-management time, and not in
some random function that happens to cons (with no indication why it
failed).
--
Duane Rettig ·····@franz.com Franz Inc. http://www.franz.com/
555 12th St., Suite 1450 http://www.555citycenter.com/
Oakland, Ca. 94607 Phone: (510) 452-2000; Fax: (510) 452-0182
* Duane Rettig wrote:
> In short, don't go the route of allocating virtual swap; it will
> inevitably lead to disaster. The rub is, that disaster will likely
> happen on a lisp, because it will be one of the few systems that can
> take an application to such large memory requirements, and so thus Lisp
> will be accused for the failure.
I don't really have time or energy to disagree with you, but this made
me laugh. I guess you've never made a snapshot of a large file system
if you think typical Lisp's memory requirements are large.
--tim
Tim Bradshaw <···@cley.com> writes:
> * Duane Rettig wrote:
>
> > In short, don't go the route of allocating virtual swap; it will
> > inevitably lead to disaster. The rub is, that disaster will likely
> > happen on a lisp, because it will be one of the few systems that can
> > take an application to such large memory requirements, and so thus Lisp
> > will be accused for the failure.
>
> I don't really have time or energy to disagree with you, but this made
> me laugh. I guess you've never made a snapshot of a large file system
> if you think typical Lisp's memory requirements are large.
I'm not talking about typical Lisp, but about some Lisp applications -
real ones. The fact that Lisp allows an application to grow as large
as the operating system will let it means that it thus can, and if/when
it does it will run into resource problems not encountered by apps written
in languages that don't scale well.
The correlary to Murphy's law which states that computer programs tend
to expand to fill available memory is probably apropos here, but since
Lisp apps scale so well they tend to hit memory limits much faster than
other apps, which are just now finding their way above the 32-bit range.
--
Duane Rettig ·····@franz.com Franz Inc. http://www.franz.com/
555 12th St., Suite 1450 http://www.555citycenter.com/
Oakland, Ca. 94607 Phone: (510) 452-2000; Fax: (510) 452-0182
* Duane Rettig wrote:
> I'm not talking about typical Lisp, but about some Lisp applications -
> real ones. The fact that Lisp allows an application to grow as large
> as the operating system will let it means that it thus can, and if/when
> it does it will run into resource problems not encountered by apps written
> in languages that don't scale well.
> The correlary to Murphy's law which states that computer programs tend
> to expand to fill available memory is probably apropos here, but since
> Lisp apps scale so well they tend to hit memory limits much faster than
> other apps, which are just now finding their way above the 32-bit range.
Sorry? I'm talking about C systems which allocate terabytes (OK, I
think only up to 8 or 16 TB at present - I'm not sure what the UFS
filesystem size limit is now) of address space. C (and I suppose C++)
applications scale *absolutely fine* if they're competently written,
and there are plenty of examples of C/C++ systems that use very large
amounts of memory even outside `strange' examples like the above. The
database that lives where I'm working at present seems to be using 8GB
for cache as well as (I think) mmapping some enormous sparse file, and
it's not a very big database.
Of course, applications that cause the machine to run traumatically
out of physical backing for memory have problems, and OSs often handle
this case badly. But saying that these facilities therefore shouldn't
exist is like saying that you should manage memory yourself because
GCs have bad edge cases.
Tim Bradshaw <···@cley.com> writes:
> Of course, applications that cause the machine to run traumatically
> out of physical backing for memory have problems, and OSs often handle
> this case badly. But saying that these facilities therefore shouldn't
> exist is like saying that you should manage memory yourself because
> GCs have bad edge cases.
But that's not quite what Duane proposed. When I started reading his
post, I was gonna jump in with an objection, but by the time I got to
the end of it, I was quite happy with what he was proposing. That
database could just as well be mapping a gigantic sparse file, but
only asking for backing on the parts it's using. This would have the
advantage that anything it thinks it has access to, it actually does,
and it could sanely recover from out-of-memory conditions, because the
memory it thinks it has set aside for this purpose is really set aside.
--
/|_ .-----------------------.
,' .\ / | No to Imperialist war |
,--' _,' | Wage class war! |
/ / `-----------------------'
( -. |
| ) |
(`-. '--.)
`. )----'
···@famine.OCF.Berkeley.EDU (Thomas F. Burdick) writes:
> Tim Bradshaw <···@cley.com> writes:
>
> > Of course, applications that cause the machine to run traumatically
> > out of physical backing for memory have problems, and OSs often handle
> > this case badly. But saying that these facilities therefore shouldn't
> > exist is like saying that you should manage memory yourself because
> > GCs have bad edge cases.
>
> But that's not quite what Duane proposed. When I started reading his
> post, I was gonna jump in with an objection, but by the time I got to
> the end of it, I was quite happy with what he was proposing. That
> database could just as well be mapping a gigantic sparse file, but
> only asking for backing on the parts it's using. This would have the
> advantage that anything it thinks it has access to, it actually does,
> and it could sanely recover from out-of-memory conditions, because the
> memory it thinks it has set aside for this purpose is really set aside.
Precisely. I don't think that it's explicit memory management to
demand that every allocation be explicit, even if through a hook.
It would be nice if along with the MAP_NORESERVE flag provided by
solaris, its mmap() also marked the unbacked addresses as no-access,
and also provided a hook function that would be called whenever
those addresses were accessed - a user trap handler, of sorts.
Then, precise knowledge of what kinds of accesses the program wants
to allow could be provided, and error messages could be explicit
if such accesses are not allowed. The default might be to generate
a SEGV, as usual.
--
Duane Rettig ·····@franz.com Franz Inc. http://www.franz.com/
555 12th St., Suite 1450 http://www.555citycenter.com/
Oakland, Ca. 94607 Phone: (510) 452-2000; Fax: (510) 452-0182
* Duane Rettig wrote:F
> Precisely. I don't think that it's explicit memory management to
> demand that every allocation be explicit, even if through a hook.
> It would be nice if along with the MAP_NORESERVE flag provided by
> solaris, its mmap() also marked the unbacked addresses as no-access,
> and also provided a hook function that would be called whenever
> those addresses were accessed - a user trap handler, of sorts.
> Then, precise knowledge of what kinds of accesses the program wants
> to allow could be provided, and error messages could be explicit
> if such accesses are not allowed. The default might be to generate
> a SEGV, as usual.
So, the first thing that will happen is that a standard `just allocate
the memory' hook will appear in the standard library. This will
require a trap into user code on every fault, which will be slow. So
a couple of years later, this will be migrated into the OS, where it
will be faster. Almost everyone will use this version (which is
identical to what we have now), because almost no-one will want to do
their memory allocation themselves.
If what you're asking for is a *new* facility, so that the people who
do want to do their own memory management can, then I'm in favour of
that, but there's no chance it will replace the thing that exists now.
The ability to just ask for large address spaces and have the system
manage the allocation of backing for them is just too useful.
--tim
Tim Bradshaw <···@cley.com> writes:
> * Duane Rettig wrote:
> > I'm not talking about typical Lisp, but about some Lisp applications -
> > real ones. The fact that Lisp allows an application to grow as large
> > as the operating system will let it means that it thus can, and if/when
> > it does it will run into resource problems not encountered by apps written
> > in languages that don't scale well.
>
> > The correlary to Murphy's law which states that computer programs tend
> > to expand to fill available memory is probably apropos here, but since
> > Lisp apps scale so well they tend to hit memory limits much faster than
> > other apps, which are just now finding their way above the 32-bit range.
>
> Sorry? I'm talking about C systems which allocate terabytes (OK, I
> think only up to 8 or 16 TB at present - I'm not sure what the UFS
> filesystem size limit is now) of address space. C (and I suppose C++)
> applications scale *absolutely fine* if they're competently written,
> and there are plenty of examples of C/C++ systems that use very large
> amounts of memory even outside `strange' examples like the above.
The problem is not how the system performs when one has the memory,
the problem is in the guarantee. the old addage "nobody will ever
need more than 840K of memory" pops into mind. The issue is that
when (not if) the machine runs out of memory, those comptetently written
C programs are _going_ to break!
The
> database that lives where I'm working at present seems to be using 8GB
> for cache as well as (I think) mmapping some enormous sparse file, and
> it's not a very big database.
So however large the file is, if just a second helping of that file size
can be allocated for swap space, then you're covered, right? If the
file might grow or be filled in arbitrarily, then you're going to run
out of file space as well as virtual memory space, but guess which one
will give you the unintelligible error diagnostic?
> Of course, applications that cause the machine to run traumatically
> out of physical backing for memory have problems, and OSs often handle
> this case badly.
This should not necessarily be the case. The reason this happens is
because vendors don't get beat up on when the applications fail while
running on the edge conditions.
> But saying that these facilities therefore shouldn't
> exist is like saying that you should manage memory yourself because
> GCs have bad edge cases.
No, it's not, It is instead like saying that when the edge cases
occur, the vendor of the gc must be beat up on to fix those edge cases.
This may result in the vendor removing some of the bells and whistles
that cause the edge cases, or perhaps rewiring those bells and whistles,
or perhaps telling you not to use such bells and whistles as a
workaround. One way that a vendor might tell you to work around
a gc problem is to do explicit management, but personally, if it
happened to me, I'd probably accept the workaround but expect a real
fix later on.
--
Duane Rettig ·····@franz.com Franz Inc. http://www.franz.com/
555 12th St., Suite 1450 http://www.555citycenter.com/
Oakland, Ca. 94607 Phone: (510) 452-2000; Fax: (510) 452-0182
Duane Rettig <·····@franz.com> wrote in message news:<·············@franz.com>...
> So however large the file is, if just a second helping of that file size
> can be allocated for swap space, then you're covered, right? If the
> file might grow or be filled in arbitrarily, then you're going to run
> out of file space as well as virtual memory space, but guess which one
> will give you the unintelligible error diagnostic?
>
I thought about this some more walking to work, and I realised that
there's a tacit assumption here that running out of file space is
`better' than rollong over and dying. It may be, but it can be
catastrophically worse as well. Here's two examples that have
happened to me, one in recent history, one longer ago.
1. The primary NIS server (running a major commercial Unix) ran out of
space in /. A few minutes later we discovered that the passwd
updating program didn't update /etc/passwd safely, and would leave a
truncated copy. Prompt loss of service to several thousand machines.
We had good backups, but we weren't copying various crucial system
files to safe locations every few minutes, so we had to get it back
from tape, which took an hour or so. Systems I run nowadays make
copies of this stuff, unfortunately to data volumes (which is better
than nothing); the next systems I build are going to have a /sacred
filesystem which is used *only* for things like this, (and something
that sounds alarms and kills the power to the building if it ever
fills...)
2. More recently, a big data volume on a fileserver filled up. Rather
soon we discovered that none of our (locally written) applications
check the return value of close(2), which you must do with NFS, and a
lot of data was lost. Worse, the files on the server were
inconsistent from the applications' point of view - the applications
wouldn't restart at all. This time we also had to recover from tape,
and it took a painfully long time. I also had to shoot several
people, which is never nice.
In both those cases I would much, much rather have dealt with a crash,
or even repeated crashes.
The underlying issue here is that incompetent programmers write
programs that don't work. It isn't hard to check return codes or
update critical files safely, but it is too hard for these people.
Similarly it isn't hard to set the heap limit for your application to
physmem + swap - 1GB (if it's all the machine will run) or something
equivalent if it's not. But it is too hard for these people.
I can see where Duane is coming from: he's an implementor and probably
has a large number of more-or-less incompetent users who complain
about his application because they're too incompetent to set the heap
size limit that ACL provides. He probably never hears from the ones
that set it correctly (`hey Duane, my program worked, thanks!'). And
his users pay his salary, so killing them is not really a good option.
Finally, he's a nice guy, he'd probably have qualms about killing
them even if there wasn't money involved.
But I'm a Unix systems guy: big boots, manuals used as clubs and
semi-automatic weapons are my stock in trade. I might have been nice
once, but that was so long ago I can't remember, frankly. I'm just
not interested in putting up with people who can't cope with setting
heap limits or dealing with the issues raised by overcomitted memory
which can't be avoided: I just want to get paid. The Sig-Sauer P226 I
keep in the tape safe deals with them effectively: it's a little messy
afterwards, but they don't suffer.
--tim
There's also the underlying point that this fear of overcommit is kind
of a medieval attitude (this isn't meant as an insult...). In the
middle ages, people had really strict ideas about money: don't lend
more than you have, don't lend at interest, don't make promises you
can't keep. But that is long gone now: do you really think that your
insurance company is good for all the policies it's written? Or that
your bank actually has all the cash you've deposited? Of course they
don't. If everyone went to their bank and tried to take out their
money it would fold up. Similarly insurance companies write policies
for many times their capital. There are many, many examples of this:
the modern world *lives* by writing cheques that it can't cash in a
completely literal sense. Instead, for instance, insurance companies
do hairy risk analysis and compute how much more insurance they can
write than they have, and regulatory systems (doing the same analysis,
but with less commercial pressure to twist the results) set capital
adequacy requirements which insurance companies and banks must meet
(these requirements depend on the exposure of the organisation to
risk, but are often 20% or something like that: see for instance the
Basel agreements, or The Economist for good summaries (if you don't
read it already, you should)). Sometimes, they get it wrong and bad
things happen - how bad depends on how badly they get it wrong and how
widespread it is. Insurance companies didn't calculate the financial
risk to them from asbestos claims correctly, and some of them are in
bad trouble; almost no-one got the terrorism risk right either (`yes,
we'll insure your building against someone flying a plane into it,
that never happens'...). But that doesn't invalidate the principle:
we're as well off as we are largely *because* we have an economic
system which can live with systematic overcommit. Similarly, software
people need to get used to the idea of systematic overcommit of many
kinds of resources, and doing the analysis which lets you make these
bets. The renaissance is here already.
··········@tfeb.org (Tim Bradshaw) writes:
> Duane Rettig <·····@franz.com> wrote in message news:<·············@franz.com>...
>
> > So however large the file is, if just a second helping of that file size
> > can be allocated for swap space, then you're covered, right? If the
> > file might grow or be filled in arbitrarily, then you're going to run
> > out of file space as well as virtual memory space, but guess which one
> > will give you the unintelligible error diagnostic?
> >
> I thought about this some more walking to work, and I realised that
> there's a tacit assumption here that running out of file space is
> `better' than rollong over and dying.
No, there's no such assumption there. Obviously, it is better _not_
to run out of file space than either to run out of file space or
rolling over and dying. But what is important in both of these failure
cases is not the fact that there is a failure (both cases have that)
but in the potential for recovery (where, going back to our original
issue of swap-unbacked memory, is impossible to do).
> It may be, but it can be
> catastrophically worse as well. Here's two examples that have
> happened to me, one in recent history, one longer ago.
[ two excellent examples of worse-is-better C/unix programming elided ...]
> In both those cases I would much, much rather have dealt with a crash,
> or even repeated crashes.
And I would have much rather dealt with a warning that said "hey, your
disk space is getting low", _before_ the crash, or to have had the
failure reported before it turned into a crash.
> The underlying issue here is that incompetent programmers write
> programs that don't work.
Interesting. Here is a truth that is logically flawed (or at least
not in its fully reduced form. The real truth is that _all_
programmers write programs that don't work, so whether or not they
are competent is irrelevant. At this point I would also normally
say that it is the competent programmers who look for the bugs in
their programs and crush them, but under that criterion we are all
incompetent, because after all that debug effort, there are still
bugs in the program.
> It isn't hard to check return codes or
> update critical files safely, but it is too hard for these people.
I don't think it's a question of being hard, but more of being
inconvenient. In order to check whether or not you are checking all
of the return codes, especially in cases where you are close to the
edge in resource-availability, you have to configure your test cases
to _be_ at close to the edge of resource-availability. That's not
hard, just inconvenient. Those who do so tend to have more solid
a program.
> Similarly it isn't hard to set the heap limit for your application to
> physmem + swap - 1GB (if it's all the machine will run) or something
> equivalent if it's not. But it is too hard for these people.
See above. Not hard, inconvenient.
> I can see where Duane is coming from: he's an implementor and probably
> has a large number of more-or-less incompetent users who complain
> about his application because they're too incompetent to set the heap
> size limit that ACL provides. He probably never hears from the ones
> that set it correctly (`hey Duane, my program worked, thanks!').
Actually, we do. And some of the users that complain about our
application are some of the most competent ones! In fact, I see your
name in our spr database 30 times, and that's only as the primary
contact for the spr. I certainly wouldn't call you incompetent.
But in fact, many of these complaints tend to make our product better,
so I certainly don't mind them.
> And
> his users pay his salary, so killing them is not really a good option.
> Finally, he's a nice guy, he'd probably have qualms about killing
> them even if there wasn't money involved.
> But I'm a Unix systems guy: big boots, manuals used as clubs and
> semi-automatic weapons are my stock in trade. I might have been nice
> once, but that was so long ago I can't remember, frankly. I'm just
> not interested in putting up with people who can't cope with setting
> heap limits or dealing with the issues raised by overcomitted memory
> which can't be avoided: I just want to get paid. The Sig-Sauer P226 I
> keep in the tape safe deals with them effectively: it's a little messy
> afterwards, but they don't suffer.
All this talk of killing, here. How do you _really_ feel about
customers?
:-)
> There's also the underlying point that this fear of overcommit is kind
> of a medieval attitude (this isn't meant as an insult...). In the
> middle ages, people had really strict ideas about money: don't lend
> more than you have, don't lend at interest, don't make promises you
> can't keep.
Call me Sir Duane, then.
[ examples of insurance companies and banks overcommitting ]
There are always risks when insuring or ensuring something.
These risks depend on the backing that is available, which
might invclude the probability that it might happen.
As an example, we California ski boat owners never bothered
winterizing our boats before 1990, because it never froze.
But in 1990 (I think, or it may have been 1991) we had a huge
freeze, on Christmas week pipes started bursting and had to be
fixed immediately, but we never thought any more about it until
the next spring, when we got out to the lake and our engine
compartment filled up with brown foam (the engine block, which
had still had water in it which had frozen, had cracked, thus
mixing a nice oil-and-water shake for us). We had to be towed
back to the shore. Amazingly, our insurance company was one of
two in California which payed out to have the engine fixed.
It went out of business the next year, but it had had the backing
for the disaster which it needed before it went under.
Several lessons:
1. The insurance company, though not able to weather the financial
drain, did seem to have the backing to pay out to its customers.
I don't know if all of its customers got their engine rebuild paid
for (rebulds which half the boats in California had to do), but they
paid for ours, and thus left us cleanly taken care of ("closed", in
the NFS sense of your second example).
2. It would have been nice not to have found out about the failure
out on the lake, where we had to receive a two. But, working backwards,
a. People on lakes are usually happy to give others a tow, because
they know that their turn will come
b. The circumstances under which the failure occurred were not tested
for by me; although I always test my engine out in dry-dock (by
attaching a hose to the engine and running it with water running
through the hose) it did not show the problem because the engine
was not stressed, and so the cracked block did not leak. I.e.
we were not close to the edge.
c. It was sheer laziness on our part that I did not perform the
winterizing that was required, coupled with the low risk of
it happening. The fact that it happened once raised the risk
level immediately.
3. We now always winterize our boat. We learn from the failures
how to not encounter the failures.
--
Duane Rettig ·····@franz.com Franz Inc. http://www.franz.com/
555 12th St., Suite 1450 http://www.555citycenter.com/
Oakland, Ca. 94607 Phone: (510) 452-2000; Fax: (510) 452-0182
Duane Rettig <·····@franz.com> writes:
> ··········@tfeb.org (Tim Bradshaw) writes:
>
> > which can't be avoided: I just want to get paid. The Sig-Sauer P226 I
> > keep in the tape safe deals with them effectively: it's a little messy
> > afterwards, but they don't suffer.
>
> All this talk of killing, here. How do you _really_ feel about
> customers?
>
> :-)
Hey, at least he has good taste in pistols.
--
I wouldn't mind the rat race so much if it wasn't for all the damn cats.
* Duane Rettig wrote:
>> Similarly it isn't hard to set the heap limit for your application to
>> physmem + swap - 1GB (if it's all the machine will run) or something
>> equivalent if it's not. But it is too hard for these people.
> See above. Not hard, inconvenient.
If setting a heap limit based on a trivial computation is considered
inconvenient then, well, what can I say?
--tim
Tim Bradshaw <···@cley.com> writes:
> * Duane Rettig wrote:
> >> Similarly it isn't hard to set the heap limit for your application to
> >> physmem + swap - 1GB (if it's all the machine will run) or something
> >> equivalent if it's not. But it is too hard for these people.
>
> > See above. Not hard, inconvenient.
>
> If setting a heap limit based on a trivial computation is considered
> inconvenient then, well, what can I say?
This quip is somewhat out of context from the original discussion.
It is obviously not the setting of a heap limit that is inconvenient
for a user, but what is inconvenient is actually taking the
application _to_ that limit many times and under differing
circumstances in order to _test_ whether the hitting of the limit is
fatal or otherwise traumatic.
--
Duane Rettig ·····@franz.com Franz Inc. http://www.franz.com/
555 12th St., Suite 1450 http://www.555citycenter.com/
Oakland, Ca. 94607 Phone: (510) 452-2000; Fax: (510) 452-0182
* Duane Rettig wrote:
> This quip is somewhat out of context from the original discussion.
> It is obviously not the setting of a heap limit that is inconvenient
> for a user, but what is inconvenient is actually taking the
> application _to_ that limit many times and under differing
> circumstances in order to _test_ whether the hitting of the limit is
> fatal or otherwise traumatic.
If the heap limit is set safely (and this isn't very hard to do) then
it will not be traumatic for the OS, since there will be VM spare. It
will probably be traumatic for the application, but running out of
heap is going to be so anyway. Presumably a runtime can be written
such that when the heap limit is hit it offers the option of
allocating more in order to clean up things (which can be done since
the heap limit has been set safely).
Of course something *else* running on that OS image could always come
along and cause the heap limit not to be as safe as it looks, by
eating vast amounts of memory, so in some theoretical sense it's not,
in fact, safe. Disks can die as well.
--tim
Tim Bradshaw wrote:
> There's also the underlying point that this fear of overcommit is kind
> of a medieval attitude (this isn't meant as an insult...).
...
> Similarly, software
> people need to get used to the idea of systematic overcommit of many
> kinds of resources, and doing the analysis which lets you make these
> bets. The renaissance is here already.
Well expressed. One thing to remember, though, is that customers who
originally asked for overcommit were doing it to save money -- they
didn't want to waste an entire 1G disk for swap they never used. So the
vendors said "OK, we'll add an option to overcommit VM, as long as you
NEVER TRY TO USE IT ALL. By the way, when you do, and bad things happen,
DON'T BLAME US!"
Time passes, the predictable happens, and people criticizes the OS
vendors for the gracelessness of the failures that ensue. The economics
would have been different if those early customers had said "...and we
want your OS to fail gracefully when we overdraw, and we'll pay for the
development."
This is in no way intended to explain AIX's particularly demonic
strategy of killing processes in order of size, making no exception for
X. That's just punitive.
--
Cameron MacKinnon
Toronto, Canada
* Duane Rettig wrote:
> The problem is not how the system performs when one has the memory,
> the problem is in the guarantee. the old addage "nobody will ever
> need more than 840K of memory" pops into mind. The issue is that
> when (not if) the machine runs out of memory, those comptetently written
> C programs are _going_ to break!
In the case of filesystem snapshotting, then yes, it is, and that's
just fine. You know in advance that you're racing the changes in the
filesystem you're snapshotting, because you probably don't have enough
disk to keep a copy of it even if you wanted to. There are other
applications like this, of course (though I think fs snapshots do the
most spectacular address space allocation of anything I can easily
think of).
> So however large the file is, if just a second helping of that file size
> can be allocated for swap space, then you're covered, right? If the
> file might grow or be filled in arbitrarily, then you're going to run
> out of file space as well as virtual memory space, but guess which one
> will give you the unintelligible error diagnostic?
Think *large* sparse file. Quite probably larger than the filesystem
its in. No, you can't allocate another copy for swap: you've just
spent a whole bunch of money on the big disk array to hold the file,
and the last thing you want is another big disk array to hold the
swap.
--tim
Tim Bradshaw <···@cley.com> writes:
> * Duane Rettig wrote:
> > So however large the file is, if just a second helping of that file size
> > can be allocated for swap space, then you're covered, right? If the
> > file might grow or be filled in arbitrarily, then you're going to run
> > out of file space as well as virtual memory space, but guess which one
> > will give you the unintelligible error diagnostic?
>
> Think *large* sparse file. Quite probably larger than the filesystem
> its in. No, you can't allocate another copy for swap: you've just
> spent a whole bunch of money on the big disk array to hold the file,
Huh? Is this file sparse or not? If it is sparse, then it is sparse,
and not all of the blocks are filled in.
> and the last thing you want is another big disk array to hold the
> swap.
Of course not. All you need is what will hold the same amount as the
original (sparse) file.
--
Duane Rettig ·····@franz.com Franz Inc. http://www.franz.com/
555 12th St., Suite 1450 http://www.555citycenter.com/
Oakland, Ca. 94607 Phone: (510) 452-2000; Fax: (510) 452-0182
* Duane Rettig wrote:
> Huh? Is this file sparse or not? If it is sparse, then it is sparse,
> and not all of the blocks are filled in.
yes it's sparse.
> Of course not. All you need is what will hold the same amount as the
> original (sparse) file.
If you knew how much of the file (or equivalently the address space)
you needed to use then there would be no problem.
--tim
Tim Bradshaw <···@cley.com> writes:
> * Duane Rettig wrote:
>
> > Huh? Is this file sparse or not? If it is sparse, then it is sparse,
> > and not all of the blocks are filled in.
>
> yes it's sparse.
>
> > Of course not. All you need is what will hold the same amount as the
> > original (sparse) file.
>
> If you knew how much of the file (or equivalently the address space)
> you needed to use then there would be no problem.
And if you didn't know how much of the file you needed to use there
would be no problem.
So what's the problem?
--
Duane Rettig ·····@franz.com Franz Inc. http://www.franz.com/
555 12th St., Suite 1450 http://www.555citycenter.com/
Oakland, Ca. 94607 Phone: (510) 452-2000; Fax: (510) 452-0182
* Duane Rettig wrote:
> And if you didn't know how much of the file you needed to use there
> would be no problem.
I think you don't understand the situation I'm describing, probably
because I've described it badly. It's a classic overcommit situation:
There is a large sparse file which is being mapped to memory. There
is not enough FS capacity for the file if all its pages were
allocated. You don't know which pages will be allocated, or (more
importantly) *how many*, because it depends on data you can not know
in advance.
If the system ever tries to make good on its commitment, it will fail.
It would be possible, but very uneconomic to provide enough FS space
to back the file even if it were not sparse. It would be possible,
but uneconomic, to provide enough swap space to back the file even
when it is sparse (uneconomic because you need to back it with
reliable disk, which needs to be as large as the disk space you
already have).
So you live with the risk.
--tim
Duane Rettig wrote:
> Tim Bradshaw <···@cley.com> writes:
>
>
>>* Duane Rettig wrote:
>>
>>
>>>In short, don't go the route of allocating virtual swap; it will
>>>inevitably lead to disaster. The rub is, that disaster will likely
>>>happen on a lisp, because it will be one of the few systems that can
>>>take an application to such large memory requirements, and so thus Lisp
>>>will be accused for the failure.
>>
>>I don't really have time or energy to disagree with you, but this made
>>me laugh. I guess you've never made a snapshot of a large file system
>>if you think typical Lisp's memory requirements are large.
>
>
> I'm not talking about typical Lisp, but about some Lisp applications -
> real ones. The fact that Lisp allows an application to grow as large
> as the operating system will let it means that it thus can, and if/when
> it does it will run into resource problems not encountered by apps written
> in languages that don't scale well.
That some sysadmin somewhere thinks he can run an application requiring
resources on a box that doesn't have them is no surprise. That he tries
to save money with this economical new "virtual swap" (is that "virtual
virtual memory"?) is no real surprise either. Nor that, like all cheque
kiting schemes, this one will come to a bad end if you end up overdrawn.
Now that it's bitten you once, it's a simple matter to check the
configuration if you're worried. On my OS, it's a boolean in
/proc/sys/vm/overcommit_memory
Allowing overcommit of virtual memory is now a common feature, though I
suspect that no vendors have made it the default. I'd be very surprised
if Lisp users were the ones who drove the vendors to incorporate this
feature.
> The correlary to Murphy's law which states that computer programs tend
> to expand to fill available memory is probably apropos here, but since
> Lisp apps scale so well they tend to hit memory limits much faster than
> other apps, which are just now finding their way above the 32-bit range.
[rolls eyes]
--
Cameron MacKinnon
Toronto, Canada
Cameron MacKinnon <··········@clearspot.net> writes:
> Duane Rettig wrote:
> > Tim Bradshaw <···@cley.com> writes:
> >
>
> >>* Duane Rettig wrote:
> >>
> >>
> >>>In short, don't go the route of allocating virtual swap; it will
> >>>inevitably lead to disaster. The rub is, that disaster will likely
> >>>happen on a lisp, because it will be one of the few systems that can
> >>>take an application to such large memory requirements, and so thus Lisp
> >>>will be accused for the failure.
> >>
> >>I don't really have time or energy to disagree with you, but this made
> >>me laugh. I guess you've never made a snapshot of a large file system
> >>if you think typical Lisp's memory requirements are large.
> > I'm not talking about typical Lisp, but about some Lisp applications
> > -
>
> > real ones. The fact that Lisp allows an application to grow as large
> > as the operating system will let it means that it thus can, and if/when
> > it does it will run into resource problems not encountered by apps written
> > in languages that don't scale well.
>
> That some sysadmin somewhere thinks he can run an application
> requiring resources on a box that doesn't have them is no
> surprise. That he tries to save money with this economical new
> "virtual swap" (is that "virtual virtual memory"?) is no real surprise
> either. Nor that, like all cheque kiting schemes, this one will come
> to a bad end if you end up overdrawn.
>
>
> Now that it's bitten you once, it's a simple matter to check the
> configuration if you're worried. On my OS, it's a boolean in
> /proc/sys/vm/overcommit_memory
Yes, if you know to do so. But many people think of this bad-check-writing
as a Good Thing (I'm reminded of the story about the guy who ignored
the notices in the mail from the bank about being overdrawn, and
who wrote checks anyway because he still had checks in the checkbook...)
That's why I jumped into this thread, to remind people that they are
talking about using checks without considering the account balance.
> Allowing overcommit of virtual memory is now a common feature, though
> I suspect that no vendors have made it the default.
NeXT had done so. I think you may be correct in other cases.
> I'd be very
> surprised if Lisp users were the ones who drove the vendors to
> incorporate this feature.
No, I think not. I am not certain, but it was likely a result of
various vendors trying to solve the vfork problem. Vfork, as you
know, was a BSD invention to save memory when forking by copying only
the portions of the program that were being executed in the child
process between the fork and the exec, but it started getting too
complicated when that code relied on various other parts of memory
that had been mapped in for the parent, so virtual swap + Copy-on-write
was a clever way of doing this without having to decide what portion of
the child had to be copied.
> > The correlary to Murphy's law which states that computer programs tend
> > to expand to fill available memory is probably apropos here, but since
> > Lisp apps scale so well they tend to hit memory limits much faster than
> > other apps, which are just now finding their way above the 32-bit range.
>
> [rolls eyes]
8-)
--
Duane Rettig ·····@franz.com Franz Inc. http://www.franz.com/
555 12th St., Suite 1450 http://www.555citycenter.com/
Oakland, Ca. 94607 Phone: (510) 452-2000; Fax: (510) 452-0182
Duane Rettig wrote:
> Cameron MacKinnon <··········@clearspot.net> writes:
>>Now that it's bitten you once, it's a simple matter to check the
>>configuration if you're worried. On my OS, it's a boolean in
>>/proc/sys/vm/overcommit_memory
>
>
> Yes, if you know to do so. But many people think of this bad-check-writing
> as a Good Thing (I'm reminded of the story about the guy who ignored
> the notices in the mail from the bank about being overdrawn, and
> who wrote checks anyway because he still had checks in the checkbook...)
> That's why I jumped into this thread, to remind people that they are
> talking about using checks without considering the account balance.
Ah, reviewing the thread, I see what you mean. David Steuber's original
confusion led to all this.
David said:
> I mean in the sense of reserving all the available address space for
> heap. From the way I gather that malloc works on some systems, you
> can ask for the memory and get a pointer to it. So far, so good.
> Once you start to use that memory, the OS then tries its darndest to
> give it to you. Bad things happen if there is not enough physical
> memory to back the virtual/logical memory. (I hate it when terms are
> used imprecisely. I no longer know what is being talked about.).
In answer to that, David, malloc won't promise something it can't
deliver on any UNIX but NeXT, unless the sysadmin has already flipped a
big red switch that says "DANGER". Other aspects of the complete
response have already been addressed in this thread.
--
Cameron MacKinnon
Toronto, Canada
Cameron MacKinnon <··········@clearspot.net> writes:
> David said:
> > I mean in the sense of reserving all the available address space for
> > heap. From the way I gather that malloc works on some systems, you
> > can ask for the memory and get a pointer to it. So far, so good.
> > Once you start to use that memory, the OS then tries its darndest to
> > give it to you. Bad things happen if there is not enough physical
> > memory to back the virtual/logical memory. (I hate it when terms are
> > used imprecisely. I no longer know what is being talked about.).
>
> In answer to that, David, malloc won't promise something it can't
> deliver on any UNIX but NeXT, unless the sysadmin has already flipped
> a big red switch that says "DANGER". Other aspects of the complete
> response have already been addressed in this thread.
Ok, that sounds sane. So if malloc succeeds, it is then safe to call
memset to zero out all the memory you just asked for. Thrashing may
ensue, but the system won't keel over and die.
I still don't understand the underlying mechanics of CMUCL and SBCL
memory management, but that can wait.
So let's say that Lisp foo has asked for and received all available
heap. This should mean that a foreign function call that needs to
allocate some heap space on its own should fail and signal some sort
of memory related error rather than causing complete crash of the Lisp
app, right?
Malloc's behavior would also explain the results I got with my little
mallocator program. The Linux machine I tested on has a fixed size
swap partition plus RAM. There is no more physical memory available
for applications. OS X uses swap files, so until disk space runs out,
all available memory can be consumed over time.
--
I wouldn't mind the rat race so much if it wasn't for all the damn cats.
* Cameron MacKinnon wrote:
> That some sysadmin somewhere thinks he can run an application
> requiring resources on a box that doesn't have them is no
> surprise. That he tries to save money with this economical new
> "virtual swap" (is that "virtual virtual memory"?) is no real surprise
> either. Nor that, like all cheque kiting schemes, this one will come
> to a bad end if you end up overdrawn.
This is what bugs me about all this stuff. Duane seems to be
proposing fixes for incompetence. The only fix for incompetence that
works is a bullet in the head.
--tim
Tim Bradshaw wrote:
> This is what bugs me about all this stuff. Duane seems to be
> proposing fixes for incompetence. The only fix for incompetence that
> works is a bullet in the head.
Are you suggesting that dying makes one competent? ;-D
This reminds me of an idea I had in college - evolutionary programming.
What I wanted to do was have a team of upperclassment in the CS
department go through the computer labs when freshmen were working on
programming projects, and read their code over their shoulders, watching
not only how they code but how they go about coding. We'd carry 2x4's,
and hit students in the head for making mistakes. The idea is that
they'd learn how to program well out of fear.
Sadly, I was the only one there who was willing to take action on his
politically incorrect ideas, and I didn't like the idea of having a cell
to myself, so I passed on the idea, and incompetent programmers still go
through college without a severe beating.