I need to allocate a large array of unsigned bytes for testing a program
I am developing. I am having problems allocating this array using
clisp, cmucl, and gcl. My prefered development platform is CMU-CL due
to its verbose help in optimizing my code. This is the error it gives
me when I try to allocate a 30,000,000 element array. Ideally, I would
eventually like to be able to up this to something like 500,000,000
bytes. Anyway this is the output:
-----------------------------------------------------------------------
Loaded subsystems:
Python 1.0, target Intel x86
CLOS based on PCL version: September 16 92 PCL (f)
* (ext:gc-off)
NIL
* (progn (setf tmp (make-array 30000000 :element-type 'unsigned-byte))
* 't)
Warning: Declaring TMP special.
Error in function UNIX::SIGSEGV-HANDLER: Segmentation Violation at
#x805D257.
Restarts:
0: [ABORT] Return to Top-Level.
Debug (type H for help)
(UNIX::SIGSEGV-HANDLER #<unavailable-arg>
#<unavailable-arg>
#.(SYSTEM:INT-SAP #x500FFBB4))
0] 0
--------------------------------------------------------------------
Does anyone know what would be causing this? A fix? I can allocate
arrays of this size in one of these implementations of Lisp can't I?
I would assume that a segmentation fault is indicative of a bug in
CMUCL. I wouldn't mind tracking down and attempting to fix this one
myself, but I do need a point in the right direction.
Thanks,
--
Kenneth P. Turvey <·······@pug1.SprocketShop.com>
If you pick up a starving dog and make him prosperous, he will not
bite you. This is the principal difference between a man and a dog.
-- Mark Twain
·······@www.sprocketshop.com (Kenneth P. Turvey) writes:
<
< I need to allocate a large array of unsigned bytes for testing a program
< I am developing. I am having problems allocating this array using
< clisp, cmucl, and gcl. My prefered development platform is CMU-CL due
< to its verbose help in optimizing my code. This is the error it gives
< me when I try to allocate a 30,000,000 element array. Ideally, I would
< eventually like to be able to up this to something like 500,000,000
< bytes. Anyway this is the output:
You must have lots of memory! Check out the constant
`array-total-size-limit'. I imagine you just need to put in a size
check or something. I can't make an array anywhere near the size of
array-total-size-limit, as I run out of room too fast - so a call to
the operating system may be necessary on top of a check to
array-total-size-limit.
In any case, you should (I think) get a storage-condition if you
cannot allocate the requested amount memory.
On 22 Jul 1998 07:07:01 -0400,
Steve Gonedes <········@worldnet.att.net> wrote:
>You must have lots of memory! Check out the constant
>`array-total-size-limit'. I imagine you just need to put in a size
>check or something. I can't make an array anywhere near the size of
>array-total-size-limit, as I run out of room too fast - so a call to
>the operating system may be necessary on top of a check to
>array-total-size-limit.
I'm not sure I understand this. Are you saying that I should be able to
edit array-total-size-limit or that I am constrained by it? I'll take a
look at the source. Thanks for the pointer.
I don't have an enormous amount of memory, but I did add a swap file to
virtual memory for this purpose. I don't mind the overhead involved in
page faults for this purpose, so virual memory should be good enough.
Thanks,
--
Kenneth P. Turvey <·······@pug1.SprocketShop.com>
Over grown military establishments are under any form of government
inauspicious to liberty, and are to be regarded as particularly
hostile to republican liberty.
-- George Washington
In article <······················@www.sprocketshop.com>,
Kenneth P. Turvey <·······@www.sprocketshop.com> wrote:
>I'm not sure I understand this. Are you saying that I should be able to
>edit array-total-size-limit or that I am constrained by it? I'll take a
>look at the source. Thanks for the pointer.
It's a predefined constant that indicates the limit of the implementation,
so you're constrained by it. So if you need to deal with more data than
this, you'll need to design your own data structure that spreads the data
across multiple arrays. Here's a simple implementation for vectors (since
this is one-dimensional, I'm using ARRAY-DIMENSION-LIMIT rather than
ARRAY-TOTAL-SIZE-LIMIT -- presumably A-D-L is no more than A-T-S-L):
(defun make-large-vector (size)
(let* ((n (ceiling size array-dimension-limit))
(new-array (make-array n)))
(dotimes (i n)
(setf (aref new-array i) (make-array array-dimension-limit)))
new-array))
(defun large-vector-ref (lv index)
(multiple-value-bind (i j) (floor index array-dimension-limit)
(aref (aref lv i) j)))
--
Barry Margolin, ······@bbnplanet.com
GTE Internetworking, Powered by BBN, Cambridge, MA
*** DON'T SEND TECHNICAL QUESTIONS DIRECTLY TO ME, post them to newsgroups.
From: Martti Halminen
Subject: Re: Problem with Large arrays
Date:
Message-ID: <35B59DBA.62CC@dpe.fi>
Kenneth P. Turvey wrote:
>
> I need to allocate a large array of unsigned bytes for testing a program
> I am developing. I am having problems allocating this array using
> clisp, cmucl, and gcl. My prefered development platform is CMU-CL due
> to its verbose help in optimizing my code. This is the error it gives
> me when I try to allocate a 30,000,000 element array. Ideally, I would
> eventually like to be able to up this to something like 500,000,000
> bytes. Anyway this is the output:
<snip>
> Does anyone know what would be causing this? A fix? I can allocate
> arrays of this size in one of these implementations of Lisp can't I?
>
Take a look at the value of ARRAY-TOTAL-SIZE-LIMIT in the various
implementations. I don't have CMU-CL handy, but ACL 4.3.2 gives
16777216, and I'd suspect the others to be in the same ballpark.
If the implementation doesn't support large enough arrays, you'll
probably have to write your own low-level array implementation, or find
some other way to bypass this limit.
--
________________________________________________________________
^. Martti Halminen
/ \`. Design Power Europe Oy
/ \ `. Tekniikantie 12, FIN-02150 Espoo, Finland
/\`. \ | Tel:+358 9 4354 2306, Fax:+358 9 455 8575
/__\|___\| ······················@dpe.fi http://www.dpe.fi
On 23 Jul 1998 17:20:51 GMT, forcer <······@mindless.com> wrote:
Referring to ARRAY-TOTAL-SIZE-LIMIT
>
>CLISP 19970925 16777216
>GCL 2.2.2 2147483647
>
>(platform is Linux 2.0.35/AMD K6)
>
GCL doesn't actually seem to let you allocate an array of this size.
When I try to allocate a 500,000,000 byte array I receive this error:
Correctable error: Relocatable blocks exhausted.
Currently, 7100 pages are allocated.
Use ALLOCATE-RELOCATABLE-PAGES to expand the space.
Signalled by MAKE-HUGE-HASH-TABLE.
If continued: Continues execution.
Broken at MAKE-ARRAY. Type :H for Help.
I have tried to use ALLOCATE-RELOCATABLE-PAGES and it will allow me to
bring it up to 7520, but no higher. This is apparently not sufficient
to hold the array.
CLISP just dies and says that there isn't any more room for Lisp
objects.
--
Kenneth P. Turvey <·······@pug1.SprocketShop.com>
Ninety percent of the politicians give the other ten percent a bad name.
-- Henry Kissinger
Kenneth P. Turvey wrote:
>
> On 23 Jul 1998 09:43:57 +0200,
> Marco Antoniotti <·······@galvani.parades.rm.cnr.it> wrote:
> >CMUCL 18b 536870911
> >
>
> Great!! I'll have to get CMUCL 18b. Is it stable? I have tried various
You can't get 18b. There's only 18a and the current development
branch. 18b will come out sometime in the future. :-)
However, I find the development branch to be no worse than 18a, and it
has some other neat features like much better support for complex
numbers and multi-processing.
> things with using an array of arrays, but I've had little success. I
> keep getting segmentation faults in CMUCL 18a and Allegro dies after it
There is some upper limit on the size of the heap. I think it's 512 MB
now. If you want more, you'll need to compile your own version.
> GCL will allow arrays of the size that I need, but its garbage
> collection slows it down to a snails crawl. It also seems to allocate 2
> bytes for each element of the '(unsigned-byte 5) array.
I seriously doubt that gcl really supports 2^31 elements in an array
since an array of "pointers" would take 2^(31+4) bytes, and you can't do
that on a 32-bit machine.
> CLISP simply states that there isn't any more room for lisp objects and
> quits.
CLISP used to have a limit of 16 MB of heap. Later versions, I think,
have increased this somewhat.
Ray
--
---------------------------------------------------------------------------
----> Raymond Toy ····@mindspring.com
http://www.mindspring.com/~rtoy
"R. Toy" <····@mindspring.com> writes:
> Kenneth P. Turvey wrote:
> >
> > On 23 Jul 1998 09:43:57 +0200,
> > Marco Antoniotti <·······@galvani.parades.rm.cnr.it> wrote:
> > >CMUCL 18b 536870911
> > >
> >
> > Great!! I'll have to get CMUCL 18b. Is it stable? I have tried various
>
> You can't get 18b. There's only 18a and the current development
> branch. 18b will come out sometime in the future. :-)
My mistake!
I ran the test on a Sun Ultra 1.
Cheers
--
Marco Antoniotti ===========================================
PARADES, Via San Pantaleo 66, I-00186 Rome, ITALY
tel. +39 - (0)6 - 68 80 79 23, fax. +39 - (0)6 - 68 80 79 26
http://www.parades.rm.cnr.it
>>>>> "Kenneth" == Kenneth P Turvey <·······@www.sprocketshop.com> writes:
Kenneth> I need to allocate a large array of unsigned bytes for
Kenneth> testing a program I am developing. I am having problems
Kenneth> allocating this array using clisp, cmucl, and gcl. My
Kenneth> prefered development platform is CMU-CL due to its
Kenneth> verbose help in optimizing my code. This is the error it
Kenneth> gives me when I try to allocate a 30,000,000 element
Kenneth> array. Ideally, I would eventually like to be able to up
Kenneth> this to something like 500,000,000 bytes. Anyway this is
Kenneth> the output:
I think some versions of CMUCL have a 64 MB heap size so your 30
million element array of unsigned-byte's probably uses 120 million
bytes.
Try a newer version where the heap has been increased to a max of 512
MB. For your 500 000 000 bytes, you'll probably have to compile your
own version or convince the CMUCL guys to bump the limit. Shouldn't
be hard to convice them. :-)
Ray
On 25 Jul 1998 23:30:57 -0400, Raymond Toy <···@rtp.ericsson.se> wrote:
>
>I think some versions of CMUCL have a 64 MB heap size so your 30
>million element array of unsigned-byte's probably uses 120 million
>bytes.
>
>Try a newer version where the heap has been increased to a max of 512
>MB. For your 500 000 000 bytes, you'll probably have to compile your
>own version or convince the CMUCL guys to bump the limit. Shouldn't
>be hard to convice them. :-)
Can anyone tell me where this limit is defined? Also, any tips for
someone trying to build CMUCL from the sources for the first time would be
greatly appreciated. Is there more than one reference for this procedure,
or is the one at www.cons.org it. Not that this document is unusable
or anything... just having more than one sometimes helps.
Thanks,
--
Kenneth P. Turvey <·······@pug1.SprocketShop.com>
A computer lets you make more mistakes faster than any invention in
human history with the possible exceptions of handguns and tequila.
-- Mitch Ratliffe, Technology Review, April, 1992
>>>>> "Kenneth" == Kenneth P Turvey <·······@www.sprocketshop.com> writes:
Kenneth> On 25 Jul 1998 23:30:57 -0400, Raymond Toy <···@rtp.ericsson.se> wrote:
>>
>> I think some versions of CMUCL have a 64 MB heap size so your 30
>> million element array of unsigned-byte's probably uses 120 million
>> bytes.
>>
>> Try a newer version where the heap has been increased to a max of 512
>> MB. For your 500 000 000 bytes, you'll probably have to compile your
>> own version or convince the CMUCL guys to bump the limit. Shouldn't
>> be hard to convice them. :-)
Kenneth> Can anyone tell me where this limit is defined? Also, any tips for
It's in src/lisp/x86-validate.h. You can find out what you have by
doing
* (- (lisp::dynamic-1-space-start) (lisp::dynamic-0-space-start))
536870912
So I have 512 MB of heap.
Kenneth> someone trying to build CMUCL from the sources for the first time would be
Kenneth> greatly appreciated. Is there more than one reference for this procedure,
Kenneth> or is the one at www.cons.org it. Not that this document is unusable
Kenneth> or anything... just having more than one sometimes helps.
There is more than one but I can't remember where they are.
I think it's relatively easy to change the heap size: edit
x86-validate.h appropriately. Rebuilding CMUCL won't be to hard in
this case.
However, I'd recommend joining ·········@cons.org to get more details
and help.
Ray
On 27 Jul 1998 19:13:41 -0400, Raymond Toy <···@rtp.ericsson.se> wrote:
>>>>>> "Kenneth" == Kenneth P Turvey <·······@www.sprocketshop.com> writes:
>
>It's in src/lisp/x86-validate.h. You can find out what you have by
>doing
>
>* (- (lisp::dynamic-1-space-start) (lisp::dynamic-0-space-start))
>536870912
>
>So I have 512 MB of heap.
That would be the problem. When I do this, I get something around 67
meg.
>I think it's relatively easy to change the heap size: edit
>x86-validate.h appropriately. Rebuilding CMUCL won't be to hard in
>this case.
I checked the source from the CVS tree and it defaults to 512 MB of
heap.
>
>However, I'd recommend joining ·········@cons.org to get more details
>and help.
Done.
Thanks,
--
Kenneth P. Turvey <·······@pug1.SprocketShop.com>
The whole aim of practical politics is to keep the populace alarmed
[and hence clamorous to be led to safety] by menacing it with an
endless series of hobgoblins, all of them imaginary.
-- H.L. Mencken