Does anyone have any good advice on how to make arrays (or something
like looks like them at a high level) when the number of elements is
greater than total-array-size-limit (e.g. 2^24-1 on openMCL)?
Is total-array-size-limit something to consider before going too far
with any particular lisp implementation?
My element-types are fixnum and I'm working with a c-based library via
ffi that I'm trying to slowly port to lisp.
Thanks for any help.
paul
From: Wade Humeniuk
Subject: Re: working with very large arrays (newbie)
Date:
Message-ID: <pMf5e.6919$vt1.128@edtnps90>
Paul wrote:
> Does anyone have any good advice on how to make arrays (or something
> like looks like them at a high level) when the number of elements is
> greater than total-array-size-limit (e.g. 2^24-1 on openMCL)?
>
> Is total-array-size-limit something to consider before going too far
> with any particular lisp implementation?
>
> My element-types are fixnum and I'm working with a c-based library via
> ffi that I'm trying to slowly port to lisp.
Sure! there is always a way,
for example, making an array of array-total-size-limit x array-total-size-limit
(make-array array-total-size-limit
:initial-contents (loop repeat array-total-size-limit
collect (make-array array-total-size-limit
:element-type 'unsigned-byte
:initial-element 0)))
Careful, do not try this, it is very large!
Of course you will have to build your own equivalent of aref.
Wade
Wade Humeniuk <··················@telus.net> writes:
> (make-array array-total-size-limit
> :initial-contents (loop repeat array-total-size-limit
> collect (make-array array-total-size-limit
> :element-type 'unsigned-byte
> :initial-element 0)))
>
> Careful, do not try this, it is very large!
A cursory glance suggests this is larger than the addressable memory
space. I suppose that satisfies the condition "very large" :-)
--
An ideal world is left as an excercise to the reader.
--- Paul Graham, On Lisp 8.1
David Steuber wrote:
> Wade Humeniuk <··················@telus.net> writes:
>
>
>>(make-array array-total-size-limit
>> :initial-contents (loop repeat array-total-size-limit
>> collect (make-array array-total-size-limit
>> :element-type 'unsigned-byte
>> :initial-element 0)))
>>
>>Careful, do not try this, it is very large!
>
>
> A cursory glance suggests this is larger than the addressable memory
> space. I suppose that satisfies the condition "very large" :-)
>
Yes - that would be "way large" from my perspective. Everything I'm
doing would fit in addressable memory.
Thanks everyone!
paul
In article <······················@rcn.net>, Paul <······@rcn.com>
wrote:
> David Steuber wrote:
>
> > Wade Humeniuk <··················@telus.net> writes:
> >
> >
> >>(make-array array-total-size-limit
> >> :initial-contents (loop repeat array-total-size-limit
> >> collect (make-array array-total-size-limit
> >> :element-type 'unsigned-byte
> >> :initial-element 0)))
> >>
> >>Careful, do not try this, it is very large!
> >
> >
> > A cursory glance suggests this is larger than the addressable memory
> > space. I suppose that satisfies the condition "very large" :-)
> >
>
> Yes - that would be "way large" from my perspective. Everything I'm
> doing would fit in addressable memory.
With CLOS you could design classes that support VLAs that use one
implementation when they fit in memory, and another when they don't.
The latter could copy portions to files. And with foreign function
calling, you could make use of C code that uses mmap() to optimize
access to the stuff in files.
--
Barry Margolin, ······@alum.mit.edu
Arlington, MA
*** PLEASE post questions in newsgroups, not directly to me ***
Paul <······@rcn.com> writes:
> Does anyone have any good advice on how to make arrays (or something
> like looks like them at a high level) when the number of elements is
> greater than total-array-size-limit (e.g. 2^24-1 on openMCL)?
You'll have to implement your own data structure.
> Is total-array-size-limit something to consider before going too far
> with any particular lisp implementation?
Given that ARRAY-TOTAL-SIZE-LIMIT may be as small as 1024, I'd say:
yes, definitely.
> My element-types are fixnum and I'm working with a c-based library via
> ffi that I'm trying to slowly port to lisp.
Note that fixnums are not C ints; it would be more something like
(signed-byte 32).
Perhaps you could keep the data in C, and use the FFI to access it.
--
__Pascal Bourguignon__ http://www.informatimago.com/
Nobody can fix the economy. Nobody can be trusted with their finger
on the button. Nobody's perfect. VOTE FOR NOBODY.