Greetings!
I was wondering if there would happen to be a quick way to figure out
whether a particular Lisp implementation boxes floating-point
(single-float / double-float / (complex single-float) / (complex
double-float)) numbers in arrays -- that is, without reading the
documentation :) I'd like to know in O(1) time if something like
(make-array n :element-type 'double-float)
produces an array which is bit-compatible with a C array of doubles, so
that I can pass it to a C function without copy-in & copy-out.
I would just read the documentation but I'd like an automatic test to
make maintenance easier (I don't want to subscribe to the devel lists
of ten different Lisp implementations, and the commercial ones may not
publish that information).
Best,
mfh
On Oct 13, 3:27 pm, ·············@gmail.com" <············@gmail.com>
wrote:
> I was wondering if there would happen to be a quick way to figure out
> whether a particular Lisp implementation boxes floating-point
> (single-float / double-float / (complex single-float) / (complex
> double-float)) numbers in arrays -- that is, without reading the
> documentation :)
On many Lisps you can get consing / memory usage info with TIME, but
not on all Lisps.
mfh
Technically, you should use the FFI that comes with your Lisp
implementation to create an array of doubles and then copy the lisp
array's values into that FFI created array. On multithreaded lisp
programs, there is no guarantee that another Lisp thread won't trigger
a garbage collect while another lisp thread has called into C. If that
happens, a compact garbage collector will move the array, and the
pointer that you passed to C will point to bad memory.
-Tim K.
pTymN wrote:
> Technically, you should use the FFI that comes with your Lisp
> implementation to create an array of doubles and then copy the lisp
> array's values into that FFI created array. On multithreaded lisp
> programs, there is no guarantee that another Lisp thread won't trigger
> a garbage collect while another lisp thread has called into C. If that
> happens, a compact garbage collector will move the array, and the
> pointer that you passed to C will point to bad memory.
Thanks for the tip, but I should point out that we've been considering
that very issue :) Some Lisps let users disable GC via some system-
dependent thing, in effect a WITH-GC-OFF form. This is the approach
taken by Matlisp for passing Lisp arrays into the BLAS and LAPACK.
I hesitate to use some an extreme approach because it necessitates
thread synchronization at every call to a foreign routine. Some Lisps
(ACL in particular) have an extra parameter to MAKE-ARRAY that tells
the GC not to move the array's memory around. On other Lisps, we
have a choice between using C arrays only, or Lisp arrays and copy
into C storage when passing them to a foreign routine.
Dear Lisp implementers: I like ACL's "static array" option, and it
would
make our work easier if you all would include it in your own Lisps :)
(I
also wish you would unbox floating-point complex numbers!)
Many thanks to all of you for your helpful ideas! I'm putting together
a
page that lists what I've found out through (TIME (MAKE-ARRAY 1000
:ELEMENT-TYPE 'DOUBLE-FLOAT)) and the like, and reading the
various implementations' documentation. It could be useful to some of
you so I'll post it when I'm done.
Best,
mfh
From: Marcin 'Qrczak' Kowalczyk
Subject: Re: Quick way to figure out if floats are boxed in arrays?
Date:
Message-ID: <87ejtbym4n.fsf@qrnik.zagroda>
·············@gmail.com" <············@gmail.com> writes:
> Some Lisps let users disable GC via some system-dependent thing, in
> effect a WITH-GC-OFF form. This is the approach taken by Matlisp for
> passing Lisp arrays into the BLAS and LAPACK.
A better interface is pinning a given array to its current address,
allowing garbage collection of other objects while this object is in
use by foreign code.
In some implementation other objects on the same page might get
implicitly kept alive. This allows for some leaks but it's still
better than disabling GC.
I don't know if any Lisps support that, but .NET does it that way.
--
__("< Marcin Kowalczyk
\__/ ······@knm.org.pl
^^ http://qrnik.knm.org.pl/~qrczak/
On Oct 14, 11:04 pm, Marcin 'Qrczak' Kowalczyk <······@knm.org.pl>
wrote:
> ·············@gmail.com" <············@gmail.com> writes:
> > Some Lisps let users disable GC via some system-dependent thing, in
> > effect a WITH-GC-OFF form. This is the approach taken by Matlisp for
> > passing Lisp arrays into the BLAS and LAPACK.
>
> A better interface is pinning a given array to its current address,
> allowing garbage collection of other objects while this object is in
> use by foreign code.
A number of Lisps support this idea, e.g. Allegro CL, ECL (?), GCL,
LispWorks. See the table I've been working on at:
http://www.cs.berkeley.edu/~mhoemmen/lisp-matrix/gc.html
that lists GC features (pinnable arrays, pausable GC, finalization,
weak pointers) of various CL implementations. Corrections would
be appreciated :)
mfh
From: Marcin 'Qrczak' Kowalczyk
Subject: Re: Quick way to figure out if floats are boxed in arrays?
Date:
Message-ID: <87r6x9zx9m.fsf@qrnik.zagroda>
·············@gmail.com" <············@gmail.com> writes:
> http://www.cs.berkeley.edu/~mhoemmen/lisp-matrix/gc.html
>
> that lists GC features (pinnable arrays, pausable GC, finalization,
> weak pointers) of various CL implementations. Corrections would
> be appreciated :)
http://www.haible.de/bruno/papers/cs/weak/WeakDatastructures-writeup.html
compares weak reference capabilities of various Lisps and some other
languages.
--
__("< Marcin Kowalczyk
\__/ ······@knm.org.pl
^^ http://qrnik.knm.org.pl/~qrczak/
············@gmail.com <············@gmail.com> wrote:
+---------------
| See the table I've been working on at:
| http://www.cs.berkeley.edu/~mhoemmen/lisp-matrix/gc.html
| that lists GC features (pinnable arrays, pausable GC, finalization,
| weak pointers) of various CL implementations. Corrections would
| be appreciated :)
+---------------
The column for "WITH-GC-OFF?" says that CMUCL has only
EXT:GC-{OFF,ON}, and only in CMUCL-19c. Actually, CMUCL
also has the SYSTEM:WITHOUT-GCING macro [args: (&body body)],
and has had all three of these since at least CMUCL-19d
[and probably earlier than that].
-Rob
-----
Rob Warnock <····@rpw3.org>
627 26th Avenue <URL:http://rpw3.org/>
San Mateo, CA 94403 (650)572-2607
From: Raymond Toy
Subject: Re: Quick way to figure out if floats are boxed in arrays?
Date:
Message-ID: <sxd3b9owhha.fsf@rtp.ericsson.se>
>>>>> "Rob" == Rob Warnock <····@rpw3.org> writes:
Rob> ············@gmail.com <············@gmail.com> wrote:
Rob> +---------------
Rob> | See the table I've been working on at:
Rob> | http://www.cs.berkeley.edu/~mhoemmen/lisp-matrix/gc.html
Rob> | that lists GC features (pinnable arrays, pausable GC, finalization,
Rob> | weak pointers) of various CL implementations. Corrections would
Rob> | be appreciated :)
Rob> +---------------
Rob> The column for "WITH-GC-OFF?" says that CMUCL has only
Rob> EXT:GC-{OFF,ON}, and only in CMUCL-19c. Actually, CMUCL
Rob> also has the SYSTEM:WITHOUT-GCING macro [args: (&body body)],
Rob> and has had all three of these since at least CMUCL-19d
Rob> [and probably earlier than that].
Which 19d would that be? The one that isn't out yet? :-)
My ancient copy of 17f has all three.
Ray
Raymond Toy <···········@ericsson.com> wrote:
+---------------
| >>>>> "Rob" == Rob Warnock <····@rpw3.org> writes:
| Rob> The column for "WITH-GC-OFF?" says that CMUCL has only
| Rob> EXT:GC-{OFF,ON}, and only in CMUCL-19c. Actually, CMUCL
| Rob> also has the SYSTEM:WITHOUT-GCING macro [args: (&body body)],
| Rob> and has had all three of these since at least CMUCL-19d
| Rob> [and probably earlier than that].
|
| Which 19d would that be? The one that isn't out yet? :-)
+---------------
Ahh, yez. That "19d" would be the typo that should have said "18d". ;-}
(Oops!)
+---------------
| My ancient copy of 17f has all three.
+---------------
I have one of those!!
$ head -4 cmucl.sgi/lib/subsystems/clx-library.sgif
FASL FILE output from target:clx/package.lisp.
Compiled Wednesday, 11/2/94 12:52:24 am EST on actin.aim.cs.cmu.edu
Compiler 1.0, Lisp pre-17f
Targeted for SGI, FASL version 6
$
And also a circa-1999 18b, but I couldn't run either of them
since they're MIPS/Irix binaries and I no longer have access
to that platform. (*sigh*)
-Rob
-----
Rob Warnock <····@rpw3.org>
627 26th Avenue <URL:http://rpw3.org/>
San Mateo, CA 94403 (650)572-2607
Hmm, maybe UPGRADED-ARRAY-ELEMENT-TYPE?
(upgraded-array-element-type 'double-float) ; if T then boxed?
Nicolas
From: Duane Rettig
Subject: Re: Quick way to figure out if floats are boxed in arrays?
Date:
Message-ID: <o0ejtbrj4j.fsf@gemini.franz.com>
Nicolas Neuss <··················@iwr.uni-heidelberg.de> writes:
> Hmm, maybe UPGRADED-ARRAY-ELEMENT-TYPE?
>
> (upgraded-array-element-type 'double-float) ; if T then boxed?
Since this answer was posed with a question mark, I'll solidify it to
a fine mush. If this expression returns T then the double-floats will
definitely be boxed. If it returns nil, then I am not sure if it a
priori dictates that the float is housed unboxed into a specialized
array made with that element-type. This is definitely the case in
Allegro CL (i.e. nil means unboxed), and I believe that most if not
all other CLs do the same thing (not specifically to store
double-floats unboxed into an array made with element-type
double-float, but to do so iff the above return from u-a-e-t is nil).
--
Duane Rettig ·····@franz.com Franz Inc. http://www.franz.com/
555 12th St., Suite 1450 http://www.555citycenter.com/
Oakland, Ca. 94607 Phone: (510) 452-2000; Fax: (510) 452-0182
Duane Rettig <·····@franz.com> wrote:
+---------------
| Nicolas Neuss <··················@iwr.uni-heidelberg.de> writes:
| > Hmm, maybe UPGRADED-ARRAY-ELEMENT-TYPE?
| > (upgraded-array-element-type 'double-float) ; if T then boxed?
|
| Since this answer was posed with a question mark, I'll solidify it to
| a fine mush. If this expression returns T then the double-floats will
| definitely be boxed. If it returns nil, then I am not sure if it a
| priori dictates that the float is housed unboxed into a specialized
| array made with that element-type. This is definitely the case in
| Allegro CL (i.e. nil means unboxed), and I believe that most if not
| all other CLs do the same thing (not specifically to store
| double-floats unboxed into an array made with element-type
| double-float, but to do so iff the above return from u-a-e-t is nil).
+---------------
But, but... I thought UPGRADED-ARRAY-ELEMENT-TYPE is supposed to do this:
[CLHS]
Function UPGRADED-ARRAY-ELEMENT-TYPE
...
Description:
Returns the element type of the most specialized array representation
capable of holding items of the type denoted by typespec.
So while it can certainly return T, or the same type that it was given
as an argument, or some type between that type & T, how can it possibly
return NIL?!? [1]
Anyway, I think the OP's question is unanswerable within ANSI CL.
Yes, for many CLs and typespecs it might be the case that if:
(upgraded-array-element-type 'foo) ==> foo
then a FOO can be stored in a specialized array without "boxing"
[defined as being a pointer to separate heap-allocated memory
per element]. So one might be tempted to define something like this:
(defun array-element-unboxed-type-p (type)
(equalp type (upgraded-array-element-type type)))
But as Carl Shapiro pointed out to me off-line, the CLHS makes
absolutely no guarantee whatsoever that a specialized array actually
*does* store things unboxed!! All a "specialized array" guarantees
is that you can do type discrimination on the element type. [See CLHS
"15.1.2 Specialized Arrays" & "15.1.2.2 Required Kinds of Specialized
Arrays".] That is, it is perfectly legal [though most people would
consider it not very useful] for a CL implementation to behave as
follows on some types, call one of them WEIRD-TYPE:
> (upgraded-array-element-type 'weird-type)
WEIRD-TYPE
> (make-array 10 :element-type 'weird-type i:initial-element (make-weird-type))
#(#<weird> #<weird> #<weird> #<weird> #<weird>
#<weird> #<weird> #<weird> #<weird> #<weird>)
> (array-element-type *)
WEIRD-TYPE
>
and yet have the array elements *actually* be pointers to heap-allocated
objects of type WEIRD-TYPE. the (confirming) implementation simply chose
to allow specialized arrays of WEIRD-TYPE to allow type discrimination
on that type. (Or something.)
[Likewise, ANSI CL does not *require* that STRINGs be stored as
dense arrays of unboxed elements. They might well be stored as
arrays of pointers to "something else".]
Conversely, if what the OP was *really* worried about is not "boxing"
per se but "wasted space", the above ARRAY-ELEMENT-UNBOXED-TYPE-P
would *also* give you the "wrong" answer when an implementation chooses
to use an upgraded type which -- while still *unboxed* -- is not
exactly the same size as the type you asked about. Consider these
examples from CMUCL [Allegro is probably similar]:
> (mapcar 'upgraded-array-element-type
'(float single-float double-float))
(T SINGLE-FLOAT DOUBLE-FLOAT)
> (mapcar 'upgraded-array-element-type
'(nil character base-char extended-char))
(BIT BASE-CHAR BASE-CHAR BIT) ; See [1], [2] below
> (mapcar 'upgraded-array-element-type
'(unsigned-byte (unsigned-byte 8) (unsigned-byte 16)
(unsigned-byte 17) (unsigned-byte 24) (unsigned-byte 30)
(unsigned-byte 32)))
(T (UNSIGNED-BYTE 8) (UNSIGNED-BYTE 16) (UNSIGNED-BYTE 32)
(UNSIGNED-BYTE 32) (UNSIGNED-BYTE 32) (UNSIGNED-BYTE 32))
> (mapcar 'upgraded-array-element-type
'(signed-byte (signed-byte 8) (signed-byte 16)
(signed-byte 17) (signed-byte 24) (signed-byte 30)
(signed-byte 32)))
(T (SIGNED-BYTE 8) (SIGNED-BYTE 16) FIXNUM FIXNUM FIXNUM (SIGNED-BYTE 32))
>
In particular, (UPGRADED-ARRAY-ELEMENT-TYPE '(SIGNED-BYTE 30)) ==> FIXNUM,
yet there is *no* wasted space or boxing involved in that "upgrade"!!
[But there might be a *performance* penalty (shifting).]
Similarly, (UPGRADED-ARRAY-ELEMENT-TYPE '(SIGNED-BYTE 24)) ==> FIXNUM
*does* represent a small percentage of wasted space, but still no boxing.
So, to repeat, I don't think there's any portable ANSI CL way to
know whether or not specializing an array for a given type will
result in boxing and/or "wasted space" in the array. One would
need to do some implementation-specific poking around at a very
low level to find out.
-Rob
[1] Well, except for an argument of NIL. Carl says Allegro does
(UPGRADED-ARRAY-ELEMENT-TYPE NIL) ==> NIL, which is not altogether
unreasonable. On the other hand, what CMUCL does is arguably
not "wrong" either: (UPGRADED-ARRAY-ELEMENT-TYPE NIL) ==> BIT.
[2] In CMUCL the EXTENDED-CHAR class is empty, so it's not totally
unreasonable for UPGRADED-ARRAY-ELEMENT-TYPE to return the same
thing it does for other empty classes such as NIL.
-----
Rob Warnock <····@rpw3.org>
627 26th Avenue <URL:http://rpw3.org/>
San Mateo, CA 94403 (650)572-2607
From: Duane Rettig
Subject: Re: Quick way to figure out if floats are boxed in arrays?
Date:
Message-ID: <o0hcy7b9d7.fsf@gemini.franz.com>
····@rpw3.org (Rob Warnock) writes:
> Duane Rettig <·····@franz.com> wrote:
> +---------------
> | Nicolas Neuss <··················@iwr.uni-heidelberg.de> writes:
> | > Hmm, maybe UPGRADED-ARRAY-ELEMENT-TYPE?
> | > (upgraded-array-element-type 'double-float) ; if T then boxed?
> |
> | Since this answer was posed with a question mark, I'll solidify it to
> | a fine mush. If this expression returns T then the double-floats will
> | definitely be boxed. If it returns nil, then I am not sure if it a
> | priori dictates that the float is housed unboxed into a specialized
> | array made with that element-type. This is definitely the case in
> | Allegro CL (i.e. nil means unboxed), and I believe that most if not
> | all other CLs do the same thing (not specifically to store
> | double-floats unboxed into an array made with element-type
> | double-float, but to do so iff the above return from u-a-e-t is nil).
> +---------------
>
> But, but... I thought UPGRADED-ARRAY-ELEMENT-TYPE is supposed to do this:
>
> [CLHS]
> Function UPGRADED-ARRAY-ELEMENT-TYPE
> ...
> Description:
> Returns the element type of the most specialized array representation
> capable of holding items of the type denoted by typespec.
>
> So while it can certainly return T, or the same type that it was given
> as an argument, or some type between that type & T, how can it possibly
> return NIL?!? [1]
All right; I already ate some crow. Not nil, but 'double-float...
> Anyway, I think the OP's question is unanswerable within ANSI CL.
> Yes, for many CLs and typespecs it might be the case that if:
>
> (upgraded-array-element-type 'foo) ==> foo
>
> then a FOO can be stored in a specialized array without "boxing"
> [defined as being a pointer to separate heap-allocated memory
> per element]. So one might be tempted to define something like this:
>
> (defun array-element-unboxed-type-p (type)
> (equalp type (upgraded-array-element-type type)))
Not at all, as you clarify next:
> But as Carl Shapiro pointed out to me off-line, the CLHS makes
> absolutely no guarantee whatsoever that a specialized array actually
> *does* store things unboxed!! All a "specialized array" guarantees
> is that you can do type discrimination on the element type. [See CLHS
> "15.1.2 Specialized Arrays" & "15.1.2.2 Required Kinds of Specialized
> Arrays".] That is, it is perfectly legal [though most people would
> consider it not very useful] for a CL implementation to behave as
> follows on some types, call one of them WEIRD-TYPE:
>
> > (upgraded-array-element-type 'weird-type)
>
> WEIRD-TYPE
> > (make-array 10 :element-type 'weird-type i:initial-element (make-weird-type))
>
> #(#<weird> #<weird> #<weird> #<weird> #<weird>
> #<weird> #<weird> #<weird> #<weird> #<weird>)
> > (array-element-type *)
> WEIRD-TYPE
> >
>
> and yet have the array elements *actually* be pointers to heap-allocated
> objects of type WEIRD-TYPE. the (confirming) implementation simply chose
> to allow specialized arrays of WEIRD-TYPE to allow type discrimination
> on that type. (Or something.)
>
> [Likewise, ANSI CL does not *require* that STRINGs be stored as
> dense arrays of unboxed elements. They might well be stored as
> arrays of pointers to "something else".]
This is as it should be; the boxing and unboxing of an object should
be a conceptal thing only. What I I were to implement a specialized
array of double-floats not as a packed structure of IEEE specified
64-bit fields, but instead as pointers to remote objects in some sort
of database? They are knwably double-floats, but not knowably boxed.
> Conversely, if what the OP was *really* worried about is not "boxing"
> per se but "wasted space", the above ARRAY-ELEMENT-UNBOXED-TYPE-P
> would *also* give you the "wrong" answer when an implementation chooses
> to use an upgraded type which -- while still *unboxed* -- is not
> exactly the same size as the type you asked about. Consider these
> examples from CMUCL [Allegro is probably similar]:
>
> > (mapcar 'upgraded-array-element-type
> '(float single-float double-float))
>
> (T SINGLE-FLOAT DOUBLE-FLOAT)
> > (mapcar 'upgraded-array-element-type
> '(nil character base-char extended-char))
>
> (BIT BASE-CHAR BASE-CHAR BIT) ; See [1], [2] below
> > (mapcar 'upgraded-array-element-type
> '(unsigned-byte (unsigned-byte 8) (unsigned-byte 16)
> (unsigned-byte 17) (unsigned-byte 24) (unsigned-byte 30)
> (unsigned-byte 32)))
>
> (T (UNSIGNED-BYTE 8) (UNSIGNED-BYTE 16) (UNSIGNED-BYTE 32)
> (UNSIGNED-BYTE 32) (UNSIGNED-BYTE 32) (UNSIGNED-BYTE 32))
> > (mapcar 'upgraded-array-element-type
> '(signed-byte (signed-byte 8) (signed-byte 16)
> (signed-byte 17) (signed-byte 24) (signed-byte 30)
> (signed-byte 32)))
>
> (T (SIGNED-BYTE 8) (SIGNED-BYTE 16) FIXNUM FIXNUM FIXNUM (SIGNED-BYTE 32))
> >
>
> In particular, (UPGRADED-ARRAY-ELEMENT-TYPE '(SIGNED-BYTE 30)) ==> FIXNUM,
> yet there is *no* wasted space or boxing involved in that "upgrade"!!
> [But there might be a *performance* penalty (shifting).]
>
> Similarly, (UPGRADED-ARRAY-ELEMENT-TYPE '(SIGNED-BYTE 24)) ==> FIXNUM
> *does* represent a small percentage of wasted space, but still no boxing.
All of this is true, but the _only_ way to truly consider space
wasting is to use profiling tools to find out what the implementation
does in general, and of course the best use of developer time is to
use those tools only on the highest offenders (of space and/or time
wasting).
> So, to repeat, I don't think there's any portable ANSI CL way to
> know whether or not specializing an array for a given type will
> result in boxing and/or "wasted space" in the array. One would
> need to do some implementation-specific poking around at a very
> low level to find out.
>
> -Rob
>
> [1] Well, except for an argument of NIL. Carl says Allegro does
> (UPGRADED-ARRAY-ELEMENT-TYPE NIL) ==> NIL, which is not altogether
> unreasonable. On the other hand, what CMUCL does is arguably
> not "wrong" either: (UPGRADED-ARRAY-ELEMENT-TYPE NIL) ==> BIT.
Actually, it is not possible to implement a-u-e-t completely
correctly, as is listed here:
http://www.cliki.net/Proposed%20ANSI%20Revisions%20and%20Clarifications
and as described here:
http://www.cliki.net/Issue%20ARRAY-UPGRADING-UNDECIDABLE
Also, as for type NIL, it is not clear from the spec whether it is
legal to upgrade NIL to BIT, since an array of type NIL is, under one
interpretation, a string! Witness:
http://www.cliki.net/Issue%20ARRAY-ELEMENT-TYPE-NIL
> [2] In CMUCL the EXTENDED-CHAR class is empty, so it's not totally
> unreasonable for UPGRADED-ARRAY-ELEMENT-TYPE to return the same
> thing it does for other empty classes such as NIL.
>
> -----
> Rob Warnock <····@rpw3.org>
> 627 26th Avenue <URL:http://rpw3.org/>
> San Mateo, CA 94403 (650)572-2607
>
--
Duane Rettig ·····@franz.com Franz Inc. http://www.franz.com/
555 12th St., Suite 1450 http://www.555citycenter.com/
Oakland, Ca. 94607 Phone: (510) 452-2000; Fax: (510) 452-0182
Duane Rettig <·····@franz.com> wrote:
+---------------
| Actually, it is not possible to implement a-u-e-t completely
| correctly, as is listed here:
| http://www.cliki.net/Proposed%20ANSI%20Revisions%20and%20Clarifications
| and as described here:
| http://www.cliki.net/Issue%20ARRAY-UPGRADING-UNDECIDABLE
+---------------
Thanks for those references!
+---------------
| Also, as for type NIL, it is not clear from the spec whether it is
| legal to upgrade NIL to BIT, since an array of type NIL is, under one
| interpretation, a string! Witness:
|
| http://www.cliki.net/Issue%20ARRAY-ELEMENT-TYPE-NIL
+---------------
Yup. CLISP at least upgrades NIL to BIT consistently:
[1]> (upgraded-array-element-type nil)
BIT
[2]> (make-array 0 :element-type nil)
#*
[3]> (array-element-type *)
BIT
[4]>
while CMUCL is somewhat inconsistent in its treatment:
cmu> (upgraded-array-element-type nil)
BIT
cmu> (make-array 0 :element-type nil)
""
cmu> (array-element-type *)
BASE-CHAR
cmu>
-Rob
p.s. I also found CMUCL somewhat inconsistent in its treatment
of FIXNUM versus (SIGNED-BYTE 30):
cmu> (upgraded-array-element-type '(signed-byte 30))
FIXNUM
cmu> (make-array 5 :element-type 'fixnum)
#(0 0 0 0 0)
cmu> (array-element-type *)
(SIGNED-BYTE 30)
cmu>
Since UPGRADED-ARRAY-ELEMENT-TYPE is supposed to return "the most
specialized array representation capable of holding items of the
type denoted by [the] typespec", I would expect that:
(array-element-type (make-array 5 :element-type '(signed-byte 30)))
==> FIXNUM
but in CMUCL it doesn't. Though perhaps this provides some justification?
cmu> (subtypep 'fixnum '(signed-byte 30))
T
T
cmu> (subtypep '(signed-byte 30) 'fixnum)
T
T
cmu>
-----
Rob Warnock <····@rpw3.org>
627 26th Avenue <URL:http://rpw3.org/>
San Mateo, CA 94403 (650)572-2607
From: Duane Rettig
Subject: Re: Quick way to figure out if floats are boxed in arrays?
Date:
Message-ID: <o0vemmxb58.fsf@gemini.franz.com>
····@rpw3.org (Rob Warnock) writes:
> Duane Rettig <·····@franz.com> wrote:
> +---------------
> | Actually, it is not possible to implement a-u-e-t completely
> | correctly, as is listed here:
> | http://www.cliki.net/Proposed%20ANSI%20Revisions%20and%20Clarifications
> | and as described here:
> | http://www.cliki.net/Issue%20ARRAY-UPGRADING-UNDECIDABLE
> +---------------
>
> Thanks for those references!
No problem.
> +---------------
> | Also, as for type NIL, it is not clear from the spec whether it is
> | legal to upgrade NIL to BIT, since an array of type NIL is, under one
> | interpretation, a string! Witness:
> |
> | http://www.cliki.net/Issue%20ARRAY-ELEMENT-TYPE-NIL
> +---------------
>
> Yup. CLISP at least upgrades NIL to BIT consistently:
>
> [1]> (upgraded-array-element-type nil)
> BIT
> [2]> (make-array 0 :element-type nil)
> #*
> [3]> (array-element-type *)
> BIT
> [4]>
>
> while CMUCL is somewhat inconsistent in its treatment:
>
> cmu> (upgraded-array-element-type nil)
>
> BIT
> cmu> (make-array 0 :element-type nil)
>
> ""
> cmu> (array-element-type *)
>
> BASE-CHAR
> cmu>
This is what happens when a literal interpretation of the spec is
made. It could be considered a bug, because the spec is crystal clear
in what it wants u-a-e-t to do - it wants it to create for the user an
indication of what make-array will do. However, because the spec is
in fact inconsistent, especially wrt the nil type, it can be argued
that this behavior is not a bug, and conforms perfectly, based on the
clause in 1.5.1.4.1 where in conflicting situations the more specific
specification takes effect.
We solve this problem in Allegro CL by actually creating the nil array
specialization. I can't imagine anyone using an array of element-type
nil, but it does allow other pieces of array-specialization to work
together with fewer intuitive inconsistencies.
CL-USER(1): (upgraded-array-element-type nil)
NIL
CL-USER(2): (make-array 0 :element-type nil)
#()
CL-USER(3): (array-element-type *)
NIL
CL-USER(4):
> p.s. I also found CMUCL somewhat inconsistent in its treatment
> of FIXNUM versus (SIGNED-BYTE 30):
>
> cmu> (upgraded-array-element-type '(signed-byte 30))
>
> FIXNUM
> cmu> (make-array 5 :element-type 'fixnum)
>
> #(0 0 0 0 0)
> cmu> (array-element-type *)
>
> (SIGNED-BYTE 30)
> cmu>
But what is the trouble here? There is no requirement that an
implementation normalize a type specification. In Allegro CL, we have
a function called normalize-type, which we often use (but in fact not
in a specialized-array element-type situation - note that fixnum and
(signed-byte 30) are in fact the same:
CL-USER(1): (upgraded-array-element-type '(signed-byte 30))
FIXNUM
CL-USER(2): (make-array 5 :element-type 'fixnum)
#(0 0 0 0 0)
CL-USER(3): (array-element-type *)
FIXNUM
CL-USER(4): (normalize-type 'fixnum)
(INTEGER -536870912 536870911)
CL-USER(5): (normalize-type '(signed-byte 30))
(INTEGER -536870912 536870911)
CL-USER(6):
> Since UPGRADED-ARRAY-ELEMENT-TYPE is supposed to return "the most
> specialized array representation capable of holding items of the
> type denoted by [the] typespec", I would expect that:
>
> (array-element-type (make-array 5 :element-type '(signed-byte 30)))
> ==> FIXNUM
>
> but in CMUCL it doesn't. Though perhaps this provides some justification?
>
> cmu> (subtypep 'fixnum '(signed-byte 30))
>
> T
> T
> cmu> (subtypep '(signed-byte 30) 'fixnum)
>
> T
> T
> cmu>
But what do you call two type specifications which are mutually
subtypes of each other?
Answer: equivalent type specifications.
There's no problem here...
Here's another mind-bender: Suppose an implementation has both a
(signed-byte 8) and (unsigned-byte 8) specialization (an easy thing to
imagine; since I think many CLs have these). If these same
implementations have no 7-bit specializations, what do you think that
(unsigned-byte 7) should be upgraded to? The spec for u-a-e-t says
that it must be the most-specialized array representation capable of
items of the type. But what _is_ more specialized, (signed-byte 8) or
(unsigned-byte 8)? Does this conflict preclude an implementation
from even having both types, because they cannot be ordered in
specificity?
--
Duane Rettig ·····@franz.com Franz Inc. http://www.franz.com/
555 12th St., Suite 1450 http://www.555citycenter.com/
Oakland, Ca. 94607 Phone: (510) 452-2000; Fax: (510) 452-0182
From: Christophe Rhodes
Subject: Re: Quick way to figure out if floats are boxed in arrays?
Date:
Message-ID: <sqd58u3qac.fsf@cantab.net>
Duane Rettig <·····@franz.com> writes:
> We solve this problem in Allegro CL by actually creating the nil array
> specialization. I can't imagine anyone using an array of element-type
> nil, but it does allow other pieces of array-specialization to work
> together with fewer intuitive inconsistencies.
>
> CL-USER(1): (upgraded-array-element-type nil)
> NIL
> CL-USER(2): (make-array 0 :element-type nil)
> #()
> CL-USER(3): (array-element-type *)
> NIL
> CL-USER(4):
(stringp **) => ?
For the uninitiated, those pesky NIL arrays must no longer exist, but
they must also be strings, as the type STRING means:
[...] a specialized vector whose elements are of type character or a
subtype of type character.
and NIL is a subtype of type character. (I'm sure Duane knows this,
and he's glossed over it so as to preserve a shred of sanity in his
readers; I have no such compunction.)
> [...]
>
> Here's another mind-bender: Suppose an implementation has both a
> (signed-byte 8) and (unsigned-byte 8) specialization (an easy thing to
> imagine; since I think many CLs have these). If these same
> implementations have no 7-bit specializations, what do you think that
> (unsigned-byte 7) should be upgraded to? The spec for u-a-e-t says
> that it must be the most-specialized array representation capable of
> items of the type. But what _is_ more specialized, (signed-byte 8) or
> (unsigned-byte 8)? Does this conflict preclude an implementation
> from even having both types, because they cannot be ordered in
> specificity?
Certainly not! At the most, it means that in addition to both of
those specializations the implementation must offer an (unsigned-byte
7) specialization; in general, all this means is that the
upgraded-array-element-type lattice must be closed under type
intersection.
Christophe
From: http://public.xdi.org/=pf
Subject: Re: Quick way to figure out if floats are boxed in arrays?
Date:
Message-ID: <m2u027llbv.fsf@mycroft.actrix.gen.nz>
On Fri, 13 Oct 2006 14:38:52 -0700, Duane Rettig wrote:
> Nicolas Neuss <··················@iwr.uni-heidelberg.de> writes:
>> Hmm, maybe UPGRADED-ARRAY-ELEMENT-TYPE?
>>
>> (upgraded-array-element-type 'double-float) ; if T then boxed?
> Since this answer was posed with a question mark, I'll solidify it to
> a fine mush. If this expression returns T then the double-floats will
> definitely be boxed. If it returns nil, then I am not sure if it a
> priori dictates that the float is housed unboxed into a specialized
If it returns NIL, you're in deep trouble!
If it returns DOUBLE-FLOAT, on the other hand...
--
The government consists of a gang of men exactly like you and me. They
have, taking one with another, no special talent for the business of
government; they have only a talent for getting and holding
office. Their principal device to that end is to search out groups who
pant and pine for something they can't get and to promise to give it
to them. Nine times out of ten that promise is worth nothing. The
tenth time is made good by looting A to satisfy B. In other words,
government is a broker in pillage, and every election is sort of an
advance auction sale of stolen goods.
-- H. L. Mencken
(setq reply-to
(concatenate 'string "Paul Foley " "<mycroft" '(··@) "actrix.gen.nz>"))
From: Duane Rettig
Subject: Re: Quick way to figure out if floats are boxed in arrays?
Date:
Message-ID: <o0lknjpqte.fsf@gemini.franz.com>
Paul Foley <···@below.invalid> (http://public.xdi.org/=pf) writes:
> On Fri, 13 Oct 2006 14:38:52 -0700, Duane Rettig wrote:
>
> > Nicolas Neuss <··················@iwr.uni-heidelberg.de> writes:
> >> Hmm, maybe UPGRADED-ARRAY-ELEMENT-TYPE?
> >>
> >> (upgraded-array-element-type 'double-float) ; if T then boxed?
>
> > Since this answer was posed with a question mark, I'll solidify it to
> > a fine mush. If this expression returns T then the double-floats will
> > definitely be boxed. If it returns nil, then I am not sure if it a
> > priori dictates that the float is housed unboxed into a specialized
>
> If it returns NIL, you're in deep trouble!
Nah, that just means your lisp is _really_ simple...
> If it returns DOUBLE-FLOAT, on the other hand...
Yeah; brain fart from an old fart...
--
Duane Rettig ·····@franz.com Franz Inc. http://www.franz.com/
555 12th St., Suite 1450 http://www.555citycenter.com/
Oakland, Ca. 94607 Phone: (510) 452-2000; Fax: (510) 452-0182
From: Steven Haflich
Subject: Re: Quick way to figure out if floats are boxed in arrays?
Date:
Message-ID: <45339B35.4060506@alum.mit.edu>
············@gmail.com wrote:
> I'd like to know in O(1) time if something like
>
> (make-array n :element-type 'double-float)
>
> produces an array which is bit-compatible with a C array of doubles, so
> that I can pass it to a C function without copy-in & copy-out.
I've skimmed through the many messages in this threads (but not read
them carefully) but it appears that all the language gurus and other
useless denizens of this list have ignored your basic question,
preferring to impress each other with their knowledge and opinions
about certain type upgrading issues.
The simple answer is that the layout of a specialized Lisp array will
necessarily be different from the layout of a typical C array. This
is because a Lisp array _must_ be tagged with a type and size, while
few if any C compilers implement run-time typing. Therefore a Lisp
array will have at least a couple extra words of header, either before
the actual array data storage, or else as a separate header object.
In other words, it is possible in CL to determine the cl:type-of an
object not visible to the code at compile time. In C it is not
possible -- indeed, the language is designed in a way that the question
doesn't even make sense.
It may be possible to obtain the address of the data storage and pass
that to a C routine, but details are very implementation dependent and
may expose you to the gc issues mentioned elsewhere.