From: dave linenberg
Subject: ACL Foreign Function Interface Question
Date:
Message-ID: <392B1CC7.4A53E0E2@home.com>
I have questions concerning mapping foreign types into standard Lisp
types.
I've created a foreign function interface which calls C functions that
fill foreign types.
For example, given the foreign types,
(ff:def-foreign-type dailyData
(:struct
(date :int)
(open :double)
(low :double)
(high :double)
(close :double)
(vol :int)))
(ff:def-foreign-type historicalData
(:struct
(size :int)
(dailyDataArray (:array dailyData 12500))))
I can create a historical stock data object,.... (setq hd
(ff:allocate-fobject 'historicalData))
and fill it with data.....
(fillstruct "/tmp/CAT" dp) ;; C function call which reads binary
files and gets CAT data into the foreign typed struct
My problem is that I would like to access dailyDataArray as a vector, so
I can use it in the Lisp sequence functions.
It's great that I can fill these foreign structs with data *FAST*, and
access them, but they have a completely different access style :
(ff:fslot-value-typed 'historicalData nil dp :dailyDataArray 0 :date)
(ff:fslot-value-typed 'historicalData nil dp :dailyDataArray 0 :high)
(ff:fslot-value-typed 'historicalData nil dp :dailyDataArray 0 :low)
(ff:fslot-value-typed 'historicalData nil dp :dailyDataArray)
541951920 ;; integer pointing to foreign location of
dailyDataArray..... I REALLY want a Lisp vector of dailyData !!!
I need to transform the foreign types above into a lisp
structs/vectors. Is there any *DIRECT* way to do this, without creating
a Lisp struct or CLOS class, allocating new memory essentially, and
looping through the foreign struct, filling the Lisp class or struct etc
etc? The information is there. It would be nice if we could treat these
structs and parts of structs (vectors) as normal Lisp types.
Another point... I am reading 300 to 400K binary files off of a hard
disk into Lisp (pre-allocated foreign data structures) in 3
miliseconds(!!!!) through a C interface. This is *FAST*. Calling
foreign functions judiciously can give MAJOR speed improvements.
Dave Linenberg
From: Joe Marshall
Subject: Re: ACL Foreign Function Interface Question
Date:
Message-ID: <4s7okmom.fsf@alum.mit.edu>
dave linenberg <········@home.com> writes:
> I've created a foreign function interface which calls C functions that
> fill foreign types.
> For example, given the foreign types,
>
> (ff:def-foreign-type dailyData
> (:struct
> (date :int)
> (open :double)
> (low :double)
> (high :double)
> (close :double)
> (vol :int)))
>
> (ff:def-foreign-type historicalData
> (:struct
> (size :int)
> (dailyDataArray (:array dailyData 12500))))
>
> I can create a historical stock data object,.... (setq hd
> (ff:allocate-fobject 'historicalData))
> and fill it with data.....
> (fillstruct "/tmp/CAT" dp) ;; C function call which reads binary
> files and gets CAT data into the foreign typed struct
>
> My problem is that I would like to access dailyDataArray as a vector, so
> I can use it in the Lisp sequence functions.
>
> I need to transform the foreign types above into a lisp
> structs/vectors. Is there any *DIRECT* way to do this, without creating
> a Lisp struct or CLOS class, allocating new memory essentially, and
> looping through the foreign struct, filling the Lisp class or struct etc
> etc? The information is there. It would be nice if we could treat these
> structs and parts of structs (vectors) as normal Lisp types.
The problem is that the data are layed out differently. In the C
world, you have this:
<int32 historicalData.dailyDataArray[0].date>
<float64 historicalData.dailyDataArray[0].open>
<float64 historicalData.dailyDataArray[0].low>
<float64 historicalData.dailyDataArray[0].high>
<float64 historicalData.dailyDataArray[0].close>
<int32 historicalData.dailyDataArray[0].vol>
<int32 historicalData.dailyDataArray[1].date>
<float64 historicalData.dailyDataArray[1].open>
<float64 historicalData.dailyDataArray[1].low>
<float64 historicalData.dailyDataArray[1].high>
<float64 historicalData.dailyDataArray[1].close>
<int32 historicalData.dailyDataArray[1].vol>
....
<int32 historicalData.dailyDataArray[12499].date>
<float64 historicalData.dailyDataArray[12499].open>
<float64 historicalData.dailyDataArray[12499].low>
<float64 historicalData.dailyDataArray[12499].high>
<float64 historicalData.dailyDataArray[12499].close>
<int32 historicalData.dailyDataArray[12499].vol>
In the lisp world, arrays are layed out something like this (I don't
know the details of Allegro on this point)
<32-bits array `header, size and type info'>
<32-bits array `header, size and type info'> ;; some kind of header
<32-bits array `header, size and type info'>
<32-bits tagged-pointer to element 0>
<32-bits tagged-pointer to element 1>
<32-bits tagged-pointer to element 2>
....
<32-bits tagged-pointer to element 12499>
Where each element layed out like this:
<32-bits structure header>
<32-bits structure header> ;; some number of header words
<32-bits structure header>
<32-bit tagged object slot 1> ;; Some number of slots
<32-bit tagged object slot 2>
<32-bit tagged object slot 3>
....
So the way you have it currently implemented, you really can't do much
more than copy the data out.
However ....
Can you change either the basic C structure or the routine that fills
the array? If you can, I'd suggest that you spread the data across
arrays.
Make an array for the dates, an array for the opening price, an array
for the low price, etc. Each of these arrays can be made to hold
untagged data with the appropriate declaration. Then if you are
careful to avoid tromping on the array header and letting the GC move
stuff around, you can fill the lisp arrays directly with your info.
I wouldn't recommend the other option of attempting to cons up lisp
structs from the C world, though.
> Another point... I am reading 300 to 400K binary files off of a hard
> disk into Lisp (pre-allocated foreign data structures) in 3
> miliseconds(!!!!) through a C interface. This is *FAST*. Calling
> foreign functions judiciously can give MAJOR speed improvements.
Yes. I worked on one project where I had to read files off of disk
and into a lisp string. For various reasons I didn't want to use
READ-SEQUENCE. Instead, I called mmap through the foreign function
interface and constructed a displaced string to the data (this worked
under Lucid). The end result was that I could read huge amounts of
data into strings virtually instantaneously, and with no consing.
The particular application ran something like 5 orders of magnitude
faster.
In article <············@alum.mit.edu>,
Joe Marshall <·········@alum.mit.edu> wrote:
> dave linenberg <········@home.com> writes:
>
>
>
> Can you change either the basic C structure or the routine that fills
> the array? If you can, I'd suggest that you spread the data across
> arrays.
>
> Make an array for the dates, an array for the opening price, an array
> for the low price, etc. Each of these arrays can be made to hold
> untagged data with the appropriate declaration. Then if you are
> careful to avoid tromping on the array header and letting the GC move
> stuff around, you can fill the lisp arrays directly with your info.
>
Thats exactly what I did today!!! Your post reconfirms this approach.
(ff:def-foreign-call sw_get_historical_data((fn (* :char))
(size (* :int))
(title (* :char))
(date (* :int))
(open (* :double))
(low (* :double))
(high (* :double))
(close (* :double))
(vol (* :int))) :returning :int)
(setq *size* (make-array 1 :element-type 'fixnum :initial-contents
`(0)))
(setq *fn* "/tmp/CAT")
(setq *title* (make-array 10 :element-type 'character))
(setq *date* (make-array 12500 :element-type `fixnum))
(setq *open* (make-array 12500 :element-type `double-float))
(setq *low* (make-array 12500 :element-type `double-float))
(setq *high* (make-array 12500 :element-type `double-float))
(setq *close* (make-array 12500 :element-type `double-float))
(setq *vol* (make-array 12500 :element-type `fixnum)))
(SW_GET_HISTORICAL_DATA *fn* *size* *title* *date* *open* *low* *high*
*close* *vol*)
And, obviously, I had to rewrite my C interface as well.
This Works... after the call, the Lisp arrays are filled
appropriately...
*size*
#(7649)
*date*
#(20000407 20000406 20000405 20000404 20000403 20000331 20000330
20000329 20000328 20000327 ...)
*close*
#(41.375d0 42.3125d0 42.4375d0 43.4375d0 41.875d0 39.4375d0 40.75d0
39.0d0 38.3125d0 39.0625d0 ...)
...etc.
As usual, good advice from the Lisp newsgroup.
Thanks Joe...
Dave
········@home.com
Sent via Deja.com http://www.deja.com/
Before you buy.