From: John Small
Subject: Re: Clisp FFI question: variable length arrays returned from C function that are not null terminated
Date:
Message-ID: <bCsOd.32256$2p.1599@lakeread08>
R. Mattes wrote:
> On Wed, 09 Feb 2005 12:25:42 -0500, John Small wrote:
>
>
>>Problem:
>>
>>I'm trying to use CLisp FFI with a C function that returns a pointer
>>to a variant length array of strings (i.e. char *) but the array is not
>>null terminated.
>
>
> So how is one supposed to know the size of the array? Sorry, but
> this is one of the most dangerous things i ever saw.
> Somehow, this would be useless even from within C et.al.
> Can you elaborate on the real library that you try to access?
>
> cheers Ralf Mattes
MySQL API has a function called
mysql_fetch_row(result_id)
which returns a char ** pointer.
This is an array of char * pointers to the column
values for the row just fetched. The first call
to set up the query establishes how many columns
there are in the result set which can be retrieved
by another function call.
The problem is with MySQL's approach to returning this
data.
The array returned (via pointer) by mysql_fetch_row
is not null terminated. In fact columns having SQL
null values are represented in the array as null pointers
so even if MySQL null terminated the array it wouldn't
work with FFI:C-ARRAY-PTR.
There are no other functions in the MySQL API for
fetching individual column values for a particular row.
I've already written the driver for ODBC which does not
take this approach but instead binds buffers to the
column once which subsequent calls to odbc's fetch_row
then repopulates. The buffers are fixed to the max length
of the column value (as indicated in the meta data) and
the number of buffers are established as well so can
be dealt with programmatically.
Unless I can deal with a variant length array that
is not null terminated I'm stuck. I could redefinite
the FFI IDL for mysql_fetch_row and register it on
every query but that would be a disaster. I even
thought of setting another shared library just to
double buffer this result and convert it to a form
more readily accessible to FFI.
Evidently the FFI parses the IDL for the call out and
puts it in a table which the VM then uses to automatically
bring values back from C allocating memory within GC space
and transforming the value format in the copying if required.
I can't seem to get access to the low level primitives to
manipulate the pointer walking the array myself and then
having the automatic fetching into the GC space handle
the allocation, copy and any transformation.
>
>
>>Thanks to Joerg for his previous answer. I've tried implementing his
>>suggestions
>>but I failed. Below are files giving a minimal example (highlighting my
>>misunderstanding). Hopefully they are small enough that
>>comments/corrections
>>can be inserted directly at the locations of my bugs and require minimum
>>effort
>>to answer this email. Thanks for any help!!
>>
>>(BTW I'm using CLisp 2.33.1 on win32 from prebuilt binaries. (I don't
>>know how to use CVS yet and failed in the attempt to build under MSYS
>>that I just installed using the new 2.33.2 sources - still learning this
>>environment too. So I don't have FOREIGN-VARIABLE constructor yet.)
>>
>>Solution: (attempted but failed)
>>
>>---------------------------------------------------------------------------------------------------
>>// foo.c
>>
>>#include <stdio.h>
>>
>>// define for Microsoft Visual C on win32 #define DLLEXPORT
>>_declspec(dllexport)
>>
>>// Otherwise
>>// #define DLLEXPORT
>>
>>
>>char * cols[] = { "one", "two", "three", "four" };
>>
>>// Real foo will return variant length array that is // not null
>>terminated.
>>
>>DLLEXPORT char ** foo()
>>{
>> printf("\nC address of string array: %x",cols); return cols;
>>}
>>}
>>---------------------------------------------------------------------------------------------------------------
>>
>>;; Example of accessing arrays that are not null terminated ;; foo.lisp
>>;; see foo.c
>>
>>(FFI:DEF-CALL-OUT foo
>> (:library "foo.dll")
>> (:language :stdc)
>> (:name "foo")
>> (:arguments)
>> (:return-type (ffi:c-ptr (ffi:c-array ffi:c-string 3))))
>>
>>;; The real foo will return varying sized arrays that ;; are not null
>>terminated so ffi:c-array-ptr can not be used. ;; This call out spec
>>simply tests a fixed size array with a ;; specified length less than or
>>equal to the actually array returned.
>>
>>(setf s (foo))
>>
>>(format t "~%Two: ~A" (aref s 1))
>>
>>(format t "~%'one' 'two' 'three': ~A" s)
>>
>>(setf col-count 3) ;; varies on the real foo.
>>
>>;; this version generates the error indicated below
>>
>>(FFI:DEF-CALL-OUT bar
>> (:library "foo.dll")
>> (:language :stdc)
>> (:name "foo")
>> (:arguments)
>> (:return-type ffi:c-pointer))
>>
>>(setf s2 (bar))
>>
>>(format t "~%Foreign address: ~A" s2)
>>
>>
>>;; convert foreign address to c-place for FFI primitives
>>
>>(ffi:with-c-var (cols `ffi:c-pointer s2)
>>
>> (ffi:offset cols 1 `(ffi:c-array ffi:c-string ,col-count))
>>
>>;; Above line generates this error:
>>;; *** - FFI::%OFFSET: foreign variable #<FOREIGN-VARIABLE ;;
>>"EXEC-ON-STACK" #x00..> does not have the required alignment
>>
>>)
>>
>>
>>;; Alternate approach - also generates error indicated below.
>>
>>(FFI:DEF-CALL-OUT foobar
>> (:library "foo.dll")
>> (:language :stdc)
>> (:name "foo")
>> (:arguments)
>> (:return-type (ffi:c-ptr (ffi:c-array ffi:c-string 1))))
>>
>>(setf s3 (foobar))
>>
>>(format t "~%Two: ~A" s3)
>>
>>(ffi:with-c-place (cols s3)
>>
>> (ffi:offset cols 1 `(ffi:c-array ffi:c-string ,col-count))
>>
>>;; Above line generates this error:
>>;; *** - FFI::%OFFSET: argument is not a foreign variable: #("one")
>>
>>)
>
>
MySQL API has a function called