From: Sohail Somani
Subject: Wrapping a C++ library using ?FFI
Date:
Message-ID: <0Le8k.88$7%6.24@edtnps82>
Hey guys,
I'm part of the way through writing some lisp gibberish that wraps a C++
library.
The way I am doing it is by generating a C interface kind of like what
CFront would have had to do and generating a CFFI wrapper. My main goals
are:
* Inherit from a C++ class in CL
* Override virtual functions
* Bi-directional exceptions
* Crash only once in a blue moon
I was just wondering, am I duplicating work that already meets these
goals? I am not interested in half-finished work because then I'd have
to write the other half myself and so might as well write all of it :-)
I'd love to hear any ideas you might have.
TIA
Sohail
Sohail Somani wrote:
> Hey guys,
>
> I'm part of the way through writing some lisp gibberish that wraps a C++
> library.
>
> The way I am doing it is by generating a C interface kind of like what
> CFront would have had to do and generating a CFFI wrapper. My main goals
> are:
>
> * Inherit from a C++ class in CL
> * Override virtual functions
> * Bi-directional exceptions
> * Crash only once in a blue moon
>
> I was just wondering, am I duplicating work that already meets these
> goals? I am not interested in half-finished work because then I'd have
> to write the other half myself and so might as well write all of it :-)
>
> I'd love to hear any ideas you might have.
There are several projects (in various languages) to do this. Most
give up on the vagaries of C++ (e.g. name mangling, template
instantiation, vtables, and pointer casting) and rely on a "C" binding
layer to redirect between C++ and the other language.
This is tedious, brittle, and unpleasant.
I managed to convince myself that the best thing to do was to more or
less "rewrite gdb in lisp". Interactively bind in lisp using the
debug info. Skips all the C interface glue. Has a fighting chance of
not needing a recompile whenever the ABI changes. A MOP guru could
probably put together an auto-reflective binding layer.
Here's an incomplete poke in the debugger direction:
http://git.androdna.com/?p=lisp/dwarf;a=summary
In the meantime, I've started tinkering with ECL as a stopgap; there's
something elegant (and dirty) about simply embedding C code in lisp.
(And ECL seems easier to get running on bizarre platforms.)
ecls.sf.net
If you do decide try the raw debug route, drop me a line; I should be
able to rummage up some good resources. Good news is the Itanium ABI
was universally adopted by 64-bit compilers (something about them
wanting interoperability) and the standards committee is making
squeaks about introducing standards for all platforms.
- Daniel
From: Sohail Somani
Subject: Re: Wrapping a C++ library using ?FFI
Date:
Message-ID: <_Wi8k.172$7%6.142@edtnps82>
D Herring wrote:
> Sohail Somani wrote:
>> Hey guys,
>>
>> I'm part of the way through writing some lisp gibberish that wraps a
>> C++ library.
[snip]
> I managed to convince myself that the best thing to do was to more or
> less "rewrite gdb in lisp". Interactively bind in lisp using the debug
> info. Skips all the C interface glue. Has a fighting chance of not
> needing a recompile whenever the ABI changes. A MOP guru could probably
> put together an auto-reflective binding layer.
Since you mention it, there is a library that currently does something
like this (google Smoke binding KDE, but there is sparse information on
it.) The idea is that it generates all the metadata by parsing C++
headers and stuffs the data behind a mostly C interface. Then this is
used by Ruby/Python to dynamically generate bindings. Quite innovative
because the Python/Ruby bindings don't need to change for every ABI
bump. I'd imagine the Smoke library does need to do it though which is
almost the same problem.
> Here's an incomplete poke in the debugger direction:
> http://git.androdna.com/?p=lisp/dwarf;a=summary
Thanks, I'll take a look.
> In the meantime, I've started tinkering with ECL as a stopgap; there's
> something elegant (and dirty) about simply embedding C code in lisp.
> (And ECL seems easier to get running on bizarre platforms.)
>
> ecls.sf.net
I also tried the ECL route but was not sure how it helped me use C++
libraries. Maybe I just need a big fat link to the section in TFM.
> If you do decide try the raw debug route, drop me a line; I should be
> able to rummage up some good resources. Good news is the Itanium ABI
> was universally adopted by 64-bit compilers (something about them
> wanting interoperability) and the standards committee is making squeaks
> about introducing standards for all platforms.
Thanks for the helping hand! I think I've kind of decided on a route
based mostly on my fear of C++ ABIs but this thread has given me many
ideas. I sometimes say that C is the best C++ ABI which probably says a
lot about where I'm going with it...
Anyway, I hope to have something that everyone can poke and make fun of
sometime soon. Come to think of it, people poke and make fun anyway.
Have at it.
Thanks for your help!
Sohail
D Herring <········@at.tentpost.dot.com> writes:
> Here's an incomplete poke in the debugger direction:
> http://git.androdna.com/?p=lisp/dwarf;a=summary
For the git-impaired amongst us, please what would the path to the git
repository be?
--
__Pascal Bourguignon__ http://www.informatimago.com/
NEW GRAND UNIFIED THEORY DISCLAIMER: The manufacturer may
technically be entitled to claim that this product is
ten-dimensional. However, the consumer is reminded that this
confers no legal rights above and beyond those applicable to
three-dimensional objects, since the seven new dimensions are
"rolled up" into such a small "area" that they cannot be
detected.
Pascal J. Bourguignon wrote:
> D Herring <········@at.tentpost.dot.com> writes:
>> Here's an incomplete poke in the debugger direction:
>> http://git.androdna.com/?p=lisp/dwarf;a=summary
>
> For the git-impaired amongst us, please what would the path to the git
> repository be?
The website offers snapshot downloads if you don't care for git. If
you do, then try
git clone http://git.androdna.com/lisp/dwarf
I'll reiterate -- the code is rough, was never polished for release,
and I'm not the best lisper. But it does work (see warning below) for
reading Dwarf debug info, presenting it as lisp data; and I had a few
functions for finding classes, class members, etc.
Quick-start:
- download and install
- libelf
http://www.mr511.de/software/
http://www.mr511.de/software/libelf-0.8.10.tar.gz
- libdwarf
http://reality.sgiweb.org/davea/dwarf.html
http://reality.sgiweb.org/davea/libdwarf-20080615.tar.gz
(dwarfdump is interesting but not necessary)
(I had to do `make CFLAGS="-fPIC -I." libdwarf.so` to get the
shared library -- something fishy there)
Make sure both .so's are in a known library path.
- [optional] compile my test app and lib -- something simple to look
at (tweak the makefile as needed)
`make CPP`
- fire up lisp
(load "sdb.lisp")
(example-sdb) ;; should print reams of info describing the sdb app
Some other examples live in sdb.lisp, starting with
(defparameter *libqtgui* "/opt/qt4.3/lib/libQtGui.so.4.3.0.debug")
which obviously needs some end-user tweaking.
Warning: The code seems to have sprung a memory leak somewhere
between new versions of libelf, libdwarf, and SBCL, and a transition
from x86 to x86_64. It appears that (example-sdb) still works [small
file], but the Qt examples hang while exhausting 4GB of memory --
YMMV. ISTR putting 32-bit specific sizes in the code... need to
revisit this some time.
If anyone is interested in helping, I would gladly pull this project
off the back burner. Too many directions to explore, and too little
long-term focus.
Later,
Daniel
D Herring <········@at.tentpost.dot.com> writes:
> Pascal J. Bourguignon wrote:
>> D Herring <········@at.tentpost.dot.com> writes:
>>> Here's an incomplete poke in the debugger direction:
>>> http://git.androdna.com/?p=lisp/dwarf;a=summary
>> For the git-impaired amongst us, please what would the path to the
>> git repository be?
>
> The website offers snapshot downloads if you don't care for git. If
> you do, then try
>
> git clone http://git.androdna.com/lisp/dwarf
Thank you. (I tried fetch, fetch-html, pull, etc...)
> If anyone is interested in helping, I would gladly pull this project
> off the back burner.
I hope to be able to work on/with it a few hours a week.
> Too many directions to explore, and too little
> long-term focus.
Too little time.
--
__Pascal Bourguignon__ http://www.informatimago.com/
In deep sleep hear sound,
Cat vomit hairball somewhere.
Will find in morning.
P� Wed, 25 Jun 2008 00:42:04 +0200, skrev Sohail Somani
<······@taggedtype.net>:
> Hey guys,
>
> I'm part of the way through writing some lisp gibberish that wraps a C++
> library.
>
> The way I am doing it is by generating a C interface kind of like what
> CFront would have had to do and generating a CFFI wrapper. My main goals
> are:
>
> * Inherit from a C++ class in CL
> * Override virtual functions
> * Bi-directional exceptions
> * Crash only once in a blue moon
>
> I was just wondering, am I duplicating work that already meets these
> goals? I am not interested in half-finished work because then I'd have
> to write the other half myself and so might as well write all of it :-)
>
> I'd love to hear any ideas you might have.
>
> TIA
>
> Sohail
Well there was a project a few years back, but it is, I belive, half
finished.
http://common-lisp.net/project/fetter/
Appart from the lack of a standard for linking names in C++ the main
hurdle is that CLOS is dynamic while C++ classes are static.
Python developers faced some of the same hurdles. Perhaps this can give
you some ideas.
http://www.boost.org/doc/libs/1_35_0/libs/python/doc/internals.html
--------------
John Thingstad
From: Sohail Somani
Subject: Re: Wrapping a C++ library using ?FFI
Date:
Message-ID: <pOf8k.104$7%6.84@edtnps82>
John Thingstad wrote:
>
> Well there was a project a few years back, but it is, I belive, half
> finished.
> http://common-lisp.net/project/fetter/
Thanks for the link. This project looked interesting until I realized
that it actually needs to understand the compiler's ABI. That doesn't
sound very promising! The git repository has had recent updates so
obviously its useful for someone else but it appears to mainly be used
and useful for C code. Hmm... Maybe I can still use it after all.
> Appart from the lack of a standard for linking names in C++ the main
> hurdle is that CLOS is dynamic while C++ classes are static.
I think another hurdle for me is that there is no way to portably access
Lisp from C++ (not the other way around.) If I could figure out how to
portably access Lisp from C++, then I would have less need for an
intermediate C library.
> Python developers faced some of the same hurdles. Perhaps this can give
> you some ideas.
> http://www.boost.org/doc/libs/1_35_0/libs/python/doc/internals.html
Boost Python is one of my favourite language binding mechanisms ever but
I don't see how I could do something similar in CL without depending on
a particular CL implementation.
Thanks for your response.
Sohail
P� Wed, 25 Jun 2008 01:53:57 +0200, skrev Sohail Somani
<······@taggedtype.net>:
>
>> Python developers faced some of the same hurdles. Perhaps this can give
>> you some ideas.
>> http://www.boost.org/doc/libs/1_35_0/libs/python/doc/internals.html
>
> Boost Python is one of my favourite language binding mechanisms ever but
> I don't see how I could do something similar in CL without depending on
> a particular CL implementation.
You might find it less problematic to just pick a implementation and worry
about porting it when you have it working.
--------------
John Thingstad
From: Sohail Somani
Subject: Re: Wrapping a C++ library using ?FFI
Date:
Message-ID: <Iag8k.114$7%6.17@edtnps82>
John Thingstad wrote:
> P� Wed, 25 Jun 2008 01:53:57 +0200, skrev Sohail Somani
> <······@taggedtype.net>:
>
>>
>>> Python developers faced some of the same hurdles. Perhaps this can
>>> give you some ideas.
>>> http://www.boost.org/doc/libs/1_35_0/libs/python/doc/internals.html
>>
>
>> Boost Python is one of my favourite language binding mechanisms ever
>> but I don't see how I could do something similar in CL without
>> depending on a particular CL implementation.
>
> You might find it less problematic to just pick a implementation and
> worry about porting it when you have it working.
But of course. I am not particularly interested in portability right
now, just very much later :-)
Sohail Somani wrote:
> John Thingstad wrote:
>>
>> Well there was a project a few years back, but it is, I belive, half
>> finished.
>> http://common-lisp.net/project/fetter/
>
> Thanks for the link. This project looked interesting until I realized
> that it actually needs to understand the compiler's ABI. That doesn't
> sound very promising! The git repository has had recent updates so
> obviously its useful for someone else but it appears to mainly be used
> and useful for C code. Hmm... Maybe I can still use it after all.
>
>> Appart from the lack of a standard for linking names in C++ the main
>> hurdle is that CLOS is dynamic while C++ classes are static.
>
> I think another hurdle for me is that there is no way to portably access
> Lisp from C++ (not the other way around.) If I could figure out how to
> portably access Lisp from C++, then I would have less need for an
> intermediate C library.
I'm probably reading this out of context, but C and C++ can access or call
Lisp via the portable CFFI wrapper library:
http://common-lisp.net/project/cffi/manual/html_node/defcallback.html#defcallback
http://common-lisp.net/project/cffi/manual/html_node/Tutorial_002dCallbacks.html#Tutorial_002dCallbacks
--
Lars Rune N�stdal
http://nostdal.org/
Lars Rune N�stdal wrote:
> Sohail Somani wrote:
>> John Thingstad wrote:
>> I think another hurdle for me is that there is no way to portably
>> access Lisp from C++ (not the other way around.) If I could figure out
>> how to portably access Lisp from C++, then I would have less need for
>> an intermediate C library.
>
> I'm probably reading this out of context, but C and C++ can access or call
> Lisp via the portable CFFI wrapper library (link to defcallback):
I was not clear enough. What I meant to say was that there was no simple
and portable way to (say) define classes in Lisp without writing the
code to do so.
For example, if you look at the Boost Python example, it is clear that
the Python C API (no matter how fugly it is) allows you to register and
define classes, force GC, etc, etc. One would have to write the
equivalent of that C-API for Lisp which might be an interesting project
for someone who is interested in that sort of thing.
Myself, I like to write as little C++ as possible so even if it were
available, I doubt I'd use it much.
I'm sure that's as clear as mud but so is my brain these days.
In article <················@taggedtype.net>,
Sohail Somani <······@taggedtype.net> wrote:
> Lars Rune N�stdal wrote:
> > Sohail Somani wrote:
> >> John Thingstad wrote:
>
> >> I think another hurdle for me is that there is no way to portably
> >> access Lisp from C++ (not the other way around.) If I could figure out
> >> how to portably access Lisp from C++, then I would have less need for
> >> an intermediate C library.
> >
> > I'm probably reading this out of context, but C and C++ can access or call
> > Lisp via the portable CFFI wrapper library (link to defcallback):
>
> I was not clear enough. What I meant to say was that there was no simple
> and portable way to (say) define classes in Lisp without writing the
> code to do so.
Use the MOP for that.
Example from AMOP:
(defclass plane (moving-object graphics-object)
((altitude :initform 0 :accessor plane-altitude)
(speed))
(:default-initargs :engine *jet*))
this would be similar:
(ensure-class 'plane
:direct-superclasses '(moving-object graphics-object)
:direct-slots (list (list :name 'altitude
:initform 0
:initfunction #'(lambda () 0)
:readers '(plane-altitude)
:writers '((setf plane-altitude)))
(list :name 'speed))
:direct-default-initargs (list (list :engine
'*jet*
#'(lambda () *jet*))))
>
> For example, if you look at the Boost Python example, it is clear that
> the Python C API (no matter how fugly it is) allows you to register and
> define classes, force GC, etc, etc. One would have to write the
> equivalent of that C-API for Lisp which might be an interesting project
> for someone who is interested in that sort of thing.
>
> Myself, I like to write as little C++ as possible so even if it were
> available, I doubt I'd use it much.
>
> I'm sure that's as clear as mud but so is my brain these days.
--
http://lispm.dyndns.org/
Sohail Somani <······@taggedtype.net> writes:
> Hey guys,
>
> I'm part of the way through writing some lisp gibberish that wraps a
> C++ library.
>
> The way I am doing it is by generating a C interface kind of like what
> CFront would have had to do and generating a CFFI wrapper. My main
> goals are:
>
> * Inherit from a C++ class in CL
> * Override virtual functions
The C++ Object System is quite different from CL.
The best approach here will be to write a C++OS in Lisp.
Otherwise, the main problem you will have is that any C++ binding has
to be C++ compiler dependant. You will have to know well the
implementation of your C++ compiler.
> * Bi-directional exceptions
> * Crash only once in a blue moon
>
> I was just wondering, am I duplicating work that already meets these
> goals? I am not interested in half-finished work because then I'd have
> to write the other half myself and so might as well write all of it
> :-)
AFAIK, you'll have to write it whole, then.
> I'd love to hear any ideas you might have.
>
> TIA
>
> Sohail
--
__Pascal Bourguignon__ http://www.informatimago.com/
In a World without Walls and Fences,
who needs Windows and Gates?
From: Sohail Somani
Subject: Re: Wrapping a C++ library using ?FFI
Date:
Message-ID: <btf8k.100$7%6.77@edtnps82>
Pascal J. Bourguignon wrote:
> The C++ Object System is quite different from CL.
> The best approach here will be to write a C++OS in Lisp.
This is pretty much what I'm doing.
> Otherwise, the main problem you will have is that any C++ binding has
> to be C++ compiler dependant. You will have to know well the
> implementation of your C++ compiler.
Yes, I am hoping to steer clear of any C++ compiler specifics by using a
C interface to the C++ library. I haven't thought about templates yet
and I don't really think I want to.
>> * Bi-directional exceptions
>> * Crash only once in a blue moon
>>
>> I was just wondering, am I duplicating work that already meets these
>> goals? I am not interested in half-finished work because then I'd have
>> to write the other half myself and so might as well write all of it
>> :-)
>
> AFAIK, you'll have to write it whole, then.
Are there any other halves you know of?
Thanks for your reply.
Sohail
From: Sohail Somani
Subject: Re: Wrapping a C++ library using ?FFI
Date:
Message-ID: <_oG9k.791$1o6.270@edtnps83>
This is a multi-part message in MIME format.
--------------040002010301090904060606
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
Sohail Somani wrote:
> Hey guys,
>
> I'm part of the way through writing some lisp gibberish that wraps a C++
> library.
Ok, I've done most tricky things except for inheritance which I will
tackle next. The general idea is to take the headers and generate the
glue C, C++ and CFFI code. FWIW, this is how my brain works and I found
it easiest to work this way.
I've attached a brain dump. It's kinda long but I would appreciate
comments if you have them.
I suspect that for inheritance, the generated code will need to be more
dynamic in that the cast_to_base function will need to do some sort of
table lookups.
Thanks,
Sohail
--------------040002010301090904060606
Content-Type: text/plain;
name="design_notes.txt"
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="design_notes.txt"
Connecting Lisp and C++
-----------------------
These notes are as a result of setting up Common Lisp (SBCL) to
consume (but not extend) the following API.
class animal
{
~virtual animal(){}
virtual void speak() const = 0;
};
class dog : public animal
{
~dog(){}
virtual void speak () const = 0;
};
// The lifetime of the animal is not tied to the owner.
class owner
{
// Does not own pointer
owner(animal * a);
animal * get_animal();
};
Existing software
-----------------
The only piece of software I know comes close to what I need is
Verrazano which seems to want to implement the g++ ABI via CFFI.
While this is one approach, I feel it is too fragile and dependent on
a specific compiler.
Another option is to go the Boost Python route. I found that this
would only be feasible if there was enough support from the Lisp
implementation. There definitely is no common ground for a C/C++ API
so I would have to write one myself. And even then, I'd have to deal
with the complexities of closer-mop without the benefits of Lisp which
would be quite a tedious and time consuming task.
Code generation
---------------
The glue code in both languages will be generated by parsing the
output of cpptoxml found at http://repo.or.cz/w/lqt.git .
ABI
---
To connect Common Lisp and C++, we first have to recognize that most
C++ compilers find it hard to connect to each other. The main reason
for this is that C++ mandates no standard application binary interface
(ABI).
On the other hand, C has a quite well-understood and portable way to
write source code so that multiple compilers can be binary compatible.
Therefore, one way to connect Common Lisp and C++ is to introduce a
thin C library. I will refer to this as a "flattening" of the C++ API.
Name mangling
-------------
C++ compilers have a need to encode certain information within
functions such as number of arguments, type and size of each argument
and so on. Due to historical accidents, this was accomplished via the
practice known as name mangling which is the encoding of this
information in a single C compatible function name.
For example, the function int Bar::foo(int) might be mangled into:
Bar_int_foo_int
Similarly, the function int foo(double) might be mangled into:
Bar_int_foo_double
Note that since these are C names and C does not allow overloading,
the mangling algorithm must produce unique names for two distinct
functions.
The mangling for this task will encode:
* Instance member function or static member function
* Type and constness of implicit `this' argument
* Type and constness of each function argument
Specifically, there is no need to encode the type of the return value.
Exception handling
------------------
Exceptions in C++ are essentially non-local gotos. Of course they have
very nice properties unlike gotos like stack unwinding but the general
idea is that an exception restores the stack to somewhere else in the
execution tree.
In C, there are two methods of exception handling:
* Error codes
* setjmp/longjmp
Personally, I prefer setjmp/longjmp but am not really interested in
making CFFI work with it!
Therefore, it is best to use error strings. Error codes will be
avoided because they are not extensible and are too static.
Memory management
-----------------
Each object that is allocated in the C++ library will have a wrapper
Lisp object. The Lisp object will deallocate the object when its
finalizers are run (via trivial-garbage or cffi's finalize.)
There will be some way to indicate who owns the memory allocated by
each function.
By default, pointers returned by member functions are assumed to be
owned by some other object.
Presumably a list of functions that return unowned pointers or
references is sufficient. Boost Python has some fancy schmancy way of
doing this which we may copy but not for now.
When an object is returned by value, it will be copied into
dynamically allocated space and will be set to be owned by the caller.
Example of flattened C++ API
----------------------------
Using the example above, the following C declarations will be
generated.
// Every single non-builtin object that is passed in/out of the API
// is wrapped using a tagged type
// (which are themselves dynamically allocated)
struct xtagged
{
// type == x_error means void * ptr is a UTF-8 string
// indicating the error that occurred.
int type;
void * ptr;
int caller_owns;
};
typedef xtagged* xtagged_ptr;
// For readability only
typedef xtagged_ptr xerror;
typedef xtagged_ptr xanimal;
typedef xtagged_ptr xdog;
typedef xtagged_ptr xowner;
void
xerror_destroy(xerror self);
void
xtagged_ptr_destroy(xtagged_ptr self);
// animal::speak() cm = const member function
void
xanimal_speak_cm(xanimal self);
// animal::~animal() m = member function
void
xanimal_destroy_m(xanimal self);
xdog
xdog_dog();
void
xdog_speak_cm(xdog self);
void
xdog_destroy_m(xdog self);
xowner
xowner_owner(xanimal a);
xanimal
xowner_get_animal_cm(xowner const self);
void
xowner_destroy_m(xowner self);
An implementation of one of the functions:
void
xowner_destroy_m(xowner self)
{
// A lot of reinterpret_cast seems to be necessary here
// see below
delete &X_CAST_REF(,owner,self);
delete self;
}
#define X_CAST_REF(constness,type,taggedptr) \
checked_cast<constness type>(taggedptr, xtype_ ## type)
void *
cast_to_base(xtype base, xtagged_ptr child)
{
switch(child->type)
{
case xtype_animal:
return 0;
case xtype_dog:
{
dog * ptr = reinterpret_cast<dog*>(child->ptr);
switch(base)
{
case xtype_animal:
return dynamic_cast<animal*>(ptr);
default:
return 0;
}
}
case xtype_cat:
{
cat * ptr = reinterpret_cast<cat*>(child->ptr);
switch(base)
{
case xtype_animal:
return dynamic_cast<animal*>(ptr);
default:
return 0;
}
}
default:
X_ERROR("Unexpected type for child");
}
}
template<typename Type>
Type&
checked_cast(xtagged_ptr p, xtype t)
{
X_ASSERT(p);
X_ASSERT(p->ptr);
X_DEBUG("Attempting to cast from type " << p->type << " to " << t);
if(t == p->type)
{
X_DEBUG("types match exactly: " << t);
return *(reinterpret_cast<Type*>(p->ptr));
}
else if (Type * base = reinterpret_cast<Type*>(cast_to_base(t,p)))
{
X_DEBUG("passed in type " << p->type << " is child of " << t);
X_DEBUG("Base pointer: " << base << " child pointer: " << p->ptr);
return *base;
}
else
{
X_ERROR("Type " << p->type << " is not the same as "
<< t << " and type " << t
<< " is not one of the base classes of type " << p->type);
}
}
The Lisp side
-------------
The Lisp side for the above example using CFFI looks like:
(defcstruct xtagged-ptr
(type xtype)
(ptr :pointer)
(lisp-owns :boolean))
(defcfun "xtagged_ptr_destroy" :void
(p xtagged-ptr))
(defctype xanimal xtagged-ptr)
(defctype xdog xtagged-ptr)
(defctype xowner xtagged-ptr)
(defcfun "xanimal_speak_cm" :void
(self xanimal))
(defcfun "xdog_dog" xdog)
(defcfun "xdog_speak_cm" :void
(self xdog))
(defcfun "xanimal_destroy_m" :void
(self xanimal))
(defcfun "xdog_destroy_m" :void
(self xdog))
(defcfun "xowner_owner" xowner
(animal xanimal))
(defcfun "xowner_destroy_m" :void
(self xowner))
(defcfun "xowner_get_animal_cm" xanimal
(self xowner))
(defclass xobject ()
((pointer)))
(defclass animal (xobject)
())
(defclass dog (animal)
())
(defclass cat (animal)
())
(defclass owner (xobject)
())
;;; At this time, it is unclear to me whether the
;;; following code can be made less redundant by
;;; use of a single initialize-instance. Barring that
;;; a common function should be possible.
(defmethod initialize-instance :after ((self animal)
&rest args
&key
(xwrap-ptr nil))
"xwrap-ptr is provided when we are only wrapping a returned object"
(declare (ignore args))
(when xwrap-ptr
(setf (slot-value self 'pointer) xwrap-ptr)
(finalize self
(lambda ()
(if (foreign-slot-value xwrap-ptr 'xtagged-ptr 'lisp-owns)
(xanimal-destroy-m xwrap-ptr)
(xtagged-ptr-destroy xwrap-ptr))))))
(defmethod initialize-instance :after ((self dog)
&rest args
&key
(xwrap-ptr nil)
&allow-other-keys)
(declare (ignore args))
(let ((ptr (or xwrap-ptr (xdog-dog))))
(setf (slot-value self 'pointer) ptr)
(finalize self
(lambda ()
(if (foreign-slot-value ptr 'xtagged-ptr 'lisp-owns)
(xdog-destroy-m ptr)
(xtagged-ptr-destroy ptr))))))
(defmethod initialize-instance :after ((self cat)
&rest args
&key
(xwrap-ptr nil)
&allow-other-keys)
(declare (ignore args))
(let ((ptr (or xwrap-ptr (xcat-cat))))
(setf (slot-value self 'pointer) ptr)
(finalize self
(lambda ()
(if (foreign-slot-value ptr 'xtagged-ptr 'lisp-owns)
(xcat-destroy-m ptr)
(xtagged-ptr-destroy ptr))))))
(defmethod initialize-instance :after ((self owner)
&rest args
&key
animal
(xwrap-ptr nil)
&allow-other-keys)
(declare (ignore args))
(let ((ptr (or xwrap-ptr (xowner-owner (slot-value animal 'pointer)))))
(setf (slot-value self 'pointer) ptr)
(finalize self
(lambda ()
(if (foreign-slot-value ptr 'xtagged-ptr 'lisp-owns)
(xowner-destroy-m ptr)
(xtagged-ptr-destroy ptr))))))
(defparameter *type-hash*
(let ((hash (make-hash-table)))
(loop
for (k v) in '((:animal animal)
(:cat cat)
(:dog dog)
(:owner owner))
do (setf (gethash k hash) v))
hash))
(defgeneric animal-speak (animal))
(defmethod animal-speak ((self animal))
(xanimal-speak-cm (slot-value self 'pointer)))
(defmethod animal-speak ((self dog))
(xdog-speak-cm (slot-value self 'pointer)))
(defmethod animal-speak ((self cat))
(xcat-speak-cm (slot-value self 'pointer)))
(defun owner-get-animal (owner)
(let ((animal-ptr (xowner-get-animal-cm (slot-value owner 'pointer))))
(make-instance (gethash (foreign-slot-value animal-ptr 'xtagged-ptr 'type)
*type-hash*)
:xwrap-ptr animal-ptr)))
Client code
-----------
Assuming the above gets fully generated, the client can then use the
library as follows:
(defvar *animal* nil)
(defun make-dog-speak ()
(let* ((animal (make-instance 'dog))
(owner (make-instance 'owner
:animal animal)))
(animal-speak (owner-get-animal owner))
(setf *animal* animal)))
Inheritance and overriding virtual functions
--------------------------------------------
This has not yet been prototyped.
--------------040002010301090904060606--
On 29 Jun., 08:59, Sohail Somani <······@taggedtype.net> wrote:
> The general idea is to take the headers and generate the glue C,
> C++ and CFFI code. FWIW, this is how my brain works and I found
> it easiest to work this way.
[...]
> The only piece of software I know comes close to what I need is
> Verrazano which seems to want to implement the g++ ABI via CFFI.
Did you take a look at SWIG? There seems to be rudimentary support
for C++ in SWIG for CFFI: http://www.swig.org/Doc1.3/Lisp.html#Lisp_nn6
Matthias
From: Sohail Somani
Subject: Re: Wrapping a C++ library using ?FFI
Date:
Message-ID: <tfO9k.943$7%6.870@edtnps82>
Matthias Benkard wrote:
> On 29 Jun., 08:59, Sohail Somani <······@taggedtype.net> wrote:
> Did you take a look at SWIG? There seems to be rudimentary support
> for C++ in SWIG for CFFI: http://www.swig.org/Doc1.3/Lisp.html#Lisp_nn6
Yeah I did but only the Allegro support is quite complete.
Also, I don't like the way SWIG handles overloading.
Sohail Somani wrote:
> Sohail Somani wrote:
>>
>> I'm part of the way through writing some lisp gibberish that wraps a
>> C++ library.
>
> Ok, I've done most tricky things except for inheritance which I will
> tackle next. The general idea is to take the headers and generate the
> glue C, C++ and CFFI code. FWIW, this is how my brain works and I found
> it easiest to work this way.
>
> I've attached a brain dump. It's kinda long but I would appreciate
> comments if you have them.
>
> I suspect that for inheritance, the generated code will need to be more
> dynamic in that the cast_to_base function will need to do some sort of
> table lookups.
If that floats your boat, then have at it. Below, I've written a
quick brain dump of why I don't think its a good long-term solution.
But sometimes "worse is better".
As a quick note, ECL's "embedded C" strikes me as being cleaner than
the more portable "write a C interface and a CFFI binding to the C
interface". [Invoke the C++ compiler through some variant of (require
:cmp) (setf compiler::*cc* "g++") (compile-file "c++binding.lisp")]
Portability gains from using the C layer are lost by the need to keep
the build environment compatible with that of the C++ library. This
includes getting all the include paths and #defines and other stuff
100% compatible with what the real build system uses. A little -m64
or -m32 can have far-reaching effects.
IIRC, boost::python does the best here since it is easily injected
into the regular build system. Systems like SWIG require the user to
scan command lines and try copying over all relevant details. And C++
template instantiation has enough bugs in real compilers that I don't
trust any binding tool (other than the gccxml kludge) to get this
completely bug-compatible with any
Another minus for the C layer -- its an abstraction which gains a
little implementation simplicity at the cost of a little runtime
performance and increased end-user pain. The cost of a C trampoline
is easily calculated. But the real cost is in requiring end-users to
install (the right version of) a C++ lib and then build a C interface
lib (with the right flags). People may rightly say "this sounds
brittle" and easily broken whenever the C++ lib is upgraded.
Taking the CFFI approach and implementing the ABI in Lisp isn't
particularly brittle (C++ ABIs are already brittle); it just takes
more work to set up (esp. for some compilers). As mentioned before
before, the Itanium ABI [1] has been standardized for use by all
x86_64 compilers (and with modifications for other 64-bit platforms).
This (and what GCC/linux traditionally uses) makes the C++ calling
convention exactly the same as the C convention, just with a hidden
"this" parameter being passed, implicit pointer arithmetic,
constructors/dtors being called, etc. Other legacy (esp MS) ABIs may
require more work; [2] is an excellent reference.
[1] http://www.codesourcery.com/cxx-abi/abi.html
[2] Manual 5 on http://www.agner.org/optimize/
Later,
Daniel
From: Sohail Somani
Subject: Re: Wrapping a C++ library using ?FFI
Date:
Message-ID: <4JP9k.965$7%6.956@edtnps82>
Thanks for your feedback!
D Herring wrote:
> Sohail Somani wrote:
>
> If that floats your boat, then have at it. Below, I've written a quick
> brain dump of why I don't think its a good long-term solution. But
> sometimes "worse is better".
I like this:
http://www.jwz.org/doc/worse-is-better.html
> As a quick note, ECL's "embedded C" strikes me as being cleaner than the
> more portable "write a C interface and a CFFI binding to the C
> interface". [Invoke the C++ compiler through some variant of (require
> :cmp) (setf compiler::*cc* "g++") (compile-file "c++binding.lisp")]
I don't believe there are any examples of using ECL with a non-trivial
C++ library. I would like to see it though. Anyway, I don't use ECL.
> Portability gains from using the C layer are lost by the need to keep
> the build environment compatible with that of the C++ library. This
> includes getting all the include paths and #defines and other stuff 100%
> compatible with what the real build system uses. A little -m64 or -m32
> can have far-reaching effects.
I agree. Hopefully it shouldn't be too much of a problem.
> IIRC, boost::python does the best here since it is easily injected into
> the regular build system. Systems like SWIG require the user to scan
> command lines and try copying over all relevant details.
Not sure I understand what you are getting at here. Any sane build
system lets you factor out the common flags.
And C++
> template instantiation has enough bugs in real compilers that I don't
> trust any binding tool (other than the gccxml kludge) to get this
> completely bug-compatible with any
I am probably not too interested in compilers other than g++, VC++ and
Intel which are quite happy with templates.
> Another minus for the C layer -- its an abstraction which gains a little
> implementation simplicity at the cost of a little runtime performance
> and increased end-user pain. The cost of a C trampoline is easily
> calculated. But the real cost is in requiring end-users to install (the
> right version of) a C++ lib and then build a C interface lib (with the
> right flags). People may rightly say "this sounds brittle" and easily
> broken whenever the C++ lib is upgraded.
True. Though if the C++ library maintains binary compatibility, it isn't
a problem.
> Taking the CFFI approach and implementing the ABI in Lisp isn't
> particularly brittle (C++ ABIs are already brittle); it just takes more
> work to set up (esp. for some compilers).
I don't see how understanding the intricacies of an ABI are any less
brittle (from the point of view of conforming to them!)
From what I recall, even g++ broke its own ABI a few times.
> As mentioned before before,
> the Itanium ABI [1] has been standardized for use by all x86_64
> compilers (and with modifications for other 64-bit platforms). This
> (and what GCC/linux traditionally uses) makes the C++ calling convention
> exactly the same as the C convention, just with a hidden "this"
> parameter being passed, implicit pointer arithmetic, constructors/dtors
> being called, etc. Other legacy (esp MS) ABIs may require more work;
> [2] is an excellent reference.
I don't gain much by restricting myself to 64-bit compilers that
implement the C++ ABI.
I'd like to see all full, uncompromising C++ ABIs implemented in Lisp,
but I am *definitely* not interested enough to make it so (nor am I
holding my breath!)
Sohail
Sohail Somani wrote:
> Thanks for your feedback!
>
> D Herring wrote:
>> Sohail Somani wrote:
> >
>> If that floats your boat, then have at it. Below, I've written a
>> quick brain dump of why I don't think its a good long-term solution.
>> But sometimes "worse is better".
>
> I like this:
>
> http://www.jwz.org/doc/worse-is-better.html
>
>> As a quick note, ECL's "embedded C" strikes me as being cleaner than
>> the more portable "write a C interface and a CFFI binding to the C
>> interface". [Invoke the C++ compiler through some variant of (require
>> :cmp) (setf compiler::*cc* "g++") (compile-file "c++binding.lisp")]
>
> I don't believe there are any examples of using ECL with a non-trivial
> C++ library. I would like to see it though. Anyway, I don't use ECL.
Agreed; I've just started using ECL in the past couple of weeks,
simply to embed it in existing C++ apps. Look (or bug me) for a
tutorial-style writeup in a few weeks.
>> Portability gains from using the C layer are lost by the need to keep
>> the build environment compatible with that of the C++ library. This
>> includes getting all the include paths and #defines and other stuff
>> 100% compatible with what the real build system uses. A little -m64
>> or -m32 can have far-reaching effects.
>
> I agree. Hopefully it shouldn't be too much of a problem.
YMMV.
>> IIRC, boost::python does the best here since it is easily injected
>> into the regular build system. Systems like SWIG require the user to
>> scan command lines and try copying over all relevant details.
>
> Not sure I understand what you are getting at here. Any sane build
> system lets you factor out the common flags.
Some systems just use a few common flags. Others...
> And C++
>> template instantiation has enough bugs in real compilers that I don't
>> trust any binding tool (other than the gccxml kludge) to get this
>> completely bug-compatible with any
>
> I am probably not too interested in compilers other than g++, VC++ and
> Intel which are quite happy with templates.
I didn't say they weren't happy with templates -- I said they have
subtle quirks. Template instantiation rules are complicated. Many
headers (e.g. Boost.org) detect compiler-specific #defines to
determine which interface to export. Such subtleties may arise in
corner cases; but the C layer should handle most of them.
>> Another minus for the C layer -- its an abstraction which gains a
>> little implementation simplicity at the cost of a little runtime
>> performance and increased end-user pain. The cost of a C trampoline
>> is easily calculated. But the real cost is in requiring end-users to
>> install (the right version of) a C++ lib and then build a C interface
>> lib (with the right flags). People may rightly say "this sounds
>> brittle" and easily broken whenever the C++ lib is upgraded.
>
> True. Though if the C++ library maintains binary compatibility, it isn't
> a problem.
Few C++ libraries maintain binary compatibility. Even GNU's
implementation of libstdc++ has some ABI issues between versions (my
favorite affects std::string and only activates when a string is
copied via one code path).
>> Taking the CFFI approach and implementing the ABI in Lisp isn't
>> particularly brittle (C++ ABIs are already brittle); it just takes
>> more work to set up (esp. for some compilers).
>
> I don't see how understanding the intricacies of an ABI are any less
> brittle (from the point of view of conforming to them!)
>
> From what I recall, even g++ broke its own ABI a few times.
ABI issues can be itemized, detected, and worked with. The largest
g++ ABI change, from 2/early 3 days to the present 3.3[?]/4.x ABI, was
well-documented and correlates with the release of the Itanium ABI.
>> As mentioned before before, the Itanium ABI [1] has been standardized
>> for use by all x86_64 compilers (and with modifications for other
>> 64-bit platforms). This (and what GCC/linux traditionally uses) makes
>> the C++ calling convention exactly the same as the C convention, just
>> with a hidden "this" parameter being passed, implicit pointer
>> arithmetic, constructors/dtors being called, etc. Other legacy (esp
>> MS) ABIs may require more work; [2] is an excellent reference.
>
> I don't gain much by restricting myself to 64-bit compilers that
> implement the C++ ABI.
>
> I'd like to see all full, uncompromising C++ ABIs implemented in Lisp,
> but I am *definitely* not interested enough to make it so (nor am I
> holding my breath!)
Hence, I prefixed my message with a "then have at it". :)
Hopefully it will only take another year or two before "the better"
solution is generally available.
- Daniel
From: Sohail Somani
Subject: Re: Wrapping a C++ library using ?FFI
Date:
Message-ID: <v9R9k.981$7%6.708@edtnps82>
D Herring wrote:
> Sohail Somani wrote:
>>
>> I don't believe there are any examples of using ECL with a non-trivial
>> C++ library. I would like to see it though. Anyway, I don't use ECL.
>
> Agreed; I've just started using ECL in the past couple of weeks, simply
> to embed it in existing C++ apps. Look (or bug me) for a tutorial-style
> writeup in a few weeks.
This is me and the thousands upon thousands of potential Lisp users
asking you for a tutorial-style writeup in a few weeks.
>>> Portability gains from using the C layer are lost by the need to keep
>>> the build environment compatible with that of the C++ library. This
>>> includes getting all the include paths and #defines and other stuff
>>> 100% compatible with what the real build system uses. A little -m64
>>> or -m32 can have far-reaching effects.
>>
>> I agree. Hopefully it shouldn't be too much of a problem.
>
> YMMV.
I've more or less tamed the portable builds problem so I am less
concerned about portable builds than I am about portable code.
>>> IIRC, boost::python does the best here since it is easily injected
>>> into the regular build system. Systems like SWIG require the user to
>>> scan command lines and try copying over all relevant details.
>>
>> Not sure I understand what you are getting at here. Any sane build
>> system lets you factor out the common flags.
>
> Some systems just use a few common flags. Others...
*shudder*
> I didn't say they weren't happy with templates -- I said they have
> subtle quirks. Template instantiation rules are complicated. Many
> headers (e.g. Boost.org) detect compiler-specific #defines to determine
> which interface to export. Such subtleties may arise in corner cases;
> but the C layer should handle most of them.
This is the advantage of using a C layer. The C++ compiler handles it
all for us.
>> True. Though if the C++ library maintains binary compatibility, it
>> isn't a problem.
>
> Few C++ libraries maintain binary compatibility. Even GNU's
> implementation of libstdc++ has some ABI issues between versions (my
> favorite affects std::string and only activates when a string is copied
> via one code path).
I don't know exactly what you are talking about but inlined code cannot
have any hope of binary compatibility.
>>> Taking the CFFI approach and implementing the ABI in Lisp isn't
>>> particularly brittle (C++ ABIs are already brittle); it just takes
>>> more work to set up (esp. for some compilers).
>>
>> I don't see how understanding the intricacies of an ABI are any less
>> brittle (from the point of view of conforming to them!)
>>
>> From what I recall, even g++ broke its own ABI a few times.
>
> ABI issues can be itemized, detected, and worked with. The largest g++
> ABI change, from 2/early 3 days to the present 3.3[?]/4.x ABI, was
> well-documented and correlates with the release of the Itanium ABI.
You've just outlined my problem. I have no intention of letting other
people break my code on their own whims.
>> I'd like to see all full, uncompromising C++ ABIs implemented in Lisp,
>> but I am *definitely* not interested enough to make it so (nor am I
>> holding my breath!)
>
> Hence, I prefixed my message with a "then have at it". :)
Indeed you did :-)
> Hopefully it will only take another year or two before "the better"
> solution is generally available.
I would be very happy to use this once available (and if it was portable
to most major compiler ABIs.)
Sohail