Greetings all!
Has anybody managed to get Uncommonsql (in particular) or the alien
foreign interface functions (in general) to work under OpenBSD 3.0 with
CMUCL-18d? One problem seems to be related to the <load-foreign.csh>
shell script. The line (which is the default ld command):
ld -N -A /usr/local/cmucl-18d/bin/lisp -T E000000 -o /tmp/tmp30565A \
/usr/local/pgsql/lib/libpq.a -lssl -lc
keeps coming back with:
ld: /usr/local/cmucl-18d/bin/lisp: Bad file descriptor
Now, this file _does_ exist, and is simply the binary from the
cmucl-18d tar balls built for OpenBSD. <lisp> itself works fine.
Any ideas out there? Do I need to rebuild cmucl? Is there a variation
on the ld command I need to use?
Thanks!
Ed
Whoops, I was trying a lot of variants trying to figure out the problem
and I accidentally pasted the wrong lines.
The original <load-foreign.csh> uses the line:
ld -N -A /usr/local/cmucl-18d/bin/lisp -T E000000 -o /tmp/tmp30565A \
/usr/local/pgsql/lib/libpq.so.2 -lssl -lc
and this results in the error:
ld: No reference to __DYNAMIC
Regards,
Ed
In article <············@tabloid.uwaterloo.ca>,
Ed Dengler <········@mud.uwaterloo.ca> wrote:
>Greetings all!
>
>Has anybody managed to get Uncommonsql (in particular) or the alien
>foreign interface functions (in general) to work under OpenBSD 3.0 with
>CMUCL-18d? One problem seems to be related to the <load-foreign.csh>
>shell script. The line (which is the default ld command):
>
> ld -N -A /usr/local/cmucl-18d/bin/lisp -T E000000 -o /tmp/tmp30565A \
> /usr/local/pgsql/lib/libpq.a -lssl -lc
>
>keeps coming back with:
>
> ld: /usr/local/cmucl-18d/bin/lisp: Bad file descriptor
>
>Now, this file _does_ exist, and is simply the binary from the
>cmucl-18d tar balls built for OpenBSD. <lisp> itself works fine.
>
>Any ideas out there? Do I need to rebuild cmucl? Is there a variation
>on the ld command I need to use?
>
>Thanks!
>Ed
>
In article <············@tabloid.uwaterloo.ca>,
Ed Dengler <········@mud.uwaterloo.ca> wrote:
>Whoops, I was trying a lot of variants trying to figure out the problem
>and I accidentally pasted the wrong lines.
>
>The original <load-foreign.csh> uses the line:
>
> ld -N -A /usr/local/cmucl-18d/bin/lisp -T E000000 -o /tmp/tmp30565A \
> /usr/local/pgsql/lib/libpq.so.2 -lssl -lc
>
>and this results in the error:
>
> ld: No reference to __DYNAMIC
>
>Regards,
>Ed
>
I just received a suggestion (thanks kloo!) of using -Bshareable.
Unfortunately:
=> ld -Bshareable -N -A /usr/local/cmucl-18d/bin/lisp -T E000000 \
-o /tmp/tmp30565A /usr/local/pgsql/lib/libpq.so.2 -lssl -lc
gives the other problem of:
ld: /usr/local/cmucl-18d/bin/lisp: Bad file descriptor
Could there be a problem with the file format itself?
=> objdump -f /usr/local/cmucl-18d/bin/lisp
/usr/local/cmucl-18d/bin/lisp: file format a.out-i386-netbsd
architecture: i386, flags 0x000001be:
EXEC_P, HAS_LINENO, HAS_DEBUG, HAS_SYMS, HAS_LOCALS, WP_TEXT, D_PAGED
start address 0x00001020
=> objdump -f /usr/bin/ld
/usr/bin/ld: file format a.out-i386-netbsd
architecture: i386, flags 0x000001c2:
EXEC_P, DYNAMIC, WP_TEXT, D_PAGED
start address 0x00001020
Regards,
Ed
········@mud.uwaterloo.ca (Ed Dengler) writes:
> Whoops, I was trying a lot of variants trying to figure out the problem
> and I accidentally pasted the wrong lines.
>
> The original <load-foreign.csh> uses the line:
>
> ld -N -A /usr/local/cmucl-18d/bin/lisp -T E000000 -o /tmp/tmp30565A \
> /usr/local/pgsql/lib/libpq.so.2 -lssl -lc
>
> and this results in the error:
>
> ld: No reference to __DYNAMIC
Hmmm, the overall behaviour of OpenBSD's ld in combination with the
libpq libs seems strange.
The OpenBSD port doesn't support shared objects, since the binary is
statically linked, I guess.
However it seems that OpenBSD's ld has problems handling archive
files when called with the -A option, so that doesn't seem to be an
option either.
Loading normal *.o files that I compile works ok, though.
But when I explode the libpq.a lib into individual *.o files, and try
to link against those, ld again complains with:
ld: No reference to __DYNAMIC
though no shared lib is in sight.
I guess I don't understand OpenBSD's ld or something.
Maybe recompiling the libpq static library from source might work...
So for the moment, until I find the time to investigate this further,
or someone enlightens me, I don't think that libpq is going to work
with cmucl.
Recompiling cmucl isn't likely going to help here, unless someone puts
in the work that's required to make dynamic linking of the lisp binary
work, which is a bit non-trivial.
Regs, Pierre.
PS: Usually, using the mailing lists at cons.org for questions of this kind is going to be more succesful...
--
Pierre R. Mai <····@acm.org> http://www.pmsf.de/pmai/
The most likely way for the world to be destroyed, most experts agree,
is by accident. That's where we come in; we're computer professionals.
We cause accidents. -- Nathaniel Borenstein
Greetings all!
In article <··············@orion.bln.pmsf.de>,
Pierre R. Mai <····@acm.org> wrote:
>········@mud.uwaterloo.ca (Ed Dengler) writes:
>
>> Whoops, I was trying a lot of variants trying to figure out the problem
>> and I accidentally pasted the wrong lines.
>>
>> The original <load-foreign.csh> uses the line:
>>
>> ld -N -A /usr/local/cmucl-18d/bin/lisp -T E000000 -o /tmp/tmp30565A \
>> /usr/local/pgsql/lib/libpq.so.2 -lssl -lc
>>
>> and this results in the error:
>>
>> ld: No reference to __DYNAMIC
>
>Hmmm, the overall behaviour of OpenBSD's ld in combination with the
>libpq libs seems strange.
>
>The OpenBSD port doesn't support shared objects, since the binary is
>statically linked, I guess.
The most strange part is that OpenBSD is not that far removed from
FreeBSD/NetBSD, so I find it interesting if, as seems to be suggested,
the FreeBSD version is compiled/linked to support shared objects, but
the OpenBSD version is not.
>However it seems that OpenBSD's ld has problems handling archive
>files when called with the -A option, so that doesn't seem to be an
>option either.
>
>Loading normal *.o files that I compile works ok, though.
>
>But when I explode the libpq.a lib into individual *.o files, and try
>to link against those, ld again complains with:
>
>ld: No reference to __DYNAMIC
>
>though no shared lib is in sight.
>
>I guess I don't understand OpenBSD's ld or something.
>
>Maybe recompiling the libpq static library from source might work...
I thought that PostgreSQL's libpq.a should be essentially statically linked
(or can be used as such). I'll try recompiling the library and see if I
can force a fully static version.
>So for the moment, until I find the time to investigate this further,
>or someone enlightens me, I don't think that libpq is going to work
>with cmucl.
This is quite sad. I find it interesting the previous threads that
talk about how Lisp is such a good environment for development. I was
hoping to use CMUCL to derive a Lisp-based cfengine (cfengine does not
seem to work on OpenBSD due to a large number of networking code
issues) and a correlation engine for use in security work. I really
did not want to fall back back onto Java or C. But not having access
to a database easily brings up the same complaint as I had about
scheme: I don't care how good a language it is if I cannot get work
that needs to be done, done.
>Recompiling cmucl isn't likely going to help here, unless someone puts
>in the work that's required to make dynamic linking of the lisp binary
>work, which is a bit non-trivial.
>
>Regs, Pierre.
>
>PS: Usually, using the mailing lists at cons.org for questions of this kind is going to be more succesful...
Thanks for the response! Maybe I'll try reposting to the mailing list.
Regards!
Ed
The world rejoiced as ········@mud.uwaterloo.ca (Ed Dengler) wrote:
> I was hoping to use CMUCL to derive a Lisp-based cfengine
That would be a really cool thing...
cfengine is a rather neat system, albeit with a somewhat irritating
syntax. I'd expect it to be a bit nicer with an s-exp-based syntax...
If you're trying to speculate on different approaches to this, you
might consider, as data points for alternatives:
a) There's a quite slick Berkeley-DB interface, presently specific to
CMUCL. <http://www.actrix.gen.nz/users/mycroft/db/>
b) CLISP is probably a bit more portable, and that's worth
something. A configuration engine implemented using CLISP
might be adapted to run on "Some Of Those Platforms We Love
to Hate" rather more easily than CMUCL, and it's a WIN to
have better automated configuration management tools
available, even on Evil Empire platforms.
After all, one of the essays on cfengine makes, as its thesis, the
idea that ACLs are only likely to be usable if you have some sort of
automated tool to apply ACL policies.
--
(reverse (concatenate 'string ·············@" "sirhc"))
http://www.cbbrowne.com/info/linuxsysconfig.html
Rule of Scarcity
"If it's not allocated by a market, then it's more expensive than
money."
-- Mark Miller
········@mud.uwaterloo.ca (Ed Dengler) writes:
> >The OpenBSD port doesn't support shared objects, since the binary is
> >statically linked, I guess.
>
> The most strange part is that OpenBSD is not that far removed from
> FreeBSD/NetBSD, so I find it interesting if, as seems to be suggested,
> the FreeBSD version is compiled/linked to support shared objects, but
> the OpenBSD version is not.
Actually, from my experience porting CMUCL from FreeBSD to Open- and
NetBSD, the various *BSDs aren't all that similar when it comes to
low-level stuff like binary formats, linking or signals. Oftentimes,
one of the BSD's is more similar to Linux than to their kin in one
aspect or another, for example.
OpenBSD is still non-ELF by default (at least it was on 2.9, which is
what I'm still using), whereas FreeBSD did change to ELF as their
default binary format some time ago. IIRC, then FreeBSD also changed
to the GNU binutils ld at the time, whereas OpenBSD still uses their
own ld, etc.
So there is little code sharing in that part between the FreeBSD and
OpenBSD ports.
BTW foreign linking of shared objects was broken on FreeBSD for
some time, and was only (semi-)fixed recently, and still requires
special hacks, since FreeBSD's dlopen doesn't automatically pull in
transitive shared objects when opening non-shared objects... This
should hopefully get fixed for the next release.
> >So for the moment, until I find the time to investigate this further,
> >or someone enlightens me, I don't think that libpq is going to work
> >with cmucl.
>
> This is quite sad. I find it interesting the previous threads that
> talk about how Lisp is such a good environment for development. I was
> hoping to use CMUCL to derive a Lisp-based cfengine (cfengine does not
> seem to work on OpenBSD due to a large number of networking code
> issues) and a correlation engine for use in security work. I really
> did not want to fall back back onto Java or C. But not having access
> to a database easily brings up the same complaint as I had about
> scheme: I don't care how good a language it is if I cannot get work
> that needs to be done, done.
Well, the defect you are seeing is not one with CL the language, but
only with one implementation on one, only recently, supported
platform.
There are other implementations of Common Lisp, that you might want to
look at:
- SBCL, a close relative of CMUCL, also sports a port to OpenBSD, and
they might have solved those issues. I'll look into that myself,
and see if we can take aboard those fixes. But my time is currently
fairly limited, due to clients' contracts...
- CLISP is another implementation that works on OpenBSD, IIRC, and it
seems worth a look.
There might be others, since I haven't kept track of OpenBSD for some
time.
Regs, Pierre.
--
Pierre R. Mai <····@acm.org> http://www.pmsf.de/pmai/
The most likely way for the world to be destroyed, most experts agree,
is by accident. That's where we come in; we're computer professionals.
We cause accidents. -- Nathaniel Borenstein
Greetings all!
In article <··············@orion.bln.pmsf.de>,
Pierre R. Mai <····@acm.org> wrote:
>Actually, from my experience porting CMUCL from FreeBSD to Open- and
>NetBSD, the various *BSDs aren't all that similar when it comes to
>low-level stuff like binary formats, linking or signals. Oftentimes,
>one of the BSD's is more similar to Linux than to their kin in one
>aspect or another, for example.
>
>OpenBSD is still non-ELF by default (at least it was on 2.9, which is
>what I'm still using), whereas FreeBSD did change to ELF as their
>default binary format some time ago. IIRC, then FreeBSD also changed
>to the GNU binutils ld at the time, whereas OpenBSD still uses their
>own ld, etc.
Yep, as of OpenBSD 3.0, it looks like it is still an old customized
version of Gnu ld.
>So there is little code sharing in that part between the FreeBSD and
>OpenBSD ports.
Hmmm, interesting. Is it correct that the code for the FreeBSD portion
of the alien foreign interface was developed due to problems with
FreeBSD's dlopen()? If so, might it be possible that OpenBSD's
dlopen() might work directly (rather than going through a CSH script to
statically position a library)?
Can anyone point out a clean/simple description of how to recompile CMUCL
(or at least, how I would go about recompiling a portion such as alien
and reintegrating it into the CMUCL binary)? I would like to attempt
turning the direct use of dlopen() back on for OpenBSD (or is there other
problems with the use of dlopen() that would make this a fruitless line
of investigation?).
I tried looking at cons.org/cmucl, and through DejaNews, but all I
could gather is that recompiling CMUCL is difficult and everybody
recommends just getting the latest tarballs.
Regards,
Ed
········@mud.uwaterloo.ca (Ed Dengler) writes:
> >So there is little code sharing in that part between the FreeBSD and
> >OpenBSD ports.
>
> Hmmm, interesting. Is it correct that the code for the FreeBSD portion
> of the alien foreign interface was developed due to problems with
> FreeBSD's dlopen()? If so, might it be possible that OpenBSD's
> dlopen() might work directly (rather than going through a CSH script to
> statically position a library)?
I've found the time to dig more into the OpenBSD dlopen and FFI
issues, and have now successfully converted the OpenBSD port to using
dlopen+ld, like Linux and FreeBSD+ELF. This works out well, and at
least on OpenBSD 2.9, I can now load shared-objects, single non-shared
objects and archive files without undue problems.
I've put a current binary distribution including those changes in the
binaries directory of the CMUCL distribution site (mirrors should
propagate changes in the next couple of days).
> Can anyone point out a clean/simple description of how to recompile CMUCL
> (or at least, how I would go about recompiling a portion such as alien
> and reintegrating it into the CMUCL binary)? I would like to attempt
> turning the direct use of dlopen() back on for OpenBSD (or is there other
> problems with the use of dlopen() that would make this a fruitless line
> of investigation?).
There are a number of rebuilding scripts and instructions around,
maybe the EncyCMUCLopedia has links to them?
The set of scripts that I use to maintain CMUCL and a little bit of
instructions in using them are at http://www.pmsf.de/pmai/CMUCL.html
The only thing that is not fully automatic is the generation of the
contents of the setenv.lisp file, which needs to set desired
*features*. At the moment, I use the following for OpenBSD:
;;; Put code to massage *features* list here...
(in-package :user)
(pushnew :openbsd *features*)
(pushnew :bsd *features*)
(pushnew :i486 *features*)
(pushnew :mp *features*)
(pushnew :hash-new *features*)
(pushnew :random-mt19937 *features*)
(pushnew :conservative-float-type *features*)
(pushnew :gencgc *features*)
;;; Version tags
(pushnew :cmu18d *features*)
(pushnew :cmu18 *features*)
(setf *features* (remove :cmu17 *features*))
(setf *features* (remove :cmu18c *features*))
So to recompile, you really only need to get the sources and the
buildscripts, untar them in the same directory, so that the sources
are in src, and execute:
# Create a new target directory structure/config for OpenBSD:
./create-target.sh openbsd OpenBSD_gencgc OpenBSD
# edit openbsd/setenv.lisp to contain the above contents, or whatever
# you want...
# Recompile the lisp world, and dump a new kernel.core:
./build-world.sh openbsd lisp # Or whatever you need to invoke your
# current lisp binary+core
# If build-world tells you (as it will the first time) that:
# "The C header file has changed. Be sure to re-compile the startup
# code."
# You 'll need to start rebuild-lisp.sh to do that, and then reinvoke
# build-world.sh:
# Recompile lisp binary itself:
./rebuild-lisp.sh openbsd
# Restart build-world.sh now:
./build-world.sh openbsd lisp
# Now we populate the kernel.core with further compiled files,
# and dump the final lisp.core file:
./load-world.sh openbsd "18d+ 2002-05-06"
# The second argument above is the version number that the built
# core will announce. Please always put the build-date and some
# other information in there, to make it possible to differentiate
# those builds from official builds, which only contain the release.
Now you should have a new lisp.core, which you can start with
./openbsd/lisp/lisp -core ./openbsd/lisp/lisp.core -noinit -nositeinit
There are further scripts to build CLX,CMUCL/Motif,Hemlock,etc., and
to create distribution tarballs, or to clean target directories.
The above will work for recompiling sources that correspond to your
current binary. Certain changes in the sources necessitate some
bootstrapping code. In that case you'll need to put the relevant
bootstrap code (from sources like src/bootfiles/.../*, or wherever)
into the file <target>/bootstrap.lisp, before recompiling.
Under certain rare circumstances (i.e. hasn't happened since before
18c), bootstrap code will not be sufficient, and a cross-compilation
is necessary. In that case, contact the cmucl-imp mailing list, for
information. I've got cross-compile scripts, but those aren't yet
ready for general consumption.
> I tried looking at cons.org/cmucl, and through DejaNews, but all I
> could gather is that recompiling CMUCL is difficult and everybody
> recommends just getting the latest tarballs.
It has become much easier in recent times, though it is still easier
for most users to just grab the tarballs, and/or bug some maintainer.
Hope this helps,
Regs, Pierre.
--
Pierre R. Mai <····@acm.org> http://www.pmsf.de/pmai/
The most likely way for the world to be destroyed, most experts agree,
is by accident. That's where we come in; we're computer professionals.
We cause accidents. -- Nathaniel Borenstein
········@mud.uwaterloo.ca (Ed Dengler) writes:
> This is quite sad. I find it interesting the previous threads that
> talk about how Lisp is such a good environment for development. I was
> hoping to use CMUCL to derive a Lisp-based cfengine (cfengine does not
> seem to work on OpenBSD due to a large number of networking code
> issues) and a correlation engine for use in security work. I really
> did not want to fall back back onto Java or C. But not having access
> to a database easily brings up the same complaint as I had about
> scheme: I don't care how good a language it is if I cannot get work
> that needs to be done, done.
There's another set of PostgreSQL bindings at
http://ww.telent.net/cliki/Pg that you could look at; it doesn't use
foreign libraries at all, so will likely not have the problems you
describe.
Hope that helps,
Rudi