I'd like to generate a bit of discussion on how best to package Lisp
code for open source distribution. Just to establish a frame of
reference, what I ultimately would like to see come out of this is a
system at *least* as functional as what CPAN provides for the perl
community.
Since many of the contributors here are likely unfamiliar with CPAN,
let me start by describing what I know and like about it.
The first package you generally fetch from CPAN (assuming you're
using a system that didn't come with perl pre-installed) is perl
itself. The perl installation is the usual rote of running
configuration scripts and answering obscure technical questions.
But where perl first steps out from the crowd is that the answers
to these obscure questions (and the factoids learned by the
scripts) are placed into a configuration registry (flat file) for
later reference by *every other* module you download from CPAN.
So, you arduously collect this stuff once, and then it's available
to all later installation procedures via a well defined standard
module interface. [And yes, of course, this has its problems --
if you get an answer wrong there's a significant ripple effect.
But if you manage to get it right once, you rarely need to think
about it again.]
The information collected includes such useful tidbits as how to
compile a shared library on your platform, where in the local file
system to install such a library, which of a host of optional
extension librarys (Berkeley DB, gdbm, etc.) you have available, etc.
Now, the *second* package the seasoned CPAN user will fetch is called
CPAN. This provides you with an interactive top-level that has simple
commands for searching the CPAN archives by author or module name,
fetching selected modules and unpacking, building, testing, and
installing them. The modules themselves are package with meta-info
that the "CPAN shell" can use to ferret out dependancy information.
So, for instance, as soon as you try to build module Foo, modules
Bar and Baz are fetched and built because they're listed as
prerequisites.
[I'm thinking that MK:DEFSYSTEM is the right place to start here for
Lisp -- but let's not get ahead of ourselves...]
An important point here -- once some knowledgeable individual /
organization has gotten a host configured to this point, an
absolute neophyte perl user can successfully locate, fetch, build,
test and install any of hundreds of very useful perl packages. I
personally believe this single fact has a great deal to do with the
popularity and success of perl.
Obviously, all this and more could be integrated into a package and
an archive infrastructure for Common Lisp. The perl stuff really is
amazingly low-tech -- an ordinary FTP server is all that's required
to act as a CPAN mirror site -- and yet it does the job quite well.
The biggest barrier to an infrastructure like this for Common Lisp is
probably that none of the existing base of freely available code uses
it. The package system may not be ideally suited for this sort of
task -- but it's certainly as good as perl's. There's already a
useful base of code in CL-HTTP for doing HTTP requests from Lisp.
We could probably standardize on some logical pathname host names
for defining "standard" locations to install packages.
I'm willing to spend some time on this, and to make a host site
available at least long enough to experiment with the idea a bit.
I invite anyone who's interested in a bit of dialog on this to chime
right in. My primary motivation is that I have a couple of packages
nearing completion that I'd like to make available -- and I just
went through the exercise of defining a configuration scheme that
let me install from the same tarball / archive on both Unix and NT
machines for my own internal use.
-- Kaelin
From: Reini Urban
Subject: Re: Packaging for open-source distribution
Date:
Message-ID: <3811c3dd.6159887@judy>
Some things to add what is fine/problematic about CPAN, and should be
honored by a probable "CLAN".
(I'm CPAN author, =contributor, and frequent user)
CPAN is based on a low-tech ftp/website. Retrieving info and setting up
mirrors is very easy, in contrast to setup CL-HTTPD which would be a
fine idea but maybe too hi-tech.
The CPAN.pm module which helps in fetching modules via command-line
would be nice if it would work on all platforms on all configurations.
But the situation with perl is similar to lisp, complicated platforms
such as NT are not supported well enough, so e.g. CPAN.pm which is based
on libnet, which does not work out of the box on NT, is not usable to
95% of the users (adding the complexity of communicating via a proxy).
(We had two competing net libraries, Graham Barr's libnet which did not
work on Win32 until recently against Jeffrey Friedl's old network.pl)
The fact that it works nevertheless is because a typical "Worse is
better" example. one does not require CPAN.pm, it is only useful.
One only requires MakeMaker.pm which works out-of-the-box, even on the
ActiveState perl-for-dummies package.
The hi-tech aspect of CPAN which makes it highly usable is centralized,
the redirection script on http://www.perl.com/CPAN which redirects the
user the nearest mirror site. (based on a database, studying the user IP
and probably the traceroute)
A CLAN site should be two-fold:
Based on CL-HTTPD with highly interactive impressing features for a
web-mirror, plus based on a simple ftp mirror without the fancy stuff.
No major mirror site will ever install CL-HTTPD just to make this CLAN
stuff work.
But setting up a mirror strategy is not that important for Lisp modules
as they are for perl or TeX. We only have the maintainer problem (MK,
JCMa) We would not need the help of the major mirrors though it would be
nice to be carried on them. (as CPAN and CTAN)
Third, still waiting for a CL-HTTPD thing to happen is naive.
The only important point will be a DEFSYSTEM-alike, a simple installer
on the user-side, hopefully implementation and platform independent,
checking the requirements and doing the file unpacking and copying.
BTW: I'm planning such a thing for the AutoLISP community.
This community has no compatibility, only quality problems.
So it is completely different to the lisp community.
I expect another two years until it will work out good enough to be
accepted.
Kaelin Colclasure wrote:
>I'd like to generate a bit of discussion on how best to package Lisp
>code for open source distribution. Just to establish a frame of
>reference, what I ultimately would like to see come out of this is a
>system at *least* as functional as what CPAN provides for the perl
>community.
--
Reini Urban
http://xarch.tu-graz.ac.at/autocad/news/faq/autolisp.html
From: Andy Freeman
Subject: Re: Packaging for open-source distribution
Date:
Message-ID: <7ut6me$5nu$1@nnrp1.deja.com>
In article <···············@newsin1.ispchannel.com> "Kaelin Colclasure"
<······@everest.com> wrote:
> I'd like to generate a bit of discussion on how best to package Lisp
> code for open source distribution.
Great!
> useful base of code in CL-HTTP for doing HTTP requests from Lisp.
> We could probably standardize on some logical pathname host names
> for defining "standard" locations to install packages.
Now it's unusable, at least for me.
For various social/technical reasons, I have to use Win9x for
net access. However, I actually work in linux, without net access.
It's a single computer, dual-boot, so I move things from the net
to linux by downloading them using Win9x, rebooting to linux, and
then moving them from the Win partition to a linux partition.
Given the interest in dual-boot setup, I'm not the only person
who works this way.
There are other organizational arrangements that make automated
fetching unusable. (Some paranoid organizations have one machine
that can FTP. Other folks have to use sneakernet from a net-connected
machine.)
It's good to have a structure which supports automated fetching,
but please leave the dependency infrastructure visible so that
things work even when automated fetching doesn't.
-andy
Sent via Deja.com http://www.deja.com/
Before you buy.
From: Christopher R. Barry
Subject: Re: Packaging for open-source distribution
Date:
Message-ID: <87yacts2ts.fsf@2xtreme.net>
Andy Freeman <······@earthlink.net> writes:
> In article <···············@newsin1.ispchannel.com> "Kaelin Colclasure"
> <······@everest.com> wrote:
> > I'd like to generate a bit of discussion on how best to package Lisp
> > code for open source distribution.
>
> Great!
>
> > useful base of code in CL-HTTP for doing HTTP requests from Lisp.
> > We could probably standardize on some logical pathname host names
> > for defining "standard" locations to install packages.
>
> Now it's unusable, at least for me.
>
> For various social/technical reasons, I have to use Win9x for
> net access. However, I actually work in linux, without net access.
> It's a single computer, dual-boot, so I move things from the net
> to linux by downloading them using Win9x, rebooting to linux, and
> then moving them from the Win partition to a linux partition.
Why not d/l directly to Linux? Do you have one of the dreaded
winmodems? Or are you just unable to get PPP and your modem working
with Linux? (Not necessarily an easy task.) If the latter, I would
recommend getting the Debian distribution of Linux and using the
pppconfig utility.
Christopher
From: Andy Freeman
Subject: Re: Packaging for open-source distribution
Date:
Message-ID: <7uvb4j$gs4$1@nnrp1.deja.com>
In article <··············@2xtreme.net> ······@2xtreme.net (Christopher
R. Barry) wrote:
> Andy Freeman <······@earthlink.net> writes:
> > For various social/technical reasons, I have to use Win9x for
> > net access. However, I actually work in linux, without net access.
> Why not d/l directly to Linux? Do you have one of the dreaded
> winmodems? Or are you just unable to get PPP and your modem working
> with Linux? (Not necessarily an easy task.) If the latter, I would
> recommend getting the Debian distribution of Linux and using the
> pppconfig utility.
Thanks, but I'm aware of my alternatives. For my situation,
it's the right setup.
It isn't an uncommon setup, and there are lots of other
"not uncommon setups" have basically the same restriction -
there's no net access from a machine running CL. (I'd guess
that the most common is some sort of administered gateway,
where some IS human has to do any outside access. Pointing
out how silly that can be doesn't change it and asking the
CL pioneer in such organizations to take an arrow for an
unrelated cause is counterproductive.) There are probably
significant numbers of people who don't want to net-enable
their CL image.
There's nothing wrong with offering lots of services to
people who have "the right setup". However, you may want
to addict people who have other setups. Or not.
-andy
Sent via Deja.com http://www.deja.com/
Before you buy.
Andy Freeman <······@earthlink.net> wrote in message
·················@nnrp1.deja.com...
[...]
> There's nothing wrong with offering lots of services to
> people who have "the right setup". However, you may want
> to addict people who have other setups. Or not.
[...]
I see no reason why we shouldn't accomodate installations with
limited Net access. It would ultimately be desirable, after all,
to be able to work with a CD-ROM containing a snapshot of the
CLAN archive...
An installation that encountered a missing package would simply
prompt the user to provide a path name to that package -- or
to abort the install.
-- Kaelin
From: Christopher R. Barry
Subject: Re: Packaging for open-source distribution
Date:
Message-ID: <874sfhtrx8.fsf@2xtreme.net>
"Kaelin Colclasure" <······@everest.com> writes:
[...]
You have some great ideas. I can tell you've been spending time to
think this through good.
A few things:
* Hierarchical package system like Symbolics
A strong hierarchical module system is needed. This is important
because it allows people to write add-ons that do not collide with
one another. J13 discussions have mentioned layering the standard
and this would be useful for that as well. Additionally, they can
improve backwards compatibility for future standard revisions with
the current X3.226-1994 one because you can have a COMMON-LISP
package that is ANSI-1994 compatible and then one that is ANSI-200x
compatible. Hierarchical packages ala the Lisp machine I don't think
are possible to implement even remotely portably in ANSI CL. You
need a lot of additional functions to do operations with
hierarchical packages as well as redefinition/extension of many
existing ANSI CL functions like READ, INTERN, SYMBOL<*>, APROPOS,
DEFPACKAGE, FIND-PACKAGE, the LOOP for-as-package clause, PRINT,
PRINT-OBJECT methods, etc etc etc..... The Symbolics got it pretty
right, and the documentation on the package system is _huge_. I
remember when I first read it; it took me several hours and I was
lost. I could do some work on implementing this--assuming I can
figure out a realistic way to go about doing it portably enough so
that it will work with all the CLs that people use without taking
more than a few days of work to port to each one after completion
(this is probably impossible, oh well)--but it will be at least a
few months before I can seriously look into this because it will be
really hard and a _very_ high quality tutorial and documentation
will need to be written for it or people will never be able to make
sense of it, and there is a lot of other stuff I need to be doing
right now. At the very least I could do the specification and
tutorial I guess.
* "Definition retrieval"
This is a little weird, but bare with me. All of us use our own
personal macros and hacks that we find useful. For example, I like
making things to preserve the precious indentation resource:
(with-open-file (stream-1 file-1 ...)
(with-open-file (stream-2 file-2 ...)
(with-open-file (stream-3 file-3 ...)
<body>)))
(with-open-files ((stream-1 file-1 ...)
(stream-2 file-2 ...)
(stream-3 file-3 ...))
<body>)
If you use OPEN to do multiple files then you need an UNWIND-PROTECT
and a PROGN which is extremely costly indentation-wise, but I'm
nitpicking and digressing....
Anyways, I'm sure we also all use useful functions and macros that
are posted to c.l.l from time to time as well. Tim, Pierre and many
others have posted code I use. Erik Naggum twice posted a DROP macro
I'm fond of.
The point I'm getting to is that often when I'm writing code in an
Emacs buffer, I'll use stuff like GET-FILE-AS-STRING, or DROP, or
SPLIT, or TOKEN<blah>, or SERIES or ITERATE and in order to use this
non-standard stuff I either:
* Have to find it on my filesystem and copy the definition into the
current buffer. (For single functions.)
* Go to the hassle of making a defsystem form just for the one
dumb file I'm working with.
* Always count on it to be loaded into Lisp, which means that if I
ever try to distribute the file, I have to figure out which
non-standard functions or systems I used and copy them into the
file anyways, or distribute SERIES or ITERATE or whatever. It also
means that every time I start Lisp I need to load these things, or
dump an image with them in.
Many Lisps don't have all of their functionality loaded in their
image on startup, but will load things like a specific debugger or a
profiler the first time you try to use a function from it. I'd like
a standard way to do this, and have it integrated with this "Lisp
Archive Network" being discussed.
Additionally, you should be able to have stuff private to your site
integrated into your local archive as well just so you can take
advantage of its retrievel features and have a uniform interface. A
company that develops a Lisp-based product would probably have their
own module in their local archive that they keep all their private
tools in, like <company-name>:PRIVATE. For example, Franz and
Harlequin would possibly have something like FRANZ:PUBLIC and
FRANZ:PRIVATE packages, like FRANZ:PUBLIC:EXAMPLES:FTP or something.
I haven't thought this out too well yet (obviously), but you get the
idea, and this will perhaps give you fodder for your own thinking.
Christopher
From: Fernando D. Mato Mira
Subject: Re: Packaging for open-source distribution
Date:
Message-ID: <381459AA.2A828A9E@iname.com>
"Christopher R. Barry" wrote:
>
> * Hierarchical package system like Symbolics
This is something I already asked for over 5 years ago.
Being able to do foo:bar:baz would make some things much nicer.
"Fernando D. Mato Mira" <········@iname.com> writes:
> "Christopher R. Barry" wrote:
>
> >
> > * Hierarchical package system like Symbolics
>
> This is something I already asked for over 5 years ago.
> Being able to do foo:bar:baz would make some things much nicer.
Actually I became convinced of the same and started to do things like
foo.bar:baz
where I name packages like "FOO" and "FOO.BAR". For the sole purpose
of keeping things in order (with some manual work) the thing seems to
be working ok.
Note that you could in principle set up some macrology to achieve this
effect at DEFPACKAGE time. However, the 'presence' and 'import'
semantics in the package system make this rather cumbersome. (I don't
recall the exact issues, but I can reconstruct them if needed).
Cheers
--
Marco Antoniotti ===========================================
PARADES, Via San Pantaleo 66, I-00186 Rome, ITALY
tel. +39 - 06 68 10 03 17, fax. +39 - 06 68 80 79 26
http://www.parades.rm.cnr.it/~marcoxa
Marco Antoniotti <·······@copernico.parades.rm.cnr.it> wrote in message
···················@copernico.parades.rm.cnr.it...
[...]
> Actually I became convinced of the same and started to do things like
>
> foo.bar:baz
>
> where I name packages like "FOO" and "FOO.BAR". For the sole purpose
> of keeping things in order (with some manual work) the thing seems to
> be working ok.
A very pragmatic solution -- I like it! It neatly addresses the whole
"package namespace" issue without requiring a substitute for the package
system specified by the standard.
> Note that you could in principle set up some macrology to achieve this
> effect at DEFPACKAGE time. However, the 'presence' and 'import'
> semantics in the package system make this rather cumbersome. (I don't
> recall the exact issues, but I can reconstruct them if needed).
Would this really be necessary? If simply requiring manual conformance
to a naming convention can work us past this problem, that would be my
preferred solution. What would be the benefits of this more elaborate
scheme? How would this impact migrating non-conformant packages?
-- Kaelin
Having had an "opportunity" to look at this a bit, are hierarchical
package names really necessary here? Or do we really want to come up
with a scheme for hierarchical *system* names?
Again, I'll admit that I'm biased towards using MK:DEFSYSTEM as a
starting point for all this. What I find in my own code base is that
I'm relying on MK:DEFSYSTEM to resolve what code to load when -- not
the standard #'require mechanism.
Am I am outlier in this practice? Is it more common to rely on
#'require, or to use some system definition facility?
MK:DEFSYSTEM does have some code that implements #'require on top of
the system definitions -- but I confess I don't even know if it's
enabled in my lisp (ACL5).
-- Kaelin
"Kaelin Colclasure" <······@everest.com> writes:
> Having had an "opportunity" to look at this a bit, are hierarchical
> package names really necessary here? Or do we really want to come up
> with a scheme for hierarchical *system* names?
I believe that while "true" hierarchical packages would be good
(whatever that means), just being careful about names and directory
structure (like Java does), plus using USE-PACKAGE judiciously, cane
carry you a long way.
Having said so, I believe that some discussion about a 'naming'
structure for packages in CL would be good. Of course this should be
used with moderation (like wine and beer, unless you are drinking Bud
or Corona :) ). E.g. CMUCL "EXTENSION" package has 350 external
symbols which deal with at least four large categories (actually
more) of functionality: compiler, file system, garbage collection,
specialized data (e.g. weak pointers).
Having each of these categories have a package named
"EXTENSIONS.COMPILER", "EXTENSIONS.FILE-SYSTEM", "EXTENSIONS.GC", etc
(you get the idea) IMHO would make things more manageable.
> Again, I'll admit that I'm biased towards using MK:DEFSYSTEM as a
> starting point for all this. What I find in my own code base is that
> I'm relying on MK:DEFSYSTEM to resolve what code to load when -- not
> the standard #'require mechanism.
Let me argue that MK:DEFSYSTEM is "more standard" than the
standard. REQUIRE loads 'modules' in an *implementation dependent* way
(if you do not specify the extra pathname list). MK:DEFSYSTEM, being
a single implementation of a specific tool, gives you an
*implementation independent* way to specify where your modules (read
'systems') reside: MK:*CENTRAL-REGISTRY*.
> Am I am outlier in this practice? Is it more common to rely on
> #'require, or to use some system definition facility?
No, you are not. I use MK:DEFSYSTEM, I am reasonably happy with it
and I do not use REQUIRE/PROVIDE much.
> MK:DEFSYSTEM does have some code that implements #'require on top of
> the system definitions -- but I confess I don't even know if it's
> enabled in my lisp (ACL5).
This would be another way to go. If anybody had bug fixes and
improvements on MK:DEFSYSTEM, I would be very happy to collect them.
One that comes off my mind, is that in MK:DEFSYSTEM, you can specify a
'system' as dependent on another 'module', but the operation (load,
compile) does not propagate to the (sub)system itself. Any taker?
Cheers
--
Marco Antoniotti ===========================================
PARADES, Via San Pantaleo 66, I-00186 Rome, ITALY
tel. +39 - 06 68 10 03 17, fax. +39 - 06 68 80 79 26
http://www.parades.rm.cnr.it/~marcoxa
Marco Antoniotti <·······@parades.rm.cnr.it> wrote in message
···················@parades.rm.cnr.it...
[...]
> Let me argue that MK:DEFSYSTEM is "more standard" than the
> standard. REQUIRE loads 'modules' in an *implementation dependent* way
> (if you do not specify the extra pathname list). MK:DEFSYSTEM, being
> a single implementation of a specific tool, gives you an
> *implementation independent* way to specify where your modules (read
> 'systems') reside: MK:*CENTRAL-REGISTRY*.
[...]
I've been pursuing this track a bit. What I've come up with is pretty
straightforward -- mostly relying on functionality provided by
MK:DEFSYSTEM.
First, I modified the stock ACL5 hosts.cl file as follows:
---8<---
;;; Local Additions
"local" '(";**;*.*" #p"/usr/local/")
"lisp" '(";**;*.*" #p"local:;lisp;")
--->8---
In my filesystem, I created:
/usr/local/lisp/
/usr/local/lisp/systems/
My version of MK:DEFSYSTEM defaults the last entry in MK:*CENTRAL-REGISTRY*
to "/usr/local/lisp/systems/", so I didn't have to change anything there.
It also has an entry (by default) that looks like:
(MAKE:HOME-SUBDIRECTORY "lisp/systems/")
SO, now I can put a foo.system file in /usr/local/lisp/systems/ to make it
generally available, or put a copy in ~/lisp/systems/ to make it available
only in my user account (possibly shadowing the generally available one).
So far, so good.
Now the stickey bits. The idea of the "lisp" logical-pathname-host is of
course so that system definitions can be written to use logical pathnames
relative to (e.g. below) this host name. So when packaging code for
distribution, you can include a system definition like the following:
---8<---
;;; xml.expat.system
(in-package :cl-user)
(defsystem :xml.expat
:source-pathname "lisp:;xml;expat;"
:source-extension "lisp"
:components ("packages"
"expat")
:depends-on ())
--->8---
This pretty much takes care of setting up a package consumer's Lisp
environment to find the package's code -- and to load or compile said
code if :xml.expat appears in the :depends-on list of another system
definition.
But what about testing out the package before installing it? This
requires munging the :source-pathname in the above system definition
to point wherever the package currently happens to be lying in the
file system. One way to manage this is to have a script of some kind
that rewrites the system definition as it copies it into
~/lisp/systems/. Is there a simpler way? Perhaps using some of the more
esoteric logical pathname functionality?
And of course, this doesn't begin to address the issue of getting the
C code associated with this particular package compiled into shared
libraries and installed... I'm working on that as well.
-- Kaelin
I apologize if someone gets this more than once -- I posted it as a
follow-up a couple of days ago, but it never appeared on my news feed.
Marco Antoniotti <·······@copernico.parades.rm.cnr.it> wrote in message
···················@copernico.parades.rm.cnr.it...
[...]
> Actually I became convinced of the same and started to do things like
>
> foo.bar:baz
>
> where I name packages like "FOO" and "FOO.BAR". For the sole purpose
> of keeping things in order (with some manual work) the thing seems to
> be working ok.
A very pragmatic solution -- I like it! It neatly addresses the whole
"package namespace" issue without requiring a substitute for the package
system specified by the standard.
> Note that you could in principle set up some macrology to achieve this
> effect at DEFPACKAGE time. However, the 'presence' and 'import'
> semantics in the package system make this rather cumbersome. (I don't
> recall the exact issues, but I can reconstruct them if needed).
Would this really be necessary? If simply requiring manual conformance
to a naming convention can work us past this problem, that would be my
preferred solution. What would be the benefits of this more elaborate
scheme? How would this impact migrating non-conformant packages?
-- Kaelin
Kaelin Colclasure <······@everest.com> wrote in message
····················@newsin1.ispchannel.com...
> I'd like to generate a bit of discussion on how best to package Lisp
> code for open source distribution. Just to establish a frame of
> reference, what I ultimately would like to see come out of this is a
> system at *least* as functional as what CPAN provides for the perl
> community.
There's an interesting thread on SlashDot called "Open-Source Component
Repository?" Of course, they're talking about C components. [Apparently
they don't realize that most Linux programmers prefer Lisp. ;-)]
http://slashdot.org/askslashdot/99/11/03/2017204.shtml
One interesting point I saw raised was the need for a standardized
license for all the code in the repository (LGPL was suggested). Perl
has this -- all CPAN code is released under Perl's license. The
concern is that without this, a great deal of intellectual effort is
wasted trying to decide if you can use a particular bit of code from
the repository.
-- Kaelin