From: Dave Roberts
Subject: Right way to package dependencies?
Date: 
Message-ID: <06ngc.16922$0b4.23918@attbi_s51>
I have a library I'm working on (my Linux DNS resolver library). It has some
dependencies on UFFI. UFFI, in turn, is dependent on ASDF. In my
resolver.lisp file, I currently have some frontmatter goop that looks like
this:

(defpackage #:resolver 
  (:use "CL")
  (:export #:do-query #:lookup #:get-id #:get-qr #:get-opcode #:get-aa
           #:get-tc #:get-rd #:get-ra #:get-rcode #:get-qdcount #:get-ancount
           #:get-nscount #:get-arcount #:get-queries #:get-answers
           #:get-name-servers #:get-additional
           ;; various symbolic constants
           #:a #:ns #:md #:mf #:cname #:soa
           #:mb #:mg #:mr #:null #:wks #:ptr #:hinfo #:minfo #:mx #:txt #:axfr
           #:mailb #:maila #:all #:unknown #:in #:cs #:ch #:hs #:any #:query
           #:iquery #:status #:reserved #:no-error #:format-error
           #:server-failure #:name-error #:not-implemented #:refused)
  (:documentation "An library to interface with Linux's resolver.so
DNS library. See resolver(3) for more information. Uses UFFI to call
out to the alien routines."))

(in-package #:resolver)

(require 'asdf)
(push #P"/home/dave/lispstuff/uffi-1.4.6/" asdf:*central-registry*)
(asdf:operate 'asdf:load-op :uffi)
(require 'uffi)


If you were to get a library, would you expect it to resolve all its
dependencies when it loads, or would you expect to have instructions that
came along with it that basically say "Load UFFI first..." ?

If you would expect it to auto-load everything it requires, how would I deal
with the dependency on UFFI and the UFFI pathname, in particular?

If you would expect it to just have the basics, with the user setting
everything up, would I strip out all the UFFI and ASDF stuff and just have
a simple "(require 'uffi)" in there, leaving it to the user to figure out
how to get UFFI loaded, etc?

I'm a firm believer in making things simple for people. Why ship a set of
instructions when you can make it all just work, that's my motto.

-- 
Dave Roberts
·············@re-move.droberts.com
http://www.findinglisp.com/blog
From: Rahul Jain
Subject: Re: Right way to package dependencies?
Date: 
Message-ID: <87ekqlyguw.fsf@nyct.net>
Dave Roberts <·············@re-move.droberts.com> writes:

> I have a library I'm working on (my Linux DNS resolver library). It has some
> dependencies on UFFI. UFFI, in turn, is dependent on ASDF. In my
> resolver.lisp file, I currently have some frontmatter goop that looks like
> this:
>
> (defpackage #:resolver 
> [...])
>
> (in-package #:resolver)

Looks ok so far.

If you're going to do the next bit at all (which you shouldn't for
reasons I'll explain below), you should do it in a separate file. Well,
there's too much to explain in one go, so let's see what's going on one
step at a time.

> (require 'asdf)

The way REQUIRE works (or doesn't) is not specified by ANSI. First of
all, your code to define your library shouldn't depend on a specific way
of loading it. That's about as sensible as every library written in C
depending on a specific style of linker. Linking (and this is about as
close to a "linker" issue as you get with Lisp) should occur outside of
your library's code.

> (push #P"/home/dave/lispstuff/uffi-1.4.6/" asdf:*central-registry*)

Of course, that makes no sense to put in a distributed source file.

> (asdf:operate 'asdf:load-op :uffi)

Since this never gets executed at compile time, you have no guarantee
that your code will ever be compilable to someone who doesn't read this
code and realize that it needs to be loaded as source first, then
compiled.

> (require 'uffi)

If REQUIRE did anything on a system that was ASDF-biased, it would call
the same as what was above.

> If you were to get a library, would you expect it to resolve all its
> dependencies when it loads, or would you expect to have instructions that
> came along with it that basically say "Load UFFI first..." ?

I would expect it to contain an ASDF definition that would define the
contents, the internal dependencies of the files in your system, and the
external dependencies on other systems.

> If you would expect it to auto-load everything it requires, how would I deal
> with the dependency on UFFI and the UFFI pathname, in particular?

You wouldn't. That's the site-adimistrator's responsibility (and right). 
You would simply declare your system to be dependent on UFFI and ASDF
would take care of the rest, according to how the site administrator
configured it.

> I'm a firm believer in making things simple for people. Why ship a set of
> instructions when you can make it all just work, that's my motto.

No one who uses lisp seems to want such a system and lisp programmers
seem to firmly believe that the use of lisp code should be reserved for
lisp programmers or those who want to at least figure out the lisp
world-view. Shipping lisp packages that behave like the host OS's
packages (including supporting dynamic linking to upgrades of
dependencies) is a no-no.

However, here in my fantasy world, I would simply
clc-register-user-package resolver.asd and then load up lisp and
(asdf:oos 'asdf:load-op :resolver) once you wrote that asd file. This
would load UFFI and then (re)compile resolver.lisp if the fasl for the
implementation I'm on is outdated and then load that fasl.

Something about water and horses. Oh well, here comes the hate-fest.

-- 
Rahul Jain
·····@nyct.net
Professional Software Developer, Amateur Quantum Mechanicist
From: Dave Roberts
Subject: Re: Right way to package dependencies?
Date: 
Message-ID: <39wgc.162998$K91.416133@attbi_s02>
Rahul Jain wrote:
[a bunch of good stuff...]

Thanks. I understand the philosophy. I think ASDF will make things automatic
the way I want them. The one thing you said that was confusing is that
REQUIRE doesn't seem to be specified well in the spec. Why? I seem to
require it (no pun intended) on SBCL. For instance, while SBCL includes
ASDF, I absolutely have to use a (REQUIRE 'asdf) before I invoke any ASDF
command.

-- Dave 

-- 
Dave Roberts
·············@re-move.droberts.com
http://www.findinglisp.com/blog
From: Rahul Jain
Subject: Re: Right way to package dependencies?
Date: 
Message-ID: <87d665w9cl.fsf@nyct.net>
Dave Roberts <·············@re-move.droberts.com> writes:

> Rahul Jain wrote:
> [a bunch of good stuff...]
>
> Thanks. I understand the philosophy. I think ASDF will make things automatic
> the way I want them. The one thing you said that was confusing is that
> REQUIRE doesn't seem to be specified well in the spec. Why?

Because different site-admins what their setups to work differently. 
REQUIRE is really to be reserved for usage in a way that the site-admins
and users (those that are familiar with using lisp) agree upon.

> I seem to require it (no pun intended) on SBCL. For instance, while
> SBCL includes ASDF, I absolutely have to use a (REQUIRE 'asdf) before
> I invoke any ASDF command.

Yes, because you need to tell SBCL that it needs to load up ASDF. That's
not the way all installations work. Sometimes you'll have a core that
already has ASDF pre-loaded. Sometimes you'll have an implementation
that does something really strange with REQUIRE, because that's what
makes sense at that site.

-- 
Rahul Jain
·····@nyct.net
Professional Software Developer, Amateur Quantum Mechanicist
From: Edi Weitz
Subject: Re: Right way to package dependencies?
Date: 
Message-ID: <m3ad196tqc.fsf@bird.agharta.de>
On Sun, 18 Apr 2004 14:20:47 GMT, Dave Roberts <·············@re-move.droberts.com> wrote:

> I seem to require it (no pun intended) on SBCL. For instance, while
> SBCL includes ASDF, I absolutely have to use a (REQUIRE 'asdf)
> before I invoke any ASDF command.

But that's specific to SBCL because ASDF-INSTALL is distributed with
SBCL and SBCL has a documented way for user code to use the
REQUIRE/PROVIDE facilities. In other Lisps that's usually different.

  <http://www.weitz.de/asdf-install/#load-asdf>

Edi.
From: ·········@random-state.net
Subject: Re: Right way to package dependencies?
Date: 
Message-ID: <c5tl9r$7kree$1@midnight.cs.hut.fi>
What follows is a quick and dirty guide to asdf-install
packaging. If something is too unclear, hit me with the chineual.

First the carrot: if you do this people will be able to install
your library with just:

 (require :asdf-install)
 (asdf-install:install :resolver)

...and it will automatically take care of any dependencies,
installing UFFI as well if necessary.

Here's how:

> (defpackage #:resolver 
>   (:use "CL")
>   (:export #:do-query #:lookup #:get-id #:get-qr #:get-opcode #:get-aa
>            #:get-tc #:get-rd #:get-ra #:get-rcode #:get-qdcount #:get-ancount
>            #:get-nscount #:get-arcount #:get-queries #:get-answers
>            #:get-name-servers #:get-additional
>            ;; various symbolic constants
>            #:a #:ns #:md #:mf #:cname #:soa
>            #:mb #:mg #:mr #:null #:wks #:ptr #:hinfo #:minfo #:mx #:txt #:axfr
>            #:mailb #:maila #:all #:unknown #:in #:cs #:ch #:hs #:any #:query
>            #:iquery #:status #:reserved #:no-error #:format-error
>            #:server-failure #:name-error #:not-implemented #:refused)
>   (:documentation "An library to interface with Linux's resolver.so
> DNS library. See resolver(3) for more information. Uses UFFI to call
> out to the alien routines."))

Looks good so far. I'll assume that code up to this point is in
"packages.lisp". The next bit, however, you want to drop:

> (in-package #:resolver)
>
> (require 'asdf)
> (push #P"/home/dave/lispstuff/uffi-1.4.6/" asdf:*central-registry*)
> (asdf:operate 'asdf:load-op :uffi)
> (require 'uffi)

Nonono. Instead of this you want to use an ASDF defsystem:

In resolver.asd:

 (defsystem :resolver
   :version "0.1"
   :depends-on (:uffi) ;; external dependencies
   :components ((:file "packages")
                ;; the files you need and their internal
                ;; dependencies, note the missing .lisp
                ;; postfix -- with the default behaviour
                ;; of asdf you drop it
                (:file "foo" :depends-on ("packages"))
                (:file "bar" :depends-on ("packages"))
                (:file "quux" :depends-on ("foo" "bar")))

You can also add documentation files to the defsystem so that it
will be packaged as well.

Add a symlink to ./sbcl/systems to resolver.asd (well, any path in
your *central-repository*), and you should be able to just
(require :resolver).

Next step in your CL packaging career is to install gpg and
generate a public key, uploading it to the keyservers and eg. your
website. At the _very_ minimum install gpg and generate the key.

Then:

 ;; Get asdf-packaging-tools

 (require :asdf-install)
 (asdf-install:install :asdf-packaging-tools)
 (require :asdf-packaging-tool)

 ;; Use these in CL user, to avoid prefix hell. You may have
 ;; heard that indiscriminate USE-PACKAGE is a bad idea (and
 ;; it is!), but for interactive use in CL-USER it's definitely
 ;; the right thing.

 (use-package :asdf)
 (use-package :asdf-packaging-tools)
 
 ;; here starts the fun

 (oos 'release-op :resolver 
   :directory "/wherever/you/want/the/tarball/" ; trailing slash!
   :force t ; not strictly necessary, but may save some confusion
  )

Plenty of output, xterm pops up and asks your gpg passphrase, 
... and you should end up having:

 "/wherever/you/want/the/tarball/resolver-0.1.tar.gz"
 "/wherever/you/want/the/tarball/resolver-0.1.tar.gz.asc"
 "/wherever/you/want/the/tarball/resolver-latest.tar.gz"
 "/wherever/you/want/the/tarball/resolver-latest.tar.gz.asc"

Where .asc is a signature file and *-latest.* are symlinks to the
corresponding -0.1.* files.

You don't _need_ to use the release-op to do this, naturally,
but in the long run it's a fair deal more convenient then
generating the tarball by hand, or maintaining a separate script
to do so.

Then just upload that resolver-*.tar.gz* kaboodle to 

 http://wherever.com/whatever/

and add a Cliki page "Resolver", which includes a description
of the library, and the line:

 :(package "http://wherever.com/whatever/resolver-latest.tar.gz")

Presto: you can announce the library and people will be able
to install it with

 (require :asdf-intall)
 (asdf-install:install :resolver)

and it will pull down UFFI as well if it isn't yet installed.

Cheers,

 -- Nikodemus
From: Dave Roberts
Subject: Re: Right way to package dependencies?
Date: 
Message-ID: <LNtjc.49360$IW1.2889759@attbi_s52>
·········@random-state.net wrote:

> Then:
> 
>  ;; Get asdf-packaging-tools
> 
>  (require :asdf-install)
>  (asdf-install:install :asdf-packaging-tools)
>  (require :asdf-packaging-tool)

So this didn't work. The ASDF install fails for the packaging tools. Seems
like they aren't in the right place on the server or something (server
error 400).

-- 
Dave Roberts
·············@re-move.droberts.com
http://www.findinglisp.com/blog
From: ·········@random-state.net
Subject: Re: Right way to package dependencies?
Date: 
Message-ID: <c6ls3j$8fep6$1@midnight.cs.hut.fi>
Dave Roberts <·············@re-move.droberts.com> wrote:

>>  (require :asdf-install)
>>  (asdf-install:install :asdf-packaging-tools)
>>  (require :asdf-packaging-tool)

> So this didn't work. The ASDF install fails for the packaging tools. Seems
> like they aren't in the right place on the server or something (server
> error 400).

Seems that my redirects were screwed. Fixed.

Thanks,

  -- Nikodemus
comp.lang.lisp / 2004 / apr