I realize that it verges on ingratitude, but I wanted to take a moment
to urge ASDF-INSTALL packagers to add version numbers to their
tarballs. I have been working with James Bielman to try to get
asdf-upgrade to work, and with most of the tarballs on Cliki not having
version numbers in their filenames, it's virtually impossible (may be
outright impossible) to reliably tell whether there's a new version of
an ASDF-INSTALLed package available.
So, in the spirit of a supplicant, I'd like to encourage ASDF-INSTALL
packagers to give their tarballs names in the
<package-name>_<version-number> style. This will make it possible for
the community to provide a (ok, somewhat) reliably functioning upgrade
system for ASDF-INSTALLed packages.
Thanks, and good night!
RPG <·········@gmail.com> wrote:
> So, in the spirit of a supplicant, I'd like to encourage ASDF-INSTALL
> packagers to give their tarballs names in the
> <package-name>_<version-number> style. This will make it possible for
> the community to provide a (ok, somewhat) reliably functioning upgrade
> system for ASDF-INSTALLed packages.
Todo list for a new release:
- Make tar file, sign it.
- Upload tar file, update website with new text, changelog
- NEW: update CLiki page
- Inform the mailing lists
- Update record on Freshmeat
From: Edi Weitz
Subject: Re: A Humble Request: ASDF-INSTALL and version numbers
Date:
Message-ID: <uwtfynytc.fsf@agharta.de>
On 13 Feb 2006 20:50:22 -0800, "RPG" <·········@gmail.com> wrote:
> So, in the spirit of a supplicant, I'd like to encourage
> ASDF-INSTALL packagers to give their tarballs names in the
> <package-name>_<version-number> style. This will make it possible
> for the community to provide a (ok, somewhat) reliably functioning
> upgrade system for ASDF-INSTALLed packages.
Well, at least for my packages I can assure you that I won't do that
because it'd imply that I have to update the CLiki link every time I
release a new version - too much work. I think a new mechanism for
finding the ASDF-INSTALL tarball should be developed first.
--
European Common Lisp Meeting 2006: <http://weitz.de/eclm2006/>
Real email: (replace (subseq ·········@agharta.de" 5) "edi")
Edi Weitz wrote:
> On 13 Feb 2006 20:50:22 -0800, "RPG" <·········@gmail.com> wrote:
>
> Well, at least for my packages I can assure you that I won't do that
> because it'd imply that I have to update the CLiki link every time I
> release a new version - too much work. I think a new mechanism for
> finding the ASDF-INSTALL tarball should be developed first.
Something like the Cygwin installer would be great.
-Luke
Thanks, Edi. My post grew out of a discussion with JB, and we were
both wondering why packages overwhelmingly were NOT tagged with version
numbers. I've only done one or two, so it was no big deal for me.
Clearly, one answer is "it's too much work."
Question: would it be best to try to fix this with tool support (seems
unlikely, although I could be wrong), or to introduce some kind of
URL-based indirection that would make the right thing happen without
effort?
From: Edi Weitz
Subject: Re: A Humble Request: ASDF-INSTALL and version numbers
Date:
Message-ID: <ubqx8ztix.fsf@agharta.de>
On 14 Feb 2006 12:34:37 -0800, "RPG" <·········@gmail.com> wrote:
> Thanks, Edi. My post grew out of a discussion with JB, and we were
> both wondering why packages overwhelmingly were NOT tagged with
> version numbers. I've only done one or two, so it was no big deal
> for me.
>
> Clearly, one answer is "it's too much work."
>
> Question: would it be best to try to fix this with tool support
> (seems unlikely, although I could be wrong), or to introduce some
> kind of URL-based indirection that would make the right thing happen
> without effort?
I don't know. I'm mainly interested in a solution that doesn't
increase my workload and the number of things to check. Maintaining a
couple of open source libs, writing documentation, and answering user
questions is enough for now... :)
--
European Common Lisp Meeting 2006: <http://weitz.de/eclm2006/>
Real email: (replace (subseq ·········@agharta.de" 5) "edi")
Edi Weitz wrote:
> On 14 Feb 2006 12:34:37 -0800, "RPG" <·········@gmail.com> wrote:
>
> > Thanks, Edi. My post grew out of a discussion with JB, and we were
> > both wondering why packages overwhelmingly were NOT tagged with
> > version numbers. I've only done one or two, so it was no big deal
> > for me.
> >
> > Clearly, one answer is "it's too much work."
> >
> > Question: would it be best to try to fix this with tool support
> > (seems unlikely, although I could be wrong), or to introduce some
> > kind of URL-based indirection that would make the right thing happen
> > without effort?
>
> I don't know. I'm mainly interested in a solution that doesn't
> increase my workload and the number of things to check. Maintaining a
> couple of open source libs, writing documentation, and answering user
> questions is enough for now... :)
Would something like the following meet your needs (this is just a
first sketch of a solution)?
1. When issuing a version, one bumps the :version argument in the asd
defsystem;
2. We add an :asdf-install-url argument to the asd defsystem
3. One signs the asd file.
4. One uploads the tarball, asd file, and signatures wherever one
wishes, with whatever name one wishes.
5. One puts a link to the asd file to Cliki (possibly a link to the
tarball, too, for backward compatibility -- see below). Note that this
step, as with current "-latest" practice, only needs to be performed
once, yet still permit automatic detection of upgrades. I.e., only
steps 1, 3 and 4 need be carried out to issue a patched version.
Now asdf-install would pull the asd file from a well-known location,
examine it, and pull the corresponding tarball.
My hope is that this would be (a) easy to code; (b)
backward-compatible with existing practice (asdf-install-prime could
pull asd files, asdf-install-current would continue to pull tarballs);
(c) support automating the upgrade process.
Some open questions:
1. What about those packages that don't have single asdf systems, but
multiple ones? Perhaps the best answer would be to require
asdf-installable systems to have one toplevel system whose name is the
same as the package name.... Upgrading might still be complicated if
it's possible to bump a sub-system version without bumping the topmost
system version ("don't do that then" might be an acceptable answer to
this).
2. Can we do something with signatures to tie the tarball and the asd
file together so that we will detect the case where they stray? This
is possibly a weakness of using the asd file as external header
information. If we had a fully-external header file, with just url,
and version info, then we could put a tarball signature into the header
file. It should be trivial to write a header-file generator (that
would also generate tarball and signatures).
3. Having the URL in the ASD file would make mirroring annoying.
Another possible argument for having an external package header file.
4. The argument for using the ASD file, as opposed to an external
header file, seems to boil down to it requiring no additional work
over current practice. Should the header file generation be fully
automated, perhaps this argument dissolves.
From: Edi Weitz
Subject: Re: A Humble Request: ASDF-INSTALL and version numbers
Date:
Message-ID: <upslok292.fsf@agharta.de>
On 15 Feb 2006 08:28:57 -0800, "RPG" <·········@gmail.com> wrote:
> Would something like the following meet your needs (this is just a
> first sketch of a solution)?
Sounds OK to me.
--
European Common Lisp Meeting 2006: <http://weitz.de/eclm2006/>
Real email: (replace (subseq ·········@agharta.de" 5) "edi")
RPG <·········@gmail.com> wrote:
+---------------
| Would something like the following meet your needs (this is just a
| first sketch of a solution)?
|
| 1. When issuing a version, one bumps the :version argument in the asd
| defsystem;
| 2. We add an :asdf-install-url argument to the asd defsystem
| 3. One signs the asd file.
| 4. One uploads the tarball, asd file, and signatures wherever one
| wishes, with whatever name one wishes.
| 5. One puts a link to the asd file to Cliki (possibly a link to the
| tarball, too, for backward compatibility -- see below). Note that this
| step, as with current "-latest" practice, only needs to be performed
| once, yet still permit automatic detection of upgrades. I.e., only
| steps 1, 3 and 4 need be carried out to issue a patched version.
|
| Now asdf-install would pull the asd file from a well-known location,
| examine it, and pull the corresponding tarball.
+---------------
Hmmm... This is starting to sound very much like the way the FreeBSD
"ports" system works:
http://www.freebsd.org/ports/index.html
http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/ports.html
except that "ports" uses a stylized "Makefile" and other auxiliary
files ("distinfo", "pkg-descr", & "pkg-plist", at a minimum) instead
of ASDF-Install. The ports system includes specifying canonical and
alternative archives for a port, as well as required prerequisites
and their versions.
The way it's usually used is to install the entire collection
of "ports", which is just the information files [for all 11000+
ports, currently ~70 MB!!], and then when you want to install
a specific port, you "cd" to the corresponding directory (e.g.,
"/usr/ports/lang/cmucl" for CMUCL) and say "make intall". But
you can also take a more a la carte approach.
The following lists the FreeBSD ports matching "lisp":
http://www.freebsd.org/cgi/ports.cgi?query=lisp&stype=all
Here are some snippets from the Makefile for the CMUCL port (loads "19b")
which show how the version/archive-site/maintainer/etc. is handled:
http://www.freebsd.org/cgi/cvsweb.cgi/~checkout~/ports/lang/cmucl/Makefile?rev=1.25
...[snip]...
PORTNAME= cmucl
PORTVERSION= 19b
CATEGORIES= lang lisp
MASTER_SITES= ftp://ftp.common-lisp.net/pub/project/cmucl/release/${PORTVERSION}/ \
http://www.pmsf.de/pub/cmucl/release/${PORTVERSION}/ \
ftp://ftp.averillpark.net/cmucl/release/${PORTVERSION}/ \
ftp://ftp.linux.org.uk/pub/lisp/cmucl/release/${PORTVERSION}/ \
ftp://ftp.tepus.com/pub/project/cmucl/release/${PORTVERSION}/
DISTNAME= ${PORTNAME}-${PORTVERSION}-x86-FreeBSD
MAINTAINER= ········@cons.org
COMMENT= The CMU implementation of Common Lisp
ONLY_FOR_ARCHS= i386
USE_BZIP2= YES
NO_WRKSUBDIR= yes
NO_BUILD= yes
MAN1= lisp.1 cmucl.1
...[snip]...
whereas the "distinfo" file contains the actual filename for the
distribution, length, and checksums (MD5 & SHA-256):
MD5 (cmucl-19b-x86-FreeBSD.tar.bz2) = 9536c4f52b59041e90ed676dcab85cdf
SIZE (cmucl-19b-x86-FreeBSD.tar.bz2) = 7283215
SHA256 (cmucl-19b-x86-FreeBSD.tar.bz2) = 2683234da65a1a295429eacc88af011baf621333a556b8413063117de7a4e41d
I'm not suggesting trying to copy "ports" exactly, but it *does*
represent a system which has been shown capable of scaling to over
10.000 packages, and seems to address many of the concerns I have
heard in this thread. At the very least, it would be good to use
it as a checklist of features needed in a Common Lisp Repository.
-Rob
-----
Rob Warnock <····@rpw3.org>
627 26th Avenue <URL:http://rpw3.org/>
San Mateo, CA 94403 (650)572-2607
Rob Warnock wrote:
> RPG <·········@gmail.com> wrote:
> +---------------
> | Would something like the following meet your needs (this is just a
> | first sketch of a solution)?
> |
> | 1. When issuing a version, one bumps the :version argument in the asd
> | defsystem;
> | 2. We add an :asdf-install-url argument to the asd defsystem
> | 3. One signs the asd file.
> | 4. One uploads the tarball, asd file, and signatures wherever one
> | wishes, with whatever name one wishes.
> | 5. One puts a link to the asd file to Cliki (possibly a link to the
> | tarball, too, for backward compatibility -- see below). Note that this
> | step, as with current "-latest" practice, only needs to be performed
> | once, yet still permit automatic detection of upgrades. I.e., only
> | steps 1, 3 and 4 need be carried out to issue a patched version.
> |
> | Now asdf-install would pull the asd file from a well-known location,
> | examine it, and pull the corresponding tarball.
> +---------------
>
> Hmmm... This is starting to sound very much like the way the FreeBSD
> "ports" system works:
>
> http://www.freebsd.org/ports/index.html
> http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/ports.html
>
> except that "ports" uses a stylized "Makefile" and other auxiliary
> files ("distinfo", "pkg-descr", & "pkg-plist", at a minimum) instead
> of ASDF-Install. The ports system includes specifying canonical and
> alternative archives for a port, as well as required prerequisites
> and their versions.
>
...snip...
> I'm not suggesting trying to copy "ports" exactly, but it *does*
> represent a system which has been shown capable of scaling to over
> 10.000 packages, and seems to address many of the concerns I have
> heard in this thread. At the very least, it would be good to use
> it as a checklist of features needed in a Common Lisp Repository.
>
Agreed. Ports and portage may also be a better source of inspiration
than the RPM-updaters, because the former include distributing source
code, whereas the latter is primarily (though not exclusively) aimed at
distributing binaries.
I think some lisp-friendly add in like the stylized Makefile will
eventually be desirable, in order to limit the amount of effort on the
part of library packagers. I.e., a simple function that can build the
tarball, build the header file, sign everything and (why not?) do the
upload.
RPG schrieb:
> Would something like the following meet your needs (this is just a
> first sketch of a solution)?
>
> 1. When issuing a version, one bumps the :version argument in the asd
> defsystem;
> 2. We add an :asdf-install-url argument to the asd defsystem
> 3. One signs the asd file.
> 4. One uploads the tarball, asd file, and signatures wherever one
> wishes, with whatever name one wishes.
> 5. One puts a link to the asd file to Cliki (possibly a link to the
> tarball, too, for backward compatibility -- see below). Note that this
> step, as with current "-latest" practice, only needs to be performed
> once, yet still permit automatic detection of upgrades. I.e., only
> steps 1, 3 and 4 need be carried out to issue a patched version.
>
> Now asdf-install would pull the asd file from a well-known location,
> examine it, and pull the corresponding tarball.
I like this idea. Perhaps it would be even better if we do not rely on
the "asdf header file" to be the included .asd file. An .asd file is
generally a normal lisp source code file which is a bit to much for the
purpose of a header file. How about adding a new ASDF operation which
can generate a header-file out of the ASDF file? This header file would
just contain a property list with the needed information.
Such an header file would be:
1) smaller than the .asd file
2) Does not contain arbitrary Lisp code (you need just READ)
3) There is not problem which .asd file should get exported
Proposal
(:asdf-header-version "0.1"
:system-name "foo"
:system-dependencies '(cl-ppcre mel-base araneida)
:system-version "0.1.12"
:locations '("http://somewhere.bla/foo.tgz"
"http://somewhere.else/foo.tgz")
...)
ciao,
Jochen
Jochen Schmidt wrote:
> Proposal
>
> (:asdf-header-version "0.1"
> :system-name "foo"
> :system-dependencies '(cl-ppcre mel-base araneida)
> :system-version "0.1.12"
> :locations '("http://somewhere.bla/foo.tgz"
> "http://somewhere.else/foo.tgz")
> ...)
As an add on: How about prepending this plist to the signature file
that already accompanies an ASDF install package? AFAIK the signature
is enclosed within "-----BEGIN PGP SIGNATURE-----" and "-----BEGIN PGP
SIGNATURE-----" and gpg should be able to extract it even if we prepend
a plist like above. The net win would be that one still has only two
files to handle - the header+signature and the package itself.
ciao,
Jochen
··@codeartist.org wrote:
> Jochen Schmidt wrote:
> > Proposal
> >
> > (:asdf-header-version "0.1"
> > :system-name "foo"
> > :system-dependencies '(cl-ppcre mel-base araneida)
> > :system-version "0.1.12"
> > :locations '("http://somewhere.bla/foo.tgz"
> > "http://somewhere.else/foo.tgz")
> > ...)
>
> As an add on: How about prepending this plist to the signature file
> that already accompanies an ASDF install package? AFAIK the signature
> is enclosed within "-----BEGIN PGP SIGNATURE-----" and "-----BEGIN PGP
> SIGNATURE-----" and gpg should be able to extract it even if we prepend
> a plist like above. The net win would be that one still has only two
> files to handle - the header+signature and the package itself.
The counter-argument to this is that we may (I think we should) want
the asdf-header to have its own signature.
··@codeartist.org wrote:
> RPG schrieb:
>
> > Would something like the following meet your needs (this is just a
> > first sketch of a solution)?
> >
> > 1. When issuing a version, one bumps the :version argument in the asd
> > defsystem;
> > 2. We add an :asdf-install-url argument to the asd defsystem
> > 3. One signs the asd file.
> > 4. One uploads the tarball, asd file, and signatures wherever one
> > wishes, with whatever name one wishes.
> > 5. One puts a link to the asd file to Cliki (possibly a link to the
> > tarball, too, for backward compatibility -- see below). Note that this
> > step, as with current "-latest" practice, only needs to be performed
> > once, yet still permit automatic detection of upgrades. I.e., only
> > steps 1, 3 and 4 need be carried out to issue a patched version.
> >
> > Now asdf-install would pull the asd file from a well-known location,
> > examine it, and pull the corresponding tarball.
>
> I like this idea. Perhaps it would be even better if we do not rely on
> the "asdf header file" to be the included .asd file. An .asd file is
> generally a normal lisp source code file which is a bit to much for the
> purpose of a header file. How about adding a new ASDF operation which
> can generate a header-file out of the ASDF file? This header file would
> just contain a property list with the needed information.
> Such an header file would be:
>
> 1) smaller than the .asd file
> 2) Does not contain arbitrary Lisp code (you need just READ)
> 3) There is not problem which .asd file should get exported
>
> Proposal
>
> (:asdf-header-version "0.1"
> :system-name "foo"
> :system-dependencies '(cl-ppcre mel-base araneida)
> :system-version "0.1.12"
> :locations '("http://somewhere.bla/foo.tgz"
> "http://somewhere.else/foo.tgz")
> ...)
I mostly agree, and I certainly think the separate header file is the
way to go.
But this is a case where I would NOT want to take advantage of Lisp's
nice programs as data (and not just use READ). This is going to be a
place where we should take at least some care about security, so we
don't have to have s-expressions, unless those will make our editors or
generators happy. We'll want to be reading strings, so something that
looks like a Unix config file (or even XML, if people like that) might
be better. We don't want too much code that's running as
root/administrator to be calling READ!
"RPG" <·········@gmail.com> writes:
> I mostly agree, and I certainly think the separate header file is the
> way to go.
>
> But this is a case where I would NOT want to take advantage of Lisp's
> nice programs as data (and not just use READ). This is going to be a
> place where we should take at least some care about security, so we
> don't have to have s-expressions, unless those will make our editors or
> generators happy. [...]
Yes we want s-expressions, nothing less will do.
The security problem is readily solved with (setf *read-eval* nil).
If you are really paranoiac, you can use your own simplified s-expr reader.
--
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
A: Top-posting.
Q: What is the most annoying thing on usenet and in e-mail?
__Pascal Bourguignon__ http://www.informatimago.com/
Pascal Bourguignon wrote:
> "RPG" <·········@gmail.com> writes:
> > I mostly agree, and I certainly think the separate header file is the
> > way to go.
> >
> > But this is a case where I would NOT want to take advantage of Lisp's
> > nice programs as data (and not just use READ). This is going to be a
> > place where we should take at least some care about security, so we
> > don't have to have s-expressions, unless those will make our editors or
> > generators happy. [...]
>
> Yes we want s-expressions, nothing less will do.
> The security problem is readily solved with (setf *read-eval* nil).
> If you are really paranoiac, you can use your own simplified s-expr reader.
It's OK with me one way or the other. The values will all be primitive
anyway...
Edi Weitz <········@agharta.de> writes:
> Well, at least for my packages I can assure you that I won't do that
> because it'd imply that I have to update the CLiki link every time I
> release a new version - too much work. I think a new mechanism for
> finding the ASDF-INSTALL tarball should be developed first.
What about using the :VERSION argument of a system definition instead?
Paolo
--
Why Lisp? http://wiki.alu.org/RtL%20Highlight%20Film
The Common Lisp Directory: http://www.cl-user.net
On 2006-02-14 08:40:39 -0500, Paolo Amoroso <·······@mclink.it> said:
> Edi Weitz <········@agharta.de> writes:
>
>> Well, at least for my packages I can assure you that I won't do that
>> because it'd imply that I have to update the CLiki link every time I
>> release a new version - too much work. I think a new mechanism for
>> finding the ASDF-INSTALL tarball should be developed first.
>
> What about using the :VERSION argument of a system definition instead?
FWIW, I can make this part of asdf-install-tester when I get back to
working on that (soon I hope!)
--
Gary Warren King
metabang.com
http://www.metabang.com/
The problem with using the :version argument is that the .ASD file is
buried inside the tarball, and relying on it would mean that one would
have to pull the tarball to see if one needed to pull the tarball!
A second issue, that has been bruited about in a different context, is
that sometimes there is a a one-to-many map from ASDF-INSTALL packages
to ASDF system definitions. E.g., in the case where there are multiple
back-ends to a function, each implemented by a separate ASDF-system.
It seems to me that one solution might be to provide a way to "tear
off" the ASDF system definition from the package, and store it "off to
the side" of the tarball where it could be more easily consulted. This
is effectively what Mandriva's URPMI scheme does for RPMs, and my guess
is that the other Linux package management schemes all do something
similar.
A scheme like that would also answer Edi's objection, because it could
make it easier to just push a new ASDF-INSTALL tarball out there,
rather than requiring a lot of URL engineering. There could be a link
on CLIKI that would point to the (GPG signed, I would imagine) header
information, and the header information would, in turn, contain a
pointer (URL) to the tarball.
One would always upload the header information for a new package
version to the same URL, so there would be no need to constantly fuss
with CLIKI pages.
The downside of this proposed solution, of course, would be that it
would radically upset the current ASDF-INSTALL scheme.
From: Edi Weitz
Subject: Re: A Humble Request: ASDF-INSTALL and version numbers
Date:
Message-ID: <uwtfy9gko.fsf@agharta.de>
On Tue, 14 Feb 2006 14:40:39 +0100, Paolo Amoroso <·······@mclink.it> wrote:
> Edi Weitz <········@agharta.de> writes:
>
>> Well, at least for my packages I can assure you that I won't do
>> that because it'd imply that I have to update the CLiki link every
>> time I release a new version - too much work. I think a new
>> mechanism for finding the ASDF-INSTALL tarball should be developed
>> first.
>
> What about using the :VERSION argument of a system definition
> instead?
I'm already doing that for most of my libs, but that's not what the OP
wanted to have.
--
European Common Lisp Meeting 2006: <http://weitz.de/eclm2006/>
Real email: (replace (subseq ·········@agharta.de" 5) "edi")
On 2006-02-14, Edi Weitz <········@agharta.de> wrote:
>> So, in the spirit of a supplicant, I'd like to encourage
>> ASDF-INSTALL packagers to give their tarballs names in the
>> <package-name>_<version-number> style. This will make it possible
>> [...]
>
> Well, at least for my packages I can assure you that I won't do that
> because it'd imply that I have to update the CLiki link every time I
> release a new version - too much work. I think a new mechanism for
> finding the ASDF-INSTALL tarball should be developed first.
Hi Edi,
I happen to make tarballs in the <name>-<version> format since I need
them that way for Debian uploads anyway. I use a script on my public
server that moves old tarball versions to an Archive directory and then links
the latest version as <name>-latest.tar.gz. Then, on cliki I just have
a link to that symlink. Thus, the cliki link never has to change.
Best,
Kevin
Kevin Rosenberg wrote:
> On 2006-02-14, Edi Weitz <········@agharta.de> wrote:
> >> So, in the spirit of a supplicant, I'd like to encourage
> >> ASDF-INSTALL packagers to give their tarballs names in the
> >> <package-name>_<version-number> style. This will make it possible
> >> [...]
> >
> > Well, at least for my packages I can assure you that I won't do that
> > because it'd imply that I have to update the CLiki link every time I
> > release a new version - too much work. I think a new mechanism for
> > finding the ASDF-INSTALL tarball should be developed first.
>
> Hi Edi,
>
> I happen to make tarballs in the <name>-<version> format since I need
> them that way for Debian uploads anyway. I use a script on my public
> server that moves old tarball versions to an Archive directory and then links
> the latest version as <name>-latest.tar.gz. Then, on cliki I just have
> a link to that symlink. Thus, the cliki link never has to change.
>
Kevin,
The problem with the UFFI solution is that the URL that one fetches
still doesn't have a version number in it. So at least the current
ASDF-UPGRADE approach can't use the URL to detect whether a tarball
fetch is needed in advance.
A protocol where the latest link was expected to simply provide
referral to a true URL would meet that need and Edi's.
However, if we were to go to the trouble of instituting an indirection
protocol at all, it's my suggestion that we might as well go the full
distance and provide a detached header. The detatched header might be
just an ASD file with some additional information in the top-level ASD
form, or it might be a more specialized form.
Ideally this would involve shifting some of the effort from the
packager onto some yet-to-be-named infrastructure.
Best,
R
From: Edi Weitz
Subject: Re: A Humble Request: ASDF-INSTALL and version numbers
Date:
Message-ID: <ubqx9amz5.fsf@agharta.de>
On Tue, 14 Feb 2006 16:45:18 +0000 (UTC), Kevin Rosenberg <·····@rosenberg.net> wrote:
> I happen to make tarballs in the <name>-<version> format since I
> need them that way for Debian uploads anyway. I use a script on my
> public server that moves old tarball versions to an Archive
> directory and then links the latest version as
> <name>-latest.tar.gz. Then, on cliki I just have a link to that
> symlink. Thus, the cliki link never has to change.
Thanks Kevin,
that's certainly better than doing it manually. But still I'm kind of
against the idea that all the work should be done by the library
authors... :)
Cheers,
Edi.
--
European Common Lisp Meeting 2006: <http://weitz.de/eclm2006/>
Real email: (replace (subseq ·········@agharta.de" 5) "edi")
On 2006-02-14, Edi Weitz <········@agharta.de> wrote:
> that's certainly better than doing it manually. But still I'm kind of
> against the idea that all the work should be done by the library
> authors... :)
As a library author, that's an easy sentiment to share ;-)
--
Kevin Rosenberg
·····@hypershots.com
Edi Weitz wrote:
> On 13 Feb 2006 20:50:22 -0800, "RPG" <·········@gmail.com> wrote:
>
> > So, in the spirit of a supplicant, I'd like to encourage
> > ASDF-INSTALL packagers to give their tarballs names in the
> > <package-name>_<version-number> style. This will make it possible
> > for the community to provide a (ok, somewhat) reliably functioning
> > upgrade system for ASDF-INSTALLed packages.
>
> Well, at least for my packages I can assure you that I won't do that
> because it'd imply that I have to update the CLiki link every time I
> release a new version - too much work. I think a new mechanism for
> finding the ASDF-INSTALL tarball should be developed first.
Is there any reason Cliki cannot automatically crawl locations "known
to contain CL libraries" and automatically update on release of a new
version? And provide a suitable feed for "subscribers" to a CL library
service?
There's no reason I can see in principle why a crawler could not locate
and parse .asd files within a .tar.gz file to retrieve the new version,
and also, as an added feature, check the package dependency list for
any change.
Proliferating headers and manually-maintained meta-data seems like just
another opportunity for things to get out of sync.
··············@hotmail.com wrote:
> Edi Weitz wrote:
> > On 13 Feb 2006 20:50:22 -0800, "RPG" <·········@gmail.com> wrote:
> >
> > > So, in the spirit of a supplicant, I'd like to encourage
> > > ASDF-INSTALL packagers to give their tarballs names in the
> > > <package-name>_<version-number> style. This will make it possible
> > > for the community to provide a (ok, somewhat) reliably functioning
> > > upgrade system for ASDF-INSTALLed packages.
> >
> > Well, at least for my packages I can assure you that I won't do that
> > because it'd imply that I have to update the CLiki link every time I
> > release a new version - too much work. I think a new mechanism for
> > finding the ASDF-INSTALL tarball should be developed first.
>
> Is there any reason Cliki cannot automatically crawl locations "known
> to contain CL libraries" and automatically update on release of a new
> version? And provide a suitable feed for "subscribers" to a CL library
> service?
>
> There's no reason I can see in principle why a crawler could not locate
> and parse .asd files within a .tar.gz file to retrieve the new version,
> and also, as an added feature, check the package dependency list for
> any change.
We've actually seen a couple of arguments by other posters for the use
of external header files. Also, I don't think asking the Cliki folks
to do this is necessarily a good strategy (what if they say no?).
Finally, what led into my request was that it was very hard to detect
the need for an update without fetching full tarballs, and even then
(if the tarballs are all called <foo>_latest.tgz), it might require
arbitrary loading and introspection on the asdf systems.
And we don't want Cliki to have to load all the asdf systems that
someone puts on it, because that's an easy way for unscrupulous people
to turn cliki into an unwilling compute (virus or spam) server! The
ASD files have arbitrary CL code in them, so they could do arbitrary
damage to the cliki server, and anyone who trusts it...
>
> Proliferating headers and manually-maintained meta-data seems like just
> another opportunity for things to get out of sync.
I think that's the argument for being able to automate the packaging
process (i.e., *less* *manually-maintained* meta-data) more than we do.
It's already too onerous for people like Edi and Kevin to do
versioning with the existing system. We need to make it easier not
harder.
Best,
r
RPG wrote:
> ··············@hotmail.com wrote:
> > Edi Weitz wrote:
> > > On 13 Feb 2006 20:50:22 -0800, "RPG" <·········@gmail.com> wrote:
> > >
> > > > So, in the spirit of a supplicant, I'd like to encourage
> > > > ASDF-INSTALL packagers to give their tarballs names in the
> > > > <package-name>_<version-number> style. This will make it possible
> > > > for the community to provide a (ok, somewhat) reliably functioning
> > > > upgrade system for ASDF-INSTALLed packages.
> > >
> > > Well, at least for my packages I can assure you that I won't do that
> > > because it'd imply that I have to update the CLiki link every time I
> > > release a new version - too much work. I think a new mechanism for
> > > finding the ASDF-INSTALL tarball should be developed first.
> >
> > Is there any reason Cliki cannot automatically crawl locations "known
> > to contain CL libraries" and automatically update on release of a new
> > version? And provide a suitable feed for "subscribers" to a CL library
> > service?
> >
> > There's no reason I can see in principle why a crawler could not locate
> > and parse .asd files within a .tar.gz file to retrieve the new version,
> > and also, as an added feature, check the package dependency list for
> > any change.
>
> We've actually seen a couple of arguments by other posters for the use
> of external header files. Also, I don't think asking the Cliki folks
> to do this is necessarily a good strategy (what if they say no?).
> Finally, what led into my request was that it was very hard to detect
> the need for an update without fetching full tarballs, and even then
> (if the tarballs are all called <foo>_latest.tgz), it might require
> arbitrary loading and introspection on the asdf systems.
"Cliki" was just a shorthand for "central authority agreeing to support
this computing work, so that we don't count on all developers to do the
same thing manually."
As for the separate header issue, I see no possible way to enforce
agreement between header and state-of-the-ASDF-system other than
requiring the developer to do it manually, the burden of which is just
what causes the problem in the first place.
> And we don't want Cliki to have to load all the asdf systems that
> someone puts on it, because that's an easy way for unscrupulous people
> to turn cliki into an unwilling compute (virus or spam) server! The
> ASD files have arbitrary CL code in them, so they could do arbitrary
> damage to the cliki server, and anyone who trusts it...
> >
> > Proliferating headers and manually-maintained meta-data seems like just
> > another opportunity for things to get out of sync.
>
> I think that's the argument for being able to automate the packaging
> process (i.e., *less* *manually-maintained* meta-data) more than we do.
> It's already too onerous for people like Edi and Kevin to do
> versioning with the existing system. We need to make it easier not
> harder.
"Easier" to me means computer automated, and as we are Lisp
programmers, I think would likely involve Lisp, and since we have a
Lisp-compatible format that system developers must keep up-to-date for
describing Lisp systems, it seems most natural to use that form.
Pascal Bourguignon addressed the security issue in his post; we have
ways to sterilize S-expression input, looking for the magic :version
"blah" keyword, and dependencies, which then probably gets stored in a
central database to support cheap queries, associated with the
modification date & checksum of the .tar.gz file.
As an aside, this whole discussion leads me to wonder about the patch
mechanism supported by Genera (which I have no concept of, beyond the
apparent ability I've seen in screenshots) to distribute patches and
versions of multiple libraries, and whether existing Lisp systems are
lacking the power to do what they did, and if creating that ability
within ASDF or ASDF-INSTALL is really what we are looking for.
··············@hotmail.com wrote:
> RPG wrote:
> > ··············@hotmail.com wrote:
> > > Edi Weitz wrote:
> > > > On 13 Feb 2006 20:50:22 -0800, "RPG" <·········@gmail.com> wrote:
> > > Is there any reason Cliki cannot automatically crawl locations "known
> > > to contain CL libraries" and automatically update on release of a new
> > > version? And provide a suitable feed for "subscribers" to a CL library
> > > service?
> > >
> > > There's no reason I can see in principle why a crawler could not locate
> > > and parse .asd files within a .tar.gz file to retrieve the new version,
> > > and also, as an added feature, check the package dependency list for
> > > any change.
>
> "Cliki" was just a shorthand for "central authority agreeing to support
> this computing work, so that we don't count on all developers to do the
> same thing manually."
The problem is that there does not exist any central authority agreeing
to support this computing work, so that the burden of looking for
updates falls on the clients (those who want to install updates). This
is really an empirical question, and not something subject to
argument. I am going to proceed on the assumption that this claim is
true. If someone else wants to proceed on the assumption that such a
central authority can be found, is interested in finding it, and
specifying an appropriate protocol, I am supportive, and will be happy
to get out of the way.
>
> As for the separate header issue, I see no possible way to enforce
> agreement between header and state-of-the-ASDF-system other than
> requiring the developer to do it manually, the burden of which is just
> what causes the problem in the first place.
I don't agree that this is such a problem. I can readily imagine a
program that will take a system, tar it up, compute a checksum, dump a
header file, and pour the tarball and header file into some specified
location.
this seems to me to also be an empirical question, and I should
probably put up or shut up, instead of sending emails. I will try to
cons together something half-baked for comments.
...snip...
> "Easier" to me means computer automated, and as we are Lisp
> programmers, I think would likely involve Lisp, and since we have a
> Lisp-compatible format that system developers must keep up-to-date for
> describing Lisp systems, it seems most natural to use that form.
Agreed. I have been trying to move myself towards this by eating my
own dog food and starting to phase out my use of perl for sysadminning
chores in favor of CL. My impression is mostly good now that there's
the admirable CL-PPCRE, but I'm not certain about portably calling
external programs.
>
> Pascal Bourguignon addressed the security issue in his post; we have
> ways to sterilize S-expression input, looking for the magic :version
> "blah" keyword, and dependencies, which then probably gets stored in a
> central database to support cheap queries, associated with the
> modification date & checksum of the .tar.gz file.
I think given our anarchic society here, allowing people to continue to
put up files whereever they want and do the URL equivalent of
symlinking from CLIKI will be best, as opposed to asking someone to
maintain a more formal central database. But I'm certainly open to
being convinced otherwise.
>
> As an aside, this whole discussion leads me to wonder about the patch
> mechanism supported by Genera (which I have no concept of, beyond the
> apparent ability I've seen in screenshots) to distribute patches and
> versions of multiple libraries, and whether existing Lisp systems are
> lacking the power to do what they did, and if creating that ability
> within ASDF or ASDF-INSTALL is really what we are looking for.
There's something like that for Allegro, and you could find
documentation for it on line. But patch files are YA proliferation of
entities to track, and they are difficult to generate and package
correctly. I think the patch technique is really most useful to people
who are distributing images, rather than source code.
I believe that this discussion has moved towards an approach which I
recap as "header files with version information and tarball pointers in
known locations, header files to contain checksums to allow detection
of tarball deviation, and signed. Said header files to be
automagically generated with some simple automation. At the expense of
being bandwidth-wasteful, this would approach would require the
absolute minimum additional effort (indeed if the packaging process
could be rolled in, would require negative additional effort!) over
existing practice while providing the possibilty of automatically
polling for updates.
"RPG" <·········@gmail.com> writes:
> Agreed. I have been trying to move myself towards this by eating my
> own dog food and starting to phase out my use of perl for sysadminning
> chores in favor of CL. My impression is mostly good now that there's
Which Lisp implementation(s) are you using for this?
Paolo
--
Why Lisp? http://wiki.alu.org/RtL%20Highlight%20Film
The Common Lisp Directory: http://www.cl-user.net
Paolo Amoroso wrote:
> "RPG" <·········@gmail.com> writes:
>
> > Agreed. I have been trying to move myself towards this by eating my
> > own dog food and starting to phase out my use of perl for sysadminning
> > chores in favor of CL. My impression is mostly good now that there's
>
> Which Lisp implementation(s) are you using for this?
>
My primary Lisp is Allegro, although I dabble with CMUCL and SBCL
(mostly in attempts to make stuff I write be portable).
Allegro has pretty good facilities for invoking programs through the
shell. I am not really up on the equivalent facilities for the
CMU-derived CLs.
As an experiment the other day I took one of my throw-away perl scripts
and translated it into CL to see how painful it would be. It wasn't at
all bad, and probably would have been better if it wasn't for the fact
that I'd written the perl first, thus sacrificing much of the advantage
of having a REPL at my disposal....
I've been meaning to write up the experience; I don't really have a
blog, though, so don't have a good platform for something this
half-baked :-)
R
From: James Bielman
Subject: Re: A Humble Request: ASDF-INSTALL and version numbers
Date:
Message-ID: <87pslla8ke.fsf@jamesjb.com>
···············@hotmail.com" <············@gmail.com> writes:
> There's no reason I can see in principle why a crawler could not
> locate and parse .asd files within a .tar.gz file to retrieve the
> new version, and also, as an added feature, check the package
> dependency list for any change.
Personally, I don't think the ASDF system definition is the right
place to be putting the version of a software distribution in the
first place. It's like giving Makefile targets a version number.
The CL-PPCRE distribution as a whole should have a version number, not
the system to build the CL-PPCRE library, or CL-PPCRE-TESTS, or what
have you.
IMHO, one of the things ASDF-INSTALL doesn't get right is the
conflation of a distribution and a system definition. A distribution
can contain multiple systems, and we shouldn't need to rely on
convention to find the special "root" system definition that has the
distribution's version, etc.
I've been working on designing a successor to ASDF-INSTALL, because I
think it will be easier to address these issues starting from scratch
then trying to retrofit ASDF-INSTALL.
James