Sunil Mishra <·······@firstworld.net> writes:
Sunil, thanx for posting this. I was going to sit down this evening
and write up something very similiar after seeing your posts to the
lispweb mailing list. I will give me initial response to this below.
> The availability of a robust network configuration utility shall encourage
> lisp programmers to make smaller systems available. We can also hope that
> some of the larger monolithic systems might be componentized, so that
> portionso fhtem become reusable. (A simple example might be a matrix
> manipulation component of a larger mathematical package.) Such sharing of
> packages would be very beneficial to the lisp community at large.
I think this is a worthy goal. A reduction in packaging overhead and
the work it takes to make sure that people even know your small system
exists should be reduced to a point where it's trivial to write system
definition for it, tar it up and submit it to an archive for
distribution via netconf.
> In the present conceptualization, netconf shall not provide any of
> these. It shall instead implement a policy for coordinating other packages
> to achieve the same end.
>
> * A network interface
> There are many packages that do this for individual platforms. In netconf
> we shall provide an abstract interface that one of these packages shall
> have to obey. It would have to be up to the user to pick one network
> interface facility. Over time, we hope that the netconf utility shall be
> augmented with third party interfaces.
Sam Steingold has been building a layer on top of various lisps, free
and commercial which provides a portale network interface. This may
offer the basic network operations that netconf needs. There are some
questions as to how much of CLOCC is also needed in order to use
net.lisp, and how we can reduce that to a minimum, while still
tracking Sam' work in the future, making the maintenance of this
network interface something that can be shared by many people, and
re-used in other systems.
> * An unpacking system
> Again, there is no need to write this. Netconf shall use tar/gzip for
> this purpose.
I agree that an existing packaging format should be used. Sam also
has a portable interface to invoking external programs, shell.lisp. I
think re-using his work would again offload it's maintenance to many
people, and make it available to other projects.
> * A compilation facility
> * A version tracking facility
> Both of these are already available in mk:defsystem. We shall have to
> interface netconf to it. I believe this can be relatively easily
> achieved. It is a matter of defining a policy for associating the system
> name and version to its place in the shared network archive.
I can see several new attributes being added to system definitions
which would make them much more easier to choose, manage, and
coordinate with a netconf interface. The information that would
normally go into a Debian packages "control" file could be added as
system attributes. Some of these I see are:
:keyword "a list of strings which are keywords indicating what the
contents of this package provide or ae otherwise associated
with"
:reccommendations "a list of other systems that are not required, but
fit nicely with this system"
:license "A string naming the license under which this system is
distributed, and perhaps some other useful information if
the license does not have a common name."
:authors "a list of persons responsible for this package, included
their email address"
:compatible-lisps "a list of identifiers indicating which lisp
implementations this system works in. It could be
:all as well, incidating the package is written to
be CL compatible in all reasonably compliant lisps"
> A system name should be easily translatable to a path. html.parser should
> translate to .../html/parser/... In this directory, the system definition
> shall be stored as .../html/parser/config/html.parser.system. The config
> directory has a special status. It is the only directory that is searched
> for system definition files when setting up the package on the local
> machine. (In order to avoid conflicts, we recommend that the corresponding
> package name for the system to be either "HTML.PARSER" or "HTML-PARSER".)
I think this is an acceptalble starting point. I also like that at
least according to your example use of netconf, obtaining and
installing a package on the local system is seperate from compiling
and loading it into a given lisp image.
Something that might be useful is to change the directory structure
to:
<netconfroot>/<version>/html/parser/...
This way multiple versions of the same package could be installed on a
machine, which is quite useful.
> The server setup policies shall resemble the user setup policies. For each
> distinct package, we shall have a config subdirectory. The remaining files
> in the package shall be stored in a tarred gzipped archive. Taking once
> again the example of html.parser, it would be stored on the server in the
> directory .../html/parser/html.parser.tgz. The system definition file,
> html.parser.system, shall be stored in
> .../html/parser/config/html.parser.system. Once again, the determination of
> packagehood is predicated on the presence of the config subdirectory in a
> package.
What also might be useful is to have netconf download a index from the
server site which is basically a concatenation of all of the system
definitions for systems the servers knows about, with just the package
description information included (version, author, description, title,
comptible lisps, etc...)
This way I could ask netconf for a list of available packages. This
is REALLY important because discovery of available tools is something
the lisp world has a really hard time with now.
> The most difficult aspect of managing the archive is version management. I
> shall talk here only about the simplest approach, where only the latest
> version is ever kept on the server for any package. The version number
> itself is tracked in the system definition file, which should make the
> comparison of two installations of the package relatively simple. (We shall
> obviously also need a version management policy, which we haven't yet
> thought out.)
I think this is a good starting point as well.
> Finally, there is the question of a submission policy. Initially, I propose
> there be an archive maintainer that manually installs packages that are
> supplied in the right format. The archive I suspect will grow quickly. We
> hope the community will be kind enough to take on some of the work in
> providing semi-automated or automated submission processing. Unlike
> installation, submissions must be at least minimally verified, which makes
> the problem more difficult. In particular, verifying machine and
> implementation specific packages will be exceedingly challenging.
I think this is a good place to start, tho I would like to see the
ability for netconf to search multiple servers, and for the
maintenance of packages on the servers to be offloaded to developers.
There are issues with regards to trust and other things that would
have to be addressed before moving to such a distributed system tho.
> One option is to require the system author to supply a test suite for
> verifying that the package is functioning correctly. If not at the server
> side, this can at least provide a means for netconf to test at the client
> side if the package is correctly functioning.
I think test suites are great, and have even written a tool for
writing them quickly, and generating reports. It's based on the
smalltalk "Extreme Programming" test suites. I will make it avialable
ASAP!
--
Craig Brozefsky <·····@red-bean.com>
Lisp Web Dev List http://www.red-bean.com/lispweb
--- The only good lisper is a coding lisper. ---
"Craig Brozefsky" <·····@red-bean.com> wrote in message
···················@piracy.red-bean.com...
> > * A network interface
> > There are many packages that do this for individual platforms. In
netconf
> > we shall provide an abstract interface that one of these packages
shall
> > have to obey. It would have to be up to the user to pick one network
> > interface facility. Over time, we hope that the netconf utility shall
be
> > augmented with third party interfaces.
>
> Sam Steingold has been building a layer on top of various lisps, free
> and commercial which provides a portale network interface. This may
> offer the basic network operations that netconf needs. There are some
> questions as to how much of CLOCC is also needed in order to use
> net.lisp, and how we can reduce that to a minimum, while still
> tracking Sam' work in the future, making the maintenance of this
> network interface something that can be shared by many people, and
> re-used in other systems.
I'm thinking of higher level network I/O. I would rather have netconf
interface with existing FTP/HTTP clients to download files given a URL than
deal with ports and sockets through lisp. If someone wants to write a ports
and sockets I/O layer for the various versions, that should be patchable
into netconf. Otherwise an interface to execute a shell command shall
suffice. I haven't yet decided if I am even going to bother trying to pipe
input from these tools into netconf.
> > * An unpacking system
> > Again, there is no need to write this. Netconf shall use tar/gzip for
> > this purpose.
>
> I agree that an existing packaging format should be used. Sam also
> has a portable interface to invoking external programs, shell.lisp. I
> think re-using his work would again offload it's maintenance to many
> people, and make it available to other projects.
I'll look into this.
> > * A compilation facility
> > * A version tracking facility
> > Both of these are already available in mk:defsystem. We shall have to
> > interface netconf to it. I believe this can be relatively easily
> > achieved. It is a matter of defining a policy for associating the
system
> > name and version to its place in the shared network archive.
>
> I can see several new attributes being added to system definitions
> which would make them much more easier to choose, manage, and
> coordinate with a netconf interface. The information that would
> normally go into a Debian packages "control" file could be added as
> system attributes. Some of these I see are:
>
> :keyword "a list of strings which are keywords indicating what the
> contents of this package provide or ae otherwise associated
> with"
When the time comes, I'd rather have all this put into the documentation
string of the system. This gives several advantages. Instead of relying on
keywords, we can use a simple machine learning style vector comparison for
matching a query to a package description, or some such. But then this is
not something we shall have to worry about for a while yet.
> :reccommendations "a list of other systems that are not required, but
> fit nicely with this system"
>
> :license "A string naming the license under which this system is
> distributed, and perhaps some other useful information if
> the license does not have a common name."
>
> :authors "a list of persons responsible for this package, included
> their email address"
>
> :compatible-lisps "a list of identifiers indicating which lisp
> implementations this system works in. It could be
> :all as well, incidating the package is written to
> be CL compatible in all reasonably compliant lisps"
Yes, something along these lines might be useful. But for the first pass I'm
planning to throw in an ad hoc naming mechanism. Unless someone wants to do
parallel development on mk:defsystem.
> > A system name should be easily translatable to a path. html.parser
should
> > translate to .../html/parser/... In this directory, the system
definition
> > shall be stored as .../html/parser/config/html.parser.system. The config
> > directory has a special status. It is the only directory that is
searched
> > for system definition files when setting up the package on the local
> > machine. (In order to avoid conflicts, we recommend that the
corresponding
> > package name for the system to be either "HTML.PARSER" or
"HTML-PARSER".)
>
> I think this is an acceptalble starting point. I also like that at
> least according to your example use of netconf, obtaining and
> installing a package on the local system is seperate from compiling
> and loading it into a given lisp image.
>
> Something that might be useful is to change the directory structure
> to:
>
> <netconfroot>/<version>/html/parser/...
I did consider something like <netconfroot>/html/parser/<version>/... but
decided against it because mk:defsystem does not currently appear to have
this kind of version selection. Also, the system definition file would also
have to include a version number to allow disambiguation, another can of
worms I didn't want to open. It might be trivially easy to implement, but
not a necessity for a first pass.
> This way multiple versions of the same package could be installed on a
> machine, which is quite useful.
>
> > The server setup policies shall resemble the user setup policies. For
each
> > distinct package, we shall have a config subdirectory. The remaining
files
> > in the package shall be stored in a tarred gzipped archive. Taking once
> > again the example of html.parser, it would be stored on the server in
the
> > directory .../html/parser/html.parser.tgz. The system definition file,
> > html.parser.system, shall be stored in
> > .../html/parser/config/html.parser.system. Once again, the determination
of
> > packagehood is predicated on the presence of the config subdirectory in
a
> > package.
>
> What also might be useful is to have netconf download a index from the
> server site which is basically a concatenation of all of the system
> definitions for systems the servers knows about, with just the package
> description information included (version, author, description, title,
> comptible lisps, etc...)
>
> This way I could ask netconf for a list of available packages. This
> is REALLY important because discovery of available tools is something
> the lisp world has a really hard time with now.
Agreed. This is an important facility, but one that can exist alongside
netconf. I can see a server side process extracting and storing this
information in a data file, and a client side utility for querying the data
file.
> > The most difficult aspect of managing the archive is version management.
I
> > shall talk here only about the simplest approach, where only the latest
> > version is ever kept on the server for any package. The version number
> > itself is tracked in the system definition file, which should make the
> > comparison of two installations of the package relatively simple. (We
shall
> > obviously also need a version management policy, which we haven't yet
> > thought out.)
>
> I think this is a good starting point as well.
>
> > Finally, there is the question of a submission policy. Initially, I
propose
> > there be an archive maintainer that manually installs packages that are
> > supplied in the right format. The archive I suspect will grow quickly.
We
> > hope the community will be kind enough to take on some of the work in
> > providing semi-automated or automated submission processing. Unlike
> > installation, submissions must be at least minimally verified, which
makes
> > the problem more difficult. In particular, verifying machine and
> > implementation specific packages will be exceedingly challenging.
>
> I think this is a good place to start, tho I would like to see the
> ability for netconf to search multiple servers, and for the
> maintenance of packages on the servers to be offloaded to developers.
> There are issues with regards to trust and other things that would
> have to be addressed before moving to such a distributed system tho.
Sorry I didn't make that explicit, but the intention all along was to allow
netconf to search more than one mirror. Distributing versions across mirrors
is relatively painless. I'm sure there is enough non-lisp software out there
for this type of synchronization.
> > One option is to require the system author to supply a test suite for
> > verifying that the package is functioning correctly. If not at the
server
> > side, this can at least provide a means for netconf to test at the
client
> > side if the package is correctly functioning.
>
> I think test suites are great, and have even written a tool for
> writing them quickly, and generating reports. It's based on the
> smalltalk "Extreme Programming" test suites. I will make it avialable
> ASAP!
Thanks for the feedback.
Sunil