Hello,
so far I have been experimenting around with using packages the correct
way. I know this might seem a trivial question to many, yet it is
occupying me for a while. Also, after doing alot of searching I could
not come up with satisfying answers to my questions below.
Here is the setup:
package file: "foo.lisp"
------------------------
;;; -*- Mode: LISP; Syntax: COMMON-LISP; Package: FOO; Base: 10 -*-
(defpackage :foo
(:use :common-lisp)
(:export :fn-foo))
(in-package :foo)
(defun fn-foo ()
(print "in foo:fn-foo"))
------------------------
base file: "bar.lisp"
---------------------
(load "foo") ; w/o this, the next command, "USE-PACKAGE", fails
(use-package :foo)
(defun fn-bar ()
(fn-foo))
---------------------
Now, on the CLISP REPL I do the following:
------------------------------------------
CL-USER> (ext:cd "/path/to/my/lisp/dir")
...
CL-USER> (load "bar")
...
T
CL-USER> (fn-bar)
"in foo:fn-foo"
"in foo:fn-foo"
CL-USER>
------------------------------------------
First of all - is this by any means the way to do it?
What mainly bothers me, is the (LOAD "foo") in 'bar.lisp'. The only way
to avoid this, would be to write it on the REPL, but then I would have
to remember in which order I load my packages each time I start a new
lisp session. Is the only other way to manage this a central
"packages.lisp" file in which I state all my packages as (LOAD "foo")'s?
Additionally, I want to avoid giving absolute path names. With the above
method, I have to have all files in the same directory, which is not
what I actually want. The best way would be to have an (*nix/winXP/osX)
environment variable telling LISP where to look for .lisp files
requested by (LOAD), setting something like the CLISP or CMUCL-specific
CUSTOM:*LOAD-PATHS*
EXTENSIONS:*LOAD-SOURCE-TYPES* and *LOAD-OBJECT-TYPES*
lists at startup. I could not find a sort of "common Common LISP
load-path environment variable" for this? I guess at best there will be
implementation specific env. variables (and if so, I would be interested
specifically in the ones for CLISP and CMUCL, please - couldn't even
find something there...)?
This also leads right to my final question - is there a way to avoid
(LOAD) alltogether? Is there no way that (USE-PACKAGE) loads the
required package files all by itself, not having to load everything
beforehand?
Thanks for any help!
-Florian
Florian Leitner wrote:
> Hello,
>
> so far I have been experimenting around with using packages the correct
> way. I know this might seem a trivial question to many, yet it is
> occupying me for a while. Also, after doing alot of searching I could
> not come up with satisfying answers to my questions below.
[...]
> What mainly bothers me, is the (LOAD "foo") in 'bar.lisp'. The only way
> to avoid this, would be to write it on the REPL, but then I would have
> to remember in which order I load my packages each time I start a new
> lisp session. Is the only other way to manage this a central
> "packages.lisp" file in which I state all my packages as (LOAD "foo")'s?
You want a system definition facility, which is not part of ANSI Common
Lisp. The most widely used ones are ASDF and MK-DEFSYSTEM. Your CL
implementation of choice may also provides its own variation.
Pascal
--
3rd European Lisp Workshop
July 3-4 - Nantes, France - co-located with ECOOP 2006
http://lisp-ecoop06.bknr.net/
Pascal Costanza schrieb:
> Florian Leitner wrote:
>> Hello,
>>
>> so far I have been experimenting around with using packages the correct
>> way. I know this might seem a trivial question to many, yet it is
>> occupying me for a while. Also, after doing alot of searching I could
>> not come up with satisfying answers to my questions below.
> [...]
>
>> What mainly bothers me, is the (LOAD "foo") in 'bar.lisp'. The only way
>> to avoid this, would be to write it on the REPL, but then I would have
>> to remember in which order I load my packages each time I start a new
>> lisp session. Is the only other way to manage this a central
>> "packages.lisp" file in which I state all my packages as (LOAD "foo")'s?
>
> You want a system definition facility, which is not part of ANSI Common
> Lisp. The most widely used ones are ASDF and MK-DEFSYSTEM. Your CL
> implementation of choice may also provides its own variation.
>
OK, thanks for the tip! (although I am astonished this is not native to
lisp - like the include and use directives in other languages?!)
>
> Pascal
>
Florian Leitner <···@gmx.net> writes:
> Pascal Costanza schrieb:
> > Florian Leitner wrote:
> >> Hello,
> >>
> >> so far I have been experimenting around with using packages the correct
> >> way. I know this might seem a trivial question to many, yet it is
> >> occupying me for a while. Also, after doing alot of searching I could
> >> not come up with satisfying answers to my questions below.
> > [...]
> >
> >> What mainly bothers me, is the (LOAD "foo") in 'bar.lisp'. The only way
> >> to avoid this, would be to write it on the REPL, but then I would have
> >> to remember in which order I load my packages each time I start a new
> >> lisp session. Is the only other way to manage this a central
> >> "packages.lisp" file in which I state all my packages as (LOAD "foo")'s?
> >
> > You want a system definition facility, which is not part of ANSI Common
> > Lisp. The most widely used ones are ASDF and MK-DEFSYSTEM. Your CL
> > implementation of choice may also provides its own variation.
> >
>
> OK, thanks for the tip! (although I am astonished this is not native to
> lisp - like the include and use directives in other languages?!)
Just use Asdf, and put all your package definitions into one
file. Like so:
defpackage.lisp:
(defpackage foo
(:use cl)
(:export fn-foo))
foo.asd:
(defpackage foo-system
(:use cl asdf))
(in-package foo-system)
(defsystem foo
:components ((:file "defpackage")
(:file "foo" :depends-on ("defpackage"))
(:file "bar" :depends-on ("foo"))))
Now by loading the Asdf system, first your package(s) will be loaded,
thus made available for the rest of the files, and then the rest of
the files in an appropriate order.
Bj�rn
In article <·····························@news.chello.at>,
Florian Leitner <···@gmx.net> wrote:
> Pascal Costanza schrieb:
> > You want a system definition facility, which is not part of ANSI Common
> > Lisp. The most widely used ones are ASDF and MK-DEFSYSTEM. Your CL
> > implementation of choice may also provides its own variation.
> >
>
> OK, thanks for the tip! (although I am astonished this is not native to
> lisp - like the include and use directives in other languages?!)
It's more like "make", which is not part of any language standards.
--
Barry Margolin, ······@alum.mit.edu
Arlington, MA
*** PLEASE post questions in newsgroups, not directly to me ***
*** PLEASE don't copy me on replies, I'll read them in the group ***
On 2006-03-28, Florian Leitner <···@gmx.net> wrote:
> Pascal Costanza schrieb:
>> You want a system definition facility, which is not part of ANSI
>> Common Lisp. The most widely used ones are ASDF and MK-DEFSYSTEM.
>> Your CL implementation of choice may also provides its own
>> variation.
>
> OK, thanks for the tip! (although I am astonished this is not native
> to lisp - like the include and use directives in other languages?!)
Nitpick: 'include' in C and 'use' in Perl are analogous to 'load' in
Lisp. A system definition facility is analogous to 'make'.
Note that Perl doesn't have a system definition facility, either:
everything a modules uses, it 'use's, and it all gets compiled
every time it loads[1].
*Perhaps*, if you did it right, you could get CL's "require" to work
like Perl's "use", but you're better off with asdf or mk-defsystem.
Just my two cents.
-- Larry
[1] except for any external C modules, of course
Florian Leitner <···@gmx.net> writes:
> [...]
> What mainly bothers me, is the (LOAD "foo") in 'bar.lisp'. The only way
> to avoid this, would be to write it on the REPL, but then I would have
> to remember in which order I load my packages each time I start a new
> lisp session. Is the only other way to manage this a central
> "packages.lisp" file in which I state all my packages as (LOAD "foo")'s?
Unless you use a system definition facility, you'll have to load your
files explicitly. You can do that in a file "loader.lisp" for example.
It is advised to put all package definitions in a separate
package.lisp file loaded first. I don't like it either, usually it's
not necessary. When the package use graph is a tree, you can put each
defpackage it its own file.
> Additionally, I want to avoid giving absolute path names. With the above
> method, I have to have all files in the same directory, which is not
> what I actually want. The best way would be to have an (*nix/winXP/osX)
> environment variable telling LISP where to look for .lisp files
> requested by (LOAD), setting something like the CLISP or CMUCL-specific
> CUSTOM:*LOAD-PATHS*
> EXTENSIONS:*LOAD-SOURCE-TYPES* and *LOAD-OBJECT-TYPES*
> lists at startup. I could not find a sort of "common Common LISP
> load-path environment variable" for this? I guess at best there will be
> implementation specific env. variables (and if so, I would be interested
> specifically in the ones for CLISP and CMUCL, please - couldn't even
> find something there...)?
Better, you can define your own user-specific variable!
Actually, I use logical pathnames for this. I install most lisp
libraries in /usr/local/share/lisp/packages/FQDN/ but this isn't
important because in my ~/.clisprc.lisp, ~/.sbclrc, ~/openmcl.lisp
~/.cmucl-init.lisp, etc I load ~/.common.lisp:
(LOAD (MERGE-PATHNAMES
(MAKE-PATHNAME :NAME ".common" :TYPE "lisp") (USER-HOMEDIR-PATHNAME)))
;; Happilly, all these implementations accept this dot in the name, pfff!
;; One day, I should rename it ~/common.lisp
Then, in .common.lisp, I define logical pathname translations for the
logical host PACKAGES.
So, eventually, I use absolute paths to refer to library packages, but
thru the indirection of the logical pathnames
To load the package: "COM.INFORMATIMAGO.COMMON-LISP.PACKAGE"
I load the file: #P"PACKAGE:COM;INFORMATIMAGO;COMMON-LISP;PACKAGE"
which is stored at:
/usr/local/share/lisp/package/com/informatimago/common-lisp/package.lisp
(actually, if it exists, this compiled file is loaded instead:
/usr/local/share/lisp/package/com/informatimago/common-lisp/${IMPLEM}/package.${EXT}
)
Now, in a project, I have a init.lisp file which makes additionnal
logical pathname translations, specific to the project. For example:
(LET ((PREFIX
(MAKE-PATHNAME
:DIRECTORY (PATHNAME-DIRECTORY
(if (getenv "PREFIX")
(CONCATENATE 'STRING (GETENV "PREFIX") "/")
(merge-pathnames
(make-pathname :directory '(:relative "install")
:defaults *LOAD-PATHNAME*)
*LOAD-PATHNAME*))))) ;; <--- NOTE!
(BASE
(MAKE-PATHNAME
:DIRECTORY (PATHNAME-DIRECTORY
(if (getenv "BASE")
(CONCATENATE 'STRING (GETENV "BASE") "/")
*LOAD-PATHNAME*)))))
(flet ((MP (prefix SUB) (MERGE-PATHNAMES SUB PREFIX))
(concat (&rest args) (apply (function concatenate) 'string args)))
((LAMBDA (&REST SPECS)
(DOLIST (SPEC SPECS)
(SETF (LOGICAL-PATHNAME-TRANSLATIONS (FIRST SPEC))
(NCONC
(HANDLER-CASE (LOGICAL-PATHNAME-TRANSLATIONS (FIRST SPEC))
(ERROR () NIL))
(MAPCAR
(LAMBDA (REST)
(if (consp rest)
(LIST
(logical-pathname (concat (SECOND SPEC) (first REST)))
(MP (third spec) (concat (fourth SPEC) (second REST))))
(LIST
(logical-pathname (concat (SECOND SPEC) REST))
(MP (third spec) (concat (fourth SPEC) REST)))))
#+CLISP '("*.*" "*")
#+sbcl '(("*.*.*" "*.*") "*.*" "*")
#-(OR CLISP sbcl) '("*" "*.*" ("*.*.*" "*.*"))
)))))
;; translations for the PROJECT logical host:
`("PROJECT" "PROJECT:**;"
,base "**/")
;; translation for the PROJECT packages:
`("PACKAGES" "PACKAGES:COM;INFORMATIMAGO;PROJECT;**;"
,base "src/**/")
;; In this project, we use freezed local copies of the libraries:
`("PACKAGES" "PACKAGES:**;"
,prefix "share/lisp/packages/**/")
`("PACKAGES" "PACKAGES:NET;COMON-LISP;UCW;**;"
,prefix "share/lisp/packages/net/common-lisp/ucw/**/"))))
Then in the loader.lisp file, I can load the project files thru the
PACKAGES logical host as always:
;; Load ../init.lisp, which defines PACKAGES and "PROJECT" logical hostnames.
(load (make-pathname
:directory (append (pathname-directory *load-truename*) '(:up))
:name "init" :TYPE "lisp" :case :local
:defaults *load-truename*))
;; ...
(defvar *compile* nil "Whether loaded files must be compiled too.")
(defun process (file) (if *compile* (compile-file file) file))
(defun load-app-file (name)
(loop
:with again = t
:while again
:do (restart-case
(progn
(setf again nil)
(LOAD (process
(make-pathname
:host "PACKAGES"
:directory '(:absolute "COM""INFORMATIMAGO""BELLEROPHON")
:name name
:type "LISP"
:version :newest))))
(reload ()
:report (lambda (stream)
(format stream "Reload ~S" *load-pathname*))
(setf again t)))))
For the libraries, in loader.lisp I have:
(unless (find-package "ASDF")
(LOAD-PACKAGE "NET.SOURCEFORGE.CCLAN.ASDF.ASDF"))
;; LOAD-PACKAGE is in COM.INFORMATIMAGO.COMMON-LISP.PACKAGE,
;; it translates the package name into an absolute logical path
;; in PACKAGES:... and loads it.
(defparameter *original-asdf-registry* ASDF:*CENTRAL-REGISTRY*)
(defun asdf-rescan-packages ()
(when *load-verbose*
(format *trace-output* "~&;; Scanning ASDF packages...~%"))
(prog1
(SORT
(DELETE-DUPLICATES
(MAPCAR
(LAMBDA (P) (MAKE-PATHNAME :NAME NIL :TYPE NIL :VERSION NIL :DEFAULTS P))
(DIRECTORY "PACKAGES:**;*.ASD"))
:test (function equal))
(LAMBDA (A B) (if (= (length a) (length b))
(string< a b)
(< (length a) (length b))))
:key (function namestring))
(when *load-verbose*
(format *trace-output* "~&;; Done.~%"))))
(defun update-asdf-registry ()
(setf ASDF:*CENTRAL-REGISTRY*
(nconc (asdf-rescan-packages)
*original-asdf-registry*)))
(update-asdf-registry)
;; Then I can use ASDF:OOS to load the asdfied libraries before
;; loading my files:
(dolist (name '("F1" "F2" ... "FN"))
(load-app-file name))
This may sound complex, but even with ASDF, once you've loaded a
library, you'll have initialization and setup forms to execute,
sometimes before loading the next library, so a loader file is very
useful here. For example, before asdf loading UCW.ADMIN, you need to
set the ucw *default-server* variable, for which you need first to
asdf load UCW. Hence a loader.lisp sequence:
(asdf:operate 'asdf:load-op :UCW :verbose *load-verbose*)
(use-package "IT.BESE.UCW")
(asdf:operate 'asdf:load-op :ucw.mod-lisp :verbose *load-verbose*)
(defvar *backend* (make-instance 'mod-lisp-backend :port 3001))
(setf *default-server* (make-instance 'standard-server :backend *backend*))
(defparameter IT.BESE.UCW-USER::*UCW-TAL-ROOT* (config-tal-root *config*))
;; Depend on IT.BESE.UCW-USER::*UCW-TAL-ROOT* and on *default-server*
(asdf:operate 'asdf:load-op :ucw.admin :verbose *load-verbose*)
(asdf:operate 'asdf:load-op :ucw.examples :verbose *load-verbose*))
> This also leads right to my final question - is there a way to avoid
> (LOAD) alltogether? Is there no way that (USE-PACKAGE) loads the
> required package files all by itself, not having to load everything
> beforehand?
See my COM.INFORMATIMATIMAGO.COMMON-LISP.PACKAGE package. It has a
DEFINE-PACKAGE macro that load the dependencies, do the DEFPACKAGE
with :USE and the IN-PACKAGE.
http://www.informatimago.com/develop/lisp/index.html
At the REPL, I can either asdf load a whole set of library packages,
(asdf-load :com.informatimago.common-lisp)
loads COM.INFORMATIMAGO.COMMON-LISP.*
or I: (load "package:com.informatimago.common-lisp.cons-to-ascii")
and the define-package form in it automatically loads all dependencies.
--
__Pascal Bourguignon__ http://www.informatimago.com/
In a World without Walls and Fences,
who needs Windows and Gates?
Pascal Bourguignon schrieb:
> Unless you use a system definition facility, you'll have to load your
> files explicitly. You can do that in a file "loader.lisp" for example.
>
> It is advised to put all package definitions in a separate
> package.lisp file loaded first. I don't like it either, usually it's
> not necessary. When the package use graph is a tree, you can put each
> defpackage it its own file.
>
> Better, you can define your own user-specific variable!
>
> ...
>
> This may sound complex, but even with ASDF, once you've loaded a
> library, you'll have initialization and setup forms to execute,
> sometimes before loading the next library, so a loader file is very
> useful here. For example, before asdf loading UCW.ADMIN, you need to
> set the ucw *default-server* variable, for which you need first to
> asdf load UCW. Hence a loader.lisp sequence:
>
> ...
>
> See my COM.INFORMATIMATIMAGO.COMMON-LISP.PACKAGE package. It has a
> DEFINE-PACKAGE macro that load the dependencies, do the DEFPACKAGE
> with :USE and the IN-PACKAGE.
>
> http://www.informatimago.com/develop/lisp/index.html
>
> At the REPL, I can either asdf load a whole set of library packages,
> (asdf-load :com.informatimago.common-lisp)
> loads COM.INFORMATIMAGO.COMMON-LISP.*
> or I: (load "package:com.informatimago.common-lisp.cons-to-ascii")
> and the define-package form in it automatically loads all dependencies.
This is an impressive array of code to get things to work - I did never
imagine this would get so complex, but the way you finally can (load
"package:path.to.it") is quite exactly what I wanted, plus something
like a base LISPLIB env var using (getenv "LISPLIB") or so.
Some parts of your code are quite hard to understand for me as a lisp
novice - if I can't resolve them, I hope I can get back to you about
this. Thank you very much for this detailed description of how to create
a package facility with lisp! Mean while I'll try to understand your
code and using your PACKAGE package.
-Florian
Florian,
Do not get your checked. You are not going crazy. IMHO, you have
stumbled onto one of the dirty little secrets of Common Lisp -- the
package system is quite broken. (Well, "broken" is putting it strongly,
but you get the idea.)
I'm not the only one that thinks so. Some pretty smart people seem to
agree. If you check out Peter Norvig's very comprehensive comparison of
Lisp and Python, it's one of a few points where he notes Lisp is
inferior as Lisp's packages/modules are "Hard to use"
(http://www.norvig.com/python-lisp.html). Also, if you look at the
suggestions sent to Paul Graham for his new language Arc
(http://www.archub.org/arcsug.txt), you'll see other experienced Lispers
mentioning it as well -- David Moon ("Common Lisp's approach of
modularizing the map from names to symbol objects instead of the map
from symbol objects to bindings is wrong"), Trevor Blackwell ("Sadly, I
find the CL module system way to cumbersome to actually use. It has to
be convenient enough to use in ordinary programming...").
I started writing a help file to explain the Lisp packaging system and
its best practices in terms familiar to someone from Python. I still
haven't finished because I still haven't got my head around it. But as
far as I now understand it, your options seem to be:
1. Manually manage loading order of single-file packages, e.g., "foo.lisp".
This is necessary because if a package refers to another pacakge, then
the dependent DEFPACKAGE form needs to load first. But this is an
obviously clumsy system and it rapidly becomes impossible to manage if
you ever use a lot of code or other people's code.
2. Split every package between _at least_ two files, one containing the
actual code, and another containing the DEFPACKAGE form.
By putting all your packages' DEFPACKAGE forms in a single file, you
don't need to worry about the order the forms are loaded.
This is one step forward but one step back, since we've now broken the
one-to-one correspondence between packages and files. If you want to add
or remove ANY package, you can't just put something in a directory. You
need to go to your packages.lisp file and find the right DEFPACKAGE
form. Also, that DEFPACKAGE is where you manage the list of that
package's imported and exported functions, so expect to go there a lot.
Of course you could keep every package's DEFPACKAGE form separately in a
foo-package.lisp, but then you're back to managing the loading order of
your *-package.lisp files. Also this situation still sucks because, as
you point out, what you really want is to think about package names not
filenames and pathnames. That leads us to asdf, your next step on this
road of disillusionment and enlightenment.
3) Split every package into three files, one containing the actual code
(foo.lisp), one containing its DEFPACKAGE form (foo-package.lisp), and a
third containing its asdf DEFSYSTEM declaration (foo.asd).
This new, third file will contain an asdf DEFSYSTEM form. This is
another package-like abstraction, layered on top of DEFPACKAGE (which is
itself layered over MAKE-PACKAGE, PROVIDE, and other deprecated horrors
we pass over in silence). A DEFSYSTEM form will give your package a
logical system name, and define how it depends on other asdf systems.
The .asd file is itself a somewhat hairy beast that typically creates
temporary "pseudo-packages" like foo.system in order to handle its
business (e.g., (defpackage foo.system ...)). But such is life.
This is, I believe, the current best practice. This is the _the minimum_
you need in order to have the computer automatically manage filenames
and loading order. And to do this you had to learn the correct way to
break your package into (at least) three files, and how to manage two
logically separate packaging abstractions. And none of this stuff is
documented in one place, not even in the usual books. To put this story
together you need to go read the ASDF manual, the packaging chapter in
PCL, and probably a few downloaded examples. Since most of us here know
other languages (and not just perversities like c++), I think we also
know it's usually not so hard in other languages.
On the other hand, once you sign on with asdf, you do get other good
things. It's the same system that will make your package
asdf-installable, so that will make it very easy to distribute it and
its dependencies over the internet. Some have commented that asdf is
more than just a packaging facility, that it's more like Make. I can't
speak to this myself, but I believe it -- it's very customizable.
<rant>
But still, IMHO, the current situation is absurd. It should not require
so much machinery and configuration to setup and use a package! As I'm
sure Peter Norvig would remind us, in Python this same task is only a
matter of creating a new file, naming it appropriately, and dropping it
into the current directory. Boom, a package! Done! It makes you wanna
holler.
I suspect the clunky Lisp packaging system is the single largest source
of "friction" inhibiting open source development within the existing CL
community, because it makes proper modularization and standardized
code-sharing so hard. I shudder to imagine the dark days before asdf.
</rant>
So what's the solution? What I'm interested in understanding, at the
moment, is if there's a way to create yet another wrapper that will
conceal all this horrible complexity. This wrapper would merely mimic
the easy packaging of lesser langauges. In other words, it would let you:
1) define a package through a single file, including its export list
and its imported dependencies
2) define package names in one-to-one correspondence with pathnames
3) automatically manage loading order
The ugly way to do this would be to create some kind of "build" facility
that takes a single file and then auto-generates the magic triplet of
foo.lisp, foo-package.lisp, and foo.asd. This is ugly, because then you
need to remember to keep running your build tool. yuck. The ideal way is
a set of macros that would integrate with asdf. But even knowing if this
is possible requires much stronger lisp fu than my own.
Sorry if this is a longer and more cantankerous answer than you
expected. I'll go batten down the windows and keep a lookout for the
lynch mob...
all the best,
alexis
Florian Leitner wrote:
> Pascal Bourguignon schrieb:
>> Unless you use a system definition facility, you'll have to load your
>> files explicitly. You can do that in a file "loader.lisp" for example.
>>
>> It is advised to put all package definitions in a separate
>> package.lisp file loaded first. I don't like it either, usually it's
>> not necessary. When the package use graph is a tree, you can put each
>> defpackage it its own file.
>>
>> Better, you can define your own user-specific variable!
>>
>> ...
>>
>> This may sound complex, but even with ASDF, once you've loaded a
>> library, you'll have initialization and setup forms to execute,
>> sometimes before loading the next library, so a loader file is very
>> useful here. For example, before asdf loading UCW.ADMIN, you need to
>> set the ucw *default-server* variable, for which you need first to
>> asdf load UCW. Hence a loader.lisp sequence:
>>
>> ...
>>
>> See my COM.INFORMATIMATIMAGO.COMMON-LISP.PACKAGE package. It has a
>> DEFINE-PACKAGE macro that load the dependencies, do the DEFPACKAGE
>> with :USE and the IN-PACKAGE.
>>
>> http://www.informatimago.com/develop/lisp/index.html
>>
>> At the REPL, I can either asdf load a whole set of library packages,
>> (asdf-load :com.informatimago.common-lisp)
>> loads COM.INFORMATIMAGO.COMMON-LISP.*
>> or I: (load "package:com.informatimago.common-lisp.cons-to-ascii")
>> and the define-package form in it automatically loads all dependencies.
>
> This is an impressive array of code to get things to work - I did never
> imagine this would get so complex, but the way you finally can (load
> "package:path.to.it") is quite exactly what I wanted, plus something
> like a base LISPLIB env var using (getenv "LISPLIB") or so.
>
> Some parts of your code are quite hard to understand for me as a lisp
> novice - if I can't resolve them, I hope I can get back to you about
> this. Thank you very much for this detailed description of how to create
> a package facility with lisp! Mean while I'll try to understand your
> code and using your PACKAGE package.
>
> -Florian
>
>
On 2006-03-30 04:38:07 -0500, Alexis Gallagher
<······@alexisgallagher.com> said:
> <rant>
> But still, IMHO, the current situation is absurd. It should not require
> so much machinery and configuration to setup and use a package! As
> I'm sure Peter Norvig would remind us, in Python this same task is only
> a matter of creating a new file, naming it appropriately, and dropping
> it into the current directory. Boom, a package! Done! It makes you
> wanna holler.
>
> I suspect the clunky Lisp packaging system is the single largest source
> of "friction" inhibiting open source development within the existing CL
> community, because it makes proper modularization and standardized
> code-sharing so hard. I shudder to imagine the dark days before asdf.
> </rant>
It's actually even worse than this - asdf does not currently require
systems to have a version number, so there's really no automated way to
tell if your currently installed system is the most recent or not
(although there are people working on a solution).
Alexis Gallagher <······@alexisgallagher.com> writes:
> <rant>
> But still, IMHO, the current situation is absurd. It should not
> require so much machinery and configuration to setup and use a
> package! As I'm sure Peter Norvig would remind us, in Python this same
> task is only a matter of creating a new file, naming it appropriately,
> and dropping it into the current directory. Boom, a package! Done! It
> makes you wanna holler.
You can do the same in CL, just accept to use LOAD.
(defun use-module (x) (load x)) ; name it module instead of package
; to avoid the confusion with common lisp packages that are not modules.
--
__Pascal Bourguignon__ http://www.informatimago.com/
CAUTION: The mass of this product contains the energy equivalent of
85 million tons of TNT per net ounce of weight.
From: Alexander Schmolck
Subject: Re: packages, load paths and environment variables
Date:
Message-ID: <yfs3bh0c4mq.fsf@oc.ex.ac.uk>
Pascal Bourguignon <······@informatimago.com> writes:
> Alexis Gallagher <······@alexisgallagher.com> writes:
>
> > <rant>
> > But still, IMHO, the current situation is absurd. It should not
> > require so much machinery and configuration to setup and use a
> > package! As I'm sure Peter Norvig would remind us, in Python this same
> > task is only a matter of creating a new file, naming it appropriately,
> > and dropping it into the current directory. Boom, a package! Done! It
> > makes you wanna holler.
>
> You can do the same in CL, just accept to use LOAD.
>
> (defun use-module (x) (load x)) ; name it module instead of package
> ; to avoid the confusion with common lisp packages that are not modules.
This is like
def use_module(x): execfile(x, globals())
which is a far cry from from python's ``import x``
'as
Alexander Schmolck <··········@gmail.com> writes:
> Pascal Bourguignon <······@informatimago.com> writes:
>
>> Alexis Gallagher <······@alexisgallagher.com> writes:
>>
>> > <rant>
>> > But still, IMHO, the current situation is absurd. It should not
>> > require so much machinery and configuration to setup and use a
>> > package! As I'm sure Peter Norvig would remind us, in Python this same
>> > task is only a matter of creating a new file, naming it appropriately,
>> > and dropping it into the current directory. Boom, a package! Done! It
>> > makes you wanna holler.
>>
>> You can do the same in CL, just accept to use LOAD.
>>
>> (defun use-module (x) (load x)) ; name it module instead of package
>> ; to avoid the confusion with common lisp packages that are not modules.
>
> This is like
>
> def use_module(x): execfile(x, globals())
>
> which is a far cry from from python's ``import x``
Indeed, this is a question of semantics. If you can impose some
restrictions and conventions (some "good practice" of modularization),
then you can define some a module API. If I knew anything of python,
I could write an equivalent IMPORT operator.
--
__Pascal Bourguignon__ http://www.informatimago.com/
"What is this talk of "release"? Klingons do not make software
"releases". Our software "escapes" leaving a bloody trail of
designers and quality assurance people in its wake."
Alexis,
I will look forward to your howto on Lisp packaging, really! It was
quite a mess for me just to sort out how the "basic" system works, even
with some very good books on Lisp.
I think, with your statements and Pascals explanations, I will for the
time being simply stick with LOAD as Pascal suggests, and build
something from his explanations using an env var for the base library
paths (as I'm used to in Perl & co). Although, admittedly, I must learn
to use asdf as well, as I've noticed quite a few interesting sources are
using it, too. And if I ever have something nice to contribute (...), it
should probably best be using the asdf system, I guess. No worries about
the length of your post, at least I now have a good picture of the
"state of the art" (or mess?).
-Florian
Alexis Gallagher schrieb:
> Florian,
>
> Do not get your checked. You are not going crazy. IMHO, you have
> stumbled onto one of the dirty little secrets of Common Lisp -- the
> package system is quite broken. (Well, "broken" is putting it strongly,
> but you get the idea.)
>
> I'm not the only one that thinks so. Some pretty smart people seem to
> agree. If you check out Peter Norvig's very comprehensive comparison of
> Lisp and Python, it's one of a few points where he notes Lisp is
> inferior as Lisp's packages/modules are "Hard to use"
> (http://www.norvig.com/python-lisp.html). Also, if you look at the
> suggestions sent to Paul Graham for his new language Arc
> (http://www.archub.org/arcsug.txt), you'll see other experienced Lispers
> mentioning it as well -- David Moon ("Common Lisp's approach of
> modularizing the map from names to symbol objects instead of the map
> from symbol objects to bindings is wrong"), Trevor Blackwell ("Sadly, I
> find the CL module system way to cumbersome to actually use. It has to
> be convenient enough to use in ordinary programming...").
>
> I started writing a help file to explain the Lisp packaging system and
> its best practices in terms familiar to someone from Python. I still
> haven't finished because I still haven't got my head around it. But as
> far as I now understand it, your options seem to be:
>
> 1. Manually manage loading order of single-file packages, e.g., "foo.lisp".
>
> This is necessary because if a package refers to another pacakge, then
> the dependent DEFPACKAGE form needs to load first. But this is an
> obviously clumsy system and it rapidly becomes impossible to manage if
> you ever use a lot of code or other people's code.
>
> 2. Split every package between _at least_ two files, one containing the
> actual code, and another containing the DEFPACKAGE form.
>
> By putting all your packages' DEFPACKAGE forms in a single file, you
> don't need to worry about the order the forms are loaded.
>
> This is one step forward but one step back, since we've now broken the
> one-to-one correspondence between packages and files. If you want to add
> or remove ANY package, you can't just put something in a directory. You
> need to go to your packages.lisp file and find the right DEFPACKAGE
> form. Also, that DEFPACKAGE is where you manage the list of that
> package's imported and exported functions, so expect to go there a lot.
>
> Of course you could keep every package's DEFPACKAGE form separately in a
> foo-package.lisp, but then you're back to managing the loading order of
> your *-package.lisp files. Also this situation still sucks because, as
> you point out, what you really want is to think about package names not
> filenames and pathnames. That leads us to asdf, your next step on this
> road of disillusionment and enlightenment.
>
> 3) Split every package into three files, one containing the actual code
> (foo.lisp), one containing its DEFPACKAGE form (foo-package.lisp), and a
> third containing its asdf DEFSYSTEM declaration (foo.asd).
>
> This new, third file will contain an asdf DEFSYSTEM form. This is
> another package-like abstraction, layered on top of DEFPACKAGE (which is
> itself layered over MAKE-PACKAGE, PROVIDE, and other deprecated horrors
> we pass over in silence). A DEFSYSTEM form will give your package a
> logical system name, and define how it depends on other asdf systems.
>
> The .asd file is itself a somewhat hairy beast that typically creates
> temporary "pseudo-packages" like foo.system in order to handle its
> business (e.g., (defpackage foo.system ...)). But such is life.
>
> This is, I believe, the current best practice. This is the _the minimum_
> you need in order to have the computer automatically manage filenames
> and loading order. And to do this you had to learn the correct way to
> break your package into (at least) three files, and how to manage two
> logically separate packaging abstractions. And none of this stuff is
> documented in one place, not even in the usual books. To put this story
> together you need to go read the ASDF manual, the packaging chapter in
> PCL, and probably a few downloaded examples. Since most of us here know
> other languages (and not just perversities like c++), I think we also
> know it's usually not so hard in other languages.
>
> On the other hand, once you sign on with asdf, you do get other good
> things. It's the same system that will make your package
> asdf-installable, so that will make it very easy to distribute it and
> its dependencies over the internet. Some have commented that asdf is
> more than just a packaging facility, that it's more like Make. I can't
> speak to this myself, but I believe it -- it's very customizable.
>
> <rant>
> But still, IMHO, the current situation is absurd. It should not require
> so much machinery and configuration to setup and use a package! As I'm
> sure Peter Norvig would remind us, in Python this same task is only a
> matter of creating a new file, naming it appropriately, and dropping it
> into the current directory. Boom, a package! Done! It makes you wanna
> holler.
>
> I suspect the clunky Lisp packaging system is the single largest source
> of "friction" inhibiting open source development within the existing CL
> community, because it makes proper modularization and standardized
> code-sharing so hard. I shudder to imagine the dark days before asdf.
> </rant>
>
> So what's the solution? What I'm interested in understanding, at the
> moment, is if there's a way to create yet another wrapper that will
> conceal all this horrible complexity. This wrapper would merely mimic
> the easy packaging of lesser langauges. In other words, it would let you:
> 1) define a package through a single file, including its export list
> and its imported dependencies
> 2) define package names in one-to-one correspondence with pathnames
> 3) automatically manage loading order
>
> The ugly way to do this would be to create some kind of "build" facility
> that takes a single file and then auto-generates the magic triplet of
> foo.lisp, foo-package.lisp, and foo.asd. This is ugly, because then you
> need to remember to keep running your build tool. yuck. The ideal way is
> a set of macros that would integrate with asdf. But even knowing if this
> is possible requires much stronger lisp fu than my own.
>
> Sorry if this is a longer and more cantankerous answer than you
> expected. I'll go batten down the windows and keep a lookout for the
> lynch mob...
>
> all the best,
> alexis
>
>
>
> Florian Leitner wrote:
>> Pascal Bourguignon schrieb:
>>> Unless you use a system definition facility, you'll have to load your
>>> files explicitly. You can do that in a file "loader.lisp" for example.
>>>
>>> It is advised to put all package definitions in a separate
>>> package.lisp file loaded first. I don't like it either, usually it's
>>> not necessary. When the package use graph is a tree, you can put each
>>> defpackage it its own file.
>>>
>>> Better, you can define your own user-specific variable!
>>>
>>> ...
>>>
>>> This may sound complex, but even with ASDF, once you've loaded a
>>> library, you'll have initialization and setup forms to execute,
>>> sometimes before loading the next library, so a loader file is very
>>> useful here. For example, before asdf loading UCW.ADMIN, you need to
>>> set the ucw *default-server* variable, for which you need first to
>>> asdf load UCW. Hence a loader.lisp sequence:
>>>
>>> ...
>>>
>>> See my COM.INFORMATIMATIMAGO.COMMON-LISP.PACKAGE package. It has a
>>> DEFINE-PACKAGE macro that load the dependencies, do the DEFPACKAGE
>>> with :USE and the IN-PACKAGE.
>>>
>>> http://www.informatimago.com/develop/lisp/index.html
>>>
>>> At the REPL, I can either asdf load a whole set of library packages,
>>> (asdf-load :com.informatimago.common-lisp) loads
>>> COM.INFORMATIMAGO.COMMON-LISP.*
>>> or I: (load "package:com.informatimago.common-lisp.cons-to-ascii")
>>> and the define-package form in it automatically loads all dependencies.
>>
>> This is an impressive array of code to get things to work - I did never
>> imagine this would get so complex, but the way you finally can (load
>> "package:path.to.it") is quite exactly what I wanted, plus something
>> like a base LISPLIB env var using (getenv "LISPLIB") or so.
>>
>> Some parts of your code are quite hard to understand for me as a lisp
>> novice - if I can't resolve them, I hope I can get back to you about
>> this. Thank you very much for this detailed description of how to create
>> a package facility with lisp! Mean while I'll try to understand your
>> code and using your PACKAGE package.
>>
>> -Florian
>>
>>
>
>
Florian Leitner wrote:
> Alexis,
>
> I will look forward to your howto on Lisp packaging, really! It was
> quite a mess for me just to sort out how the "basic" system works, even
> with some very good books on Lisp.
>
> I think, with your statements and Pascals explanations, I will for the
> time being simply stick with LOAD as Pascal suggests, and build
> something from his explanations using an env var for the base library
> paths (as I'm used to in Perl & co). Although, admittedly, I must learn
> to use asdf as well, as I've noticed quite a few interesting sources are
> using it, too. And if I ever have something nice to contribute (...), it
> should probably best be using the asdf system, I guess. No worries about
> the length of your post, at least I now have a good picture of the
> "state of the art" (or mess?).
I find claims that packages and system definitions are a mess highly
exaggerated. There are a handful of fundamental concepts you need to
understand, and the rest more or less follows from there. For packages,
it is important to know that they strings to symbols, not symbols to
concepts (like variables, functions, classes, etc.), plus the sidefact
that symbols are typically written in lower case, but by default
converted to upper case internally. For system definitions, it is
important to understand that they "merely" declare relationships between
files and other systems - what files need to be compiled and/or loaded
before other files. It's nice that such system definitions are very
declarative, so that you don't have to figure out the actual order in
which to do these things yourself.
It's true that other languages have module systems that appear to be
simpler and more straightforward to use. But it also seems to be true
that in the long run, they also end up to use more complex approaches,
i.e. make files, because the kind of configurability that make files
provide is ultimately necessary, for example to be able to produce
different versions of software from the same set of source files. Common
Lisp packages + (non-standard) system definition facilities allow you to
express such things which are, as far as I can tell, hard or impossible
to express in other environments.
Common Lisp was never intended to make simple tasks simple at the cost
of making hard things even harder. It's true that this creates some
steep learning curves.
Pascal
--
3rd European Lisp Workshop
July 3-4 - Nantes, France - co-located with ECOOP 2006
http://lisp-ecoop06.bknr.net/
Pascal Costanza wrote:
> It's true that other languages have module systems that appear to be
> simpler and more straightforward to use. But it also seems to be true
> that in the long run, they also end up to use more complex approaches,
> i.e. make files, because the kind of configurability that make files
> provide is ultimately necessary, for example to be able to produce
> different versions of software from the same set of source files.
I agree. I would add a few other example of this: running unit tests,
creating and deleting test-related assets, and compiling and linking
with foreign language interfaces. You're right -- my python project
needs a Makefile to handle those tasks, which I think could all be
handled more conveniently by asdf if that project were written in CL.
Also, FWIW, asdf-install strikes me as being better than python's setup
tools.
>
> Common Lisp was never intended to make simple tasks simple at the cost
> of making hard things even harder. It's true that this creates some
> steep learning curves.
>
Here, however, I think you misunderstand me...
I am entirely in favor of making hard things easy even if it requires a
high upfront _conceptual_ cost to learn a system. But my complaint about
the CL packaging system (meaning, DEFPACKAGE + asdf) is that it seems to
impose a _permanent administrative_ cost. Even if you understand the
ideas perfectly (I'm confident Peter Norvig and David Moon aren't
complaining because they don't "get it"), then you are still forced to
manage package loading order, to manage packagename-to-pathname
mappings, or to manage a minimum of three files per package. This is a
lot of messiness just to keep your code tidy.
Java copied CL's memory management. Python copied the REPL. Ruby copied
lambda's through its blocks. What did ocaml and Haskell copy? I don't
know, but let's concede they copied something. It certainly seems
suggestive that _none_ of these languages copied CL's packaging system,
and that all of them automatically manage package loading order and the
packagename-to-pathname mapping.
But this is my real point -- I suspect the administrative burden of CL's
system is not the necessary price of its flexibility. It could be fixed.
Pascal Bourguignon seemed confident that if he knew python, he could
write a CL equivalent to the Python "import" operator. If this is so,
then that seems to prove these complexities are not the unavoidable in CL.
In fact, I'd dearly like to fix them just for my own convenience, but I
know nothing compared to you guys, so I can't know for sure if it's
possible. So I sincerely ask the wizards of this list, is there a
fundamental reason it would be impossible to write a macro such as the
following ---
A single file "syspackagename.lisp" would have this as its first form:
(defsyspackage :syspackagename
(:uses-and-depends :syspackage2 :syspackage3 ...)
(:exports :symbol1 :symbol2))
It would have the following effects:
1. define a package called syspackagename
2. define and install an asdf system called syspackagename
3. automatically load the dependent syspackages
4. export the appropriate symbols.
The result would be that, at the REPL, (use-and-depends :syspackagename)
would automatically do whatever was necessary to load the package. And
to define the package, you wouldn't need anything besides that
(defsyspackage form). Is this possible? Or is it impossible? Maybe it
can't be done, because asdf uses multiple files and macros are
restricted to expanding within a single file. Or maybe (load ...) is one
of the prices you have to pay for macros or some other lisp-specific
feature, perhaps because of the distinctions between compile-time,
macroexpanstion-time, runtime, etc. I'm stumbling here...
all the best,
alexis
alexis gallagher <······@alexisgallagher.com> writes:
> [...]
> But this is my real point -- I suspect the administrative burden of
> CL's system is not the necessary price of its flexibility. It could be
> fixed. Pascal Bourguignon seemed confident that if he knew python, he
> could write a CL equivalent to the Python "import" operator. If this
> is so, then that seems to prove these complexities are not the
> unavoidable in CL.
The question is whether you want the Python import semantics, of the
Modula-2 module semantics, or Ada packages semantics, or Modula-3's,
or some other?
> In fact, I'd dearly like to fix them just for my own convenience, but
> I know nothing compared to you guys, so I can't know for sure if it's
> possible. So I sincerely ask the wizards of this list, is there a
> fundamental reason it would be impossible to write a macro such as the
> following ---
>
> A single file "syspackagename.lisp" would have this as its first form:
>
> (defsyspackage :syspackagename
> (:uses-and-depends :syspackage2 :syspackage3 ...)
> (:exports :symbol1 :symbol2))
>
> It would have the following effects:
> 1. define a package called syspackagename
> 2. define and install an asdf system called syspackagename
> 3. automatically load the dependent syspackages
> 4. export the appropriate symbols.
My DEFINE-PACKAGE does 1, 3 and 4.
I generate a ASD file automatically from the dependencies described in
these DEFINE-PACKAGE with:
(com.informatimago.common-lisp.make-depends:generate-asd
:com.informatimago.common-lisp sources source-type
:VERSION "1.0.0"
:implicit-dependencies '("package"))
cvs checkout instructions at:
http://www.informatimago.com/develop/lisp/index.html
You could define your own DEFSYSPACKAGE macro to do the four points.
Note however that ASDF systems are usually composed of several CL
packages. You could put each of your CL package in one ASDF system
(taking as axiom that 1 source file = 1 package = 1 system), but it
may not be the best way to do it. Or perhaps yes, YMMV. That's the
reason why the Common Lisp standard doesn't specifies anything in this
respect.
The main question would be to develop a concensus on what features are
needed for a well integrated and automatic Common Lisp module system.
--
__Pascal Bourguignon__ http://www.informatimago.com/
ADVISORY: There is an extremely small but nonzero chance that,
through a process known as "tunneling," this product may
spontaneously disappear from its present location and reappear at
any random place in the universe, including your neighbor's
domicile. The manufacturer will not be responsible for any damages
or inconveniences that may result.
alexis gallagher wrote:
>> Common Lisp was never intended to make simple tasks simple at the cost
>> of making hard things even harder. It's true that this creates some
>> steep learning curves.
>>
>
> Here, however, I think you misunderstand me...
>
> I am entirely in favor of making hard things easy even if it requires a
> high upfront _conceptual_ cost to learn a system. But my complaint about
> the CL packaging system (meaning, DEFPACKAGE + asdf) is that it seems to
> impose a _permanent administrative_ cost. Even if you understand the
> ideas perfectly (I'm confident Peter Norvig and David Moon aren't
> complaining because they don't "get it"), then you are still forced to
> manage package loading order, to manage packagename-to-pathname
> mappings, or to manage a minimum of three files per package. This is a
> lot of messiness just to keep your code tidy.
I typically have more than three files per library. It's better to have
coarse-grained packages, so I have one file that defines a package
(maybe two or three), and then a number of files that "implement" that
package, plus a system definition. This is a workable solution in my
experience. In my Closer to MOP library, this pays off because I can
cleanly separate the implementation-dependent parts from the portable code.
> Java copied CL's memory management. Python copied the REPL. Ruby copied
> lambda's through its blocks. What did ocaml and Haskell copy? I don't
> know, but let's concede they copied something. It certainly seems
> suggestive that _none_ of these languages copied CL's packaging system,
> and that all of them automatically manage package loading order and the
> packagename-to-pathname mapping.
Modula-2 and Modula-3 have a notion of separating definition modules
from implementation modules. I am pretty sure that I have seen other
languages copying that idea, but I don't remember the details (probably
some Pascal-dialects). Add the need for a make file, and you have the
same number of required files per library (but not the same flexibility).
BTW, I think there are several issues mixed up that need to be disentangled:
a) Whether you want to keep "definition/declaration" separate from
"implementation" or not.
b) Whether you want to keep definition of relationships between system
parts separate from the system parts or not.
c) Whether you prefer mapping from names to symbols or mappings from
symbols to concepts.
Wrt a) and b), it seems to me that in general, "separation of concerns"
gives you a better handle at manipulating things independently from each
other, but of course also imposes the burden to keep things synchronized
if that's what's needed.
Wrt c), I sometimes have the impression that I am the only one on this
planet who thinks that CL's package system is the right thing (tm). ;)
> In fact, I'd dearly like to fix them just for my own convenience, but I
> know nothing compared to you guys, so I can't know for sure if it's
> possible. So I sincerely ask the wizards of this list, is there a
> fundamental reason it would be impossible to write a macro such as the
> following ---
Why don't you just try it? You can learn a few things by doing this. The
basic building blocks are all there, so it's "just" a matter to combine
them in the right way. If you encounter any problems, come back and ask
for help...
Pascal
--
3rd European Lisp Workshop
July 3-4 - Nantes, France - co-located with ECOOP 2006
http://lisp-ecoop06.bknr.net/
From: Peter Seibel
Subject: Re: packages, load paths and environment variables
Date:
Message-ID: <m2hd5agl38.fsf@gigamonkeys.com>
Pascal Costanza <··@p-cos.net> writes:
> I sometimes have the impression that I am the only one on this
> planet who thinks that CL's package system is the right thing (tm).
> ;)
I largely agree with you Pascal. The key, as you say, is to realize
what problem the package system is trying to solve and which ones it's
not. The only problem is once you realize that the pacakge system is
*not* trying to solve the problems that a "module" system might try to
solve, such as keeping code in one module from redefining a function
originially defined in another module, you start to wish that Common
Lisp also had a module system.
-Peter
--
Peter Seibel * ·····@gigamonkeys.com
Gigamonkeys Consulting * http://www.gigamonkeys.com/
Practical Common Lisp * http://www.gigamonkeys.com/book/
Hi,
Peter Seibel wrote:
> I largely agree with you Pascal. The key, as you say, is to realize
> what problem the package system is trying to solve and which ones it's
> not.
I must admit I'm a Lisp newbie, but for me the package system is there
only to provide some kind of scope for symbols. Not more, not less...
For everything else - "package" definition the python way - I use ASDF:
(asdf:operate 'asdf:loap-op 'my-system)
is the same for me as
import my-package
in Python.
Sascha
"Peter Seibel" <·····@gigamonkeys.com> wrote
> Pascal Costanza <··@p-cos.net> writes:
>
>> I sometimes have the impression that I am the only one on this
>> planet who thinks that CL's package system is the right thing (tm).
>> ;)
>
> I largely agree with you Pascal. The key, as you say, is to realize
> what problem the package system is trying to solve and which ones it's
> not. The only problem is once you realize that the pacakge system is
> *not* trying to solve the problems that a "module" system might try to
> solve, such as keeping code in one module from redefining a function
> originially defined in another module, you start to wish that Common
> Lisp also had a module system.
I also like the package system so we are at least 3 now ;-).
IMO, the possibility to redefine functions from another "module" is very
useful. At the very least to add new methods to generic functions. BTW some
implementations allows you to signal an error or a warning when you redefine
a function in a package.
Marc
From: Peter Seibel
Subject: Re: packages, load paths and environment variables
Date:
Message-ID: <m2d5fxh61u.fsf@gigamonkeys.com>
"Marc Battyani" <·············@fractalconcept.com> writes:
> "Peter Seibel" <·····@gigamonkeys.com> wrote
>> Pascal Costanza <··@p-cos.net> writes:
>>
>>> I sometimes have the impression that I am the only one on this
>>> planet who thinks that CL's package system is the right thing (tm).
>>> ;)
>>
>> I largely agree with you Pascal. The key, as you say, is to realize
>> what problem the package system is trying to solve and which ones it's
>> not. The only problem is once you realize that the pacakge system is
>> *not* trying to solve the problems that a "module" system might try to
>> solve, such as keeping code in one module from redefining a function
>> originially defined in another module, you start to wish that Common
>> Lisp also had a module system.
>
> I also like the package system so we are at least 3 now ;-). IMO,
> the possibility to redefine functions from another "module" is very
> useful. At the very least to add new methods to generic functions.
That's sort of my point. If I define a generic function and export its
name, chances are I fully intend for "other" code (where "other" code
means code that would be in another module if we had modules) to be
able to use that name in a DEFMETHOD in order to extend my GF. But if
I define a regular function and export its name, while I probably
intend for other code to be able to call the function, I probably
*don't* want other code to be able to use the name in a DEFUN and
clobber my definition.
> BTW some implementations allows you to signal an error or a warning
> when you redefine a function in a package.
Yes. That is probably the best solution we have to this particular
problem. Which, somewhat unfortunately, pushes the package system in
the direction of being a module system and thus makes people expect it
to be something it's really not.
-Peter
--
Peter Seibel * ·····@gigamonkeys.com
Gigamonkeys Consulting * http://www.gigamonkeys.com/
Practical Common Lisp * http://www.gigamonkeys.com/book/
Peter Seibel wrote:
>
> Yes. That is probably the best solution we have to this particular
> problem. Which, somewhat unfortunately, pushes the package system in
> the direction of being a module system and thus makes people expect it
> to be something it's really not.
>
Okay, I'll bite. If the package system is not trying to be a module
system, then what is it trying to be? Or in other words, how do you
define a "package system" as opposed to a "module system"?
I've been meditating on the distinctions Pascal Constanza introduced,
and my ideas haven't settled yet. I am confused about what useful thing
the CL package system (excluding asdf) buys you that a module system
does not, which is why I interpreted it as a broken module system.
alexis
Alexis Gallagher <······@alexisgallagher.com> writes:
> Peter Seibel wrote:
> >
> > Yes. That is probably the best solution we have to this particular
> > problem. Which, somewhat unfortunately, pushes the package system in
> > the direction of being a module system and thus makes people expect it
> > to be something it's really not.
> >
>
> Okay, I'll bite. If the package system is not trying to be a module
> system, then what is it trying to be?
A name space system. And at that, it works very well indeed.
> Or in other words, how do you define a "package system" as opposed
> to a "module system"?
The key thing to recognize here is that a name system deals in
"denoters" while a module system deals in the things denoted.
Typically a module system needs a name space system wired into it, and
what's more, visibility rules often end up being conflated with
accessibility rules.
> I am confused about what useful thing the CL package system
> (excluding asdf) buys you that a module system does not, which is
> why I interpreted it as a broken module system.
A module system adds another layer of semantics - which vary quite a
bit among languages that provide them.
/Jon
--
'j' - a n t h o n y at romeo/charley/november com
Alexis Gallagher <······@alexisgallagher.com> writes:
> Peter Seibel wrote:
>>
>> Yes. That is probably the best solution we have to this particular
>> problem. Which, somewhat unfortunately, pushes the package system in
>> the direction of being a module system and thus makes people expect it
>> to be something it's really not.
>>
>
> Okay, I'll bite. If the package system is not trying to be a module
> system, then what is it trying to be? Or in other words, how do you
> define a "package system" as opposed to a "module system"?
A namespace system.
(defpackage :n (:use) (:export :f))
(defun n:f () "f in n")
(print (n:f))
is equivalent to C++:
class n {static const char* f(void){return"f in n";}};
printf("\n%s",n:f());
(Hint: the colon!)
> I've been meditating on the distinctions Pascal Constanza introduced,
> and my ideas haven't settled yet. I am confused about what useful
> thing the CL package system (excluding asdf) buys you that a module
> system does not, which is why I interpreted it as a broken module
> system.
It's unfortunate that the terminology is not universal.
unit, package, module, namespace, class, system, all these words have
different definitions and overloaded semantics for different languages.
:-(
--
__Pascal Bourguignon__ http://www.informatimago.com/
PUBLIC NOTICE AS REQUIRED BY LAW: Any use of this product, in any
manner whatsoever, will increase the amount of disorder in the
universe. Although no liability is implied herein, the consumer is
warned that this process will ultimately lead to the heat death of
the universe.
Pascal Bourguignon <······@informatimago.com> writes:
> class n {static const char* f(void){return"f in n";}};
To be precise, you either need to replace "class" with "struct" or
insert a "public" access specifier before declaring f() above if you
want f() to an externally accessible name.
> printf("\n%s",n:f());
You meant n::f, right?
^^
--
Steven E. Harris
From: Peter Seibel
Subject: Re: packages, load paths and environment variables
Date:
Message-ID: <m27j64hgrj.fsf@gigamonkeys.com>
Alexis Gallagher <······@alexisgallagher.com> writes:
> Peter Seibel wrote:
>>
>> Yes. That is probably the best solution we have to this particular
>> problem. Which, somewhat unfortunately, pushes the package system in
>> the direction of being a module system and thus makes people expect it
>> to be something it's really not.
>>
>
> Okay, I'll bite. If the package system is not trying to be a module
> system, then what is it trying to be? Or in other words, how do you
> define a "package system" as opposed to a "module system"?
The package system is, as others have pointed out, a namespace system.
In particular it allows you to control the mapping between strings and
names-with-object-identity, i.e. symbols.
To elaborate a bit on that, let me describe some *other* possible
solutions to this problem.
A completely trivial namespace system is to put everything in a global
namespace. This has the advantage that you can tell just by looking at
a name in a piece of code whether it's the same name as another name
in another piece of code; if they are spelled the same they're the
same, if not, not.
An almost equally trivial system would be to provide some way to
create multiple completely independent namespaces. In such a system
two names spelled "FOO" might be the same (in the same namespace) or
different (in different namespaces). However if the namespaces are
completely independent then the world is partitioned--there's no way
in one namespace to refer to names that belong to another namespace.
So you could, at the cost of a tiny bit more complexity in your
namespace system, allow such references by adding some syntax for
expressing fully qualified names, (which probably entails giving
namespaces their own names.) Now we're getting closer to something
like the Common Lisp package system. You can have two namespaces,
named BAR and BAZ, and in the BAR namespace you can refer to the name
FOO in the BAZ namespace as BAZ::FOO. Of course this introduces the
complication that now there are multiple spellings for the same name:
In the BAR namespace, the names FOO and BAR::FOO are the same.
You could stop here but it would probably be tedious to have fully
qualify every name that was defined in a different namespace. So
pretty soon you'll want a way to select certain names from other
namespaces and allow them to be referred to without qualification in
your own namespace.
Go down that path a ways, trying to balance the needs of being able to
share names from different namespaces and resolve potential conflicts
between names and you'll end up, I suspect, with something very much
like the Common Lisp package system. Which is why folks like Pascal
say it is in fact The Right Thing.
By a module system, on the other hand, I mean a system that allows
yout to control not how names are spelled but how they may be used. A
module system, for instance, might allow you to say that a function
named FOO could be called from code belonging to a different module
but the variable named FOO could not. Or that the function FOO could
be called but not defined. Or any of a number of other things. However
a module system can only be expressed in terms of things, not their
names, is the things whose manipulation you want to control and the
ways they can be manipulated is determined by what they are, not by
their names. (E.g. a function can be defined, redefined, and called
while a variable can be defined, assigned to, bound, and referenced.)
> I've been meditating on the distinctions Pascal Constanza
> introduced, and my ideas haven't settled yet. I am confused about
> what useful thing the CL package system (excluding asdf) buys you
> that a module system does not, which is why I interpreted it as a
> broken module system.
Well, the thing that's nice about the package system is similar to the
thing that's nice about s-expressions--it's code/data neutral. Just as
you can use s-expressions to represent any tree structure, including
but not limited to those tree structures that happen to be legal
Common Lisp programs, you can use the package system to control the
identity of names independent of whether those names are being used to
name code elements such as functions, variables, classes, etc. or
simply as pieces of data.
-Peter
--
Peter Seibel * ·····@gigamonkeys.com
Gigamonkeys Consulting * http://www.gigamonkeys.com/
Practical Common Lisp * http://www.gigamonkeys.com/book/
Peter Seibel wrote:
> If I define a generic function and export its
> name, chances are I fully intend for "other" code (where "other" code
> means code that would be in another module if we had modules) to be
> able to use that name in a DEFMETHOD in order to extend my GF. But if
> I define a regular function and export its name, while I probably
> intend for other code to be able to call the function, I probably
> *don't* want other code to be able to use the name in a DEFUN and
> clobber my definition.
I'd say this is one of those features that one shouldn't use on a
regular basis, but can be very useful when you really need it. So it's
good that it's possible, IMHO.
Pascal
--
3rd European Lisp Workshop
July 3-4 - Nantes, France - co-located with ECOOP 2006
http://lisp-ecoop06.bknr.net/
On 4593 September 1993, Alexis Gallagher wrote:
> Do not get your checked. You are not going crazy. IMHO, you have
> stumbled onto one of the dirty little secrets of Common Lisp -- the
> package system is quite broken. (Well, "broken" is putting it
> strongly, but you get the idea.)
I beg to differ. It's not easy to understand and it's not what you
might expect from other languages such as Perl or Python, but it does
what it was intended to do. I agree that things can get quite hairy at
times, though.
> 3) Split every package into three files, one containing the actual
> code (foo.lisp), one containing its DEFPACKAGE form
> (foo-package.lisp), and a third containing its asdf DEFSYSTEM
> declaration (foo.asd).
There is really no reason to separate package from source code files,
although it is good practice to do so.
> This new, third file will contain an asdf DEFSYSTEM form. This is
> another package-like abstraction, layered on top of DEFPACKAGE (which
> is itself layered over MAKE-PACKAGE, PROVIDE, and other deprecated
> horrors we pass over in silence).
System-definitons and packages are two totally separate issues
although one could easily imagine integrating them (no make that
"having them integrated", imagining how the integration could happen
is beyond me right now).
> On the other hand, once you sign on with asdf, you do get other good
> things. It's the same system that will make your package
> asdf-installable, so that will make it very easy to distribute it and
> its dependencies over the internet. Some have commented that asdf is
> more than just a packaging facility, that it's more like Make. I can't
> speak to this myself, but I believe it -- it's very customizable.
Viewing defsystem-facilities (which, btw predate ASDF) as make-style
compilation environments makes a lot of sense, regardless of
customizability. System definitions organize *file*
inter-dependencies, just like make does, whereas packages handle
*namespace issues*, which could but need not be related.
What comes closest to e.g. 'use foo;' in Perl is probably the usage of
features (e.g. provide and require). However, defsystem is really a
much better answer to the question: how do I organize load/compile
order?
> I suspect the clunky Lisp packaging system is the single largest
> source of "friction" inhibiting open source development within the
> existing CL community, because it makes proper modularization and
> standardized code-sharing so hard. I shudder to imagine the dark days
> before asdf.
Before asdf, several defsystem facilities have been around.In the 90s
I happily used MK-DEFSYSTEM (and one other whose name I've forgotten).
I was quite surprised seeing it more or less being replaced by ASDF
when I came back to Lisp this year.
> So what's the solution? What I'm interested in understanding, at the
> moment, is if there's a way to create yet another wrapper that will
> conceal all this horrible complexity. This wrapper would merely mimic
> the easy packaging of lesser langauges. In other words, it would let
> you:
> 1) define a package through a single file, including its export list
> and its imported dependencies
> 2) define package names in one-to-one correspondence with pathnames
> 3) automatically manage loading order
Do you want to suggest that there should be a 1:1 correspondence
between 'modules' (or whatever this system/package/feature-mix may be
named) and files? Or do I interpret to much into your idea?
Holger
--
--- http://www.coling.uni-freiburg.de/~schauer/ ---
Fachbegriffe der Informatik - Einfach erkl�rt
135: Druckertreiber
pure virtual Araber (Martin Neumann)
Holger Schauer <··············@gmx.de> writes:
> There is really no reason to separate package from source code files,
> although it is good practice to do so.
There is one reason. The same why in Modula-2 and Modula-3 you
separate definition modules from implementation modules.
If you don't have macros (which adds another complexity), the
dependencies between modules don't go directly from one implementation
module to another, but actually from one implementation module to
another definition module: to use a CL package from the sources of
another, you only need the package definition of the first (you only
need to know the exported symbols, they don't need to have themselves
a definition yet).
Therefore when you have CL sources with crossed dependencies, you need
to cut off the package definition which don't depend one on the
other.
If you had (defpackage :a (:use :b) (:export :f))
(defpackage :b (:use :a) (:export :g))
you could resolve by removing :B from the use list of A,
and add a (use-package :b) in the sources (implementation module) of A.
Now, in Lisp, dependencies can be more complicated anyways for the
macros, and their need to be loaded (along with the functions they
use) at compilation time.
When you want to use a macro such my DEFINE-PACKAGE, you're actually
imposing restrictions on the dependencies between the various lisp
sources (removing circular dependencies in my case).
When you say that there is no reason to separate the package
definition from the sources, you're saying that there's no reason to
have circular dependencies. Well, sometimes there are reasons, but I
agree that it'd be better to avoid them.
>> So what's the solution? What I'm interested in understanding, at the
>> moment, is if there's a way to create yet another wrapper that will
>> conceal all this horrible complexity. This wrapper would merely mimic
>> the easy packaging of lesser langauges. In other words, it would let
>> you:
>> 1) define a package through a single file, including its export list
>> and its imported dependencies
>> 2) define package names in one-to-one correspondence with pathnames
>> 3) automatically manage loading order
>
> Do you want to suggest that there should be a 1:1 correspondence
> between 'modules' (or whatever this system/package/feature-mix may be
> named) and files? Or do I interpret to much into your idea?
One may want to define "module" in such a way. 1) & 2) clearly impose
a 1:1 correspondance between modules and files.
Alexis, you'll have to write your own module system to apply these rules.
(or just use my DEFINE-PACKAGE which implements these three rules).
--
__Pascal Bourguignon__ http://www.informatimago.com/
This universe shipped by weight, not volume. Some expansion may have
occurred during shipment.
On 4599 September 1993, Pascal Bourguignon wrote:
> Holger Schauer <··············@gmx.de> writes:
>> There is really no reason to separate package from source code files,
>> although it is good practice to do so.
> There is one reason. The same why in Modula-2 and Modula-3 you
> separate definition modules from implementation modules.
[...]
> When you say that there is no reason to separate the package
> definition from the sources, you're saying that there's no reason to
> have circular dependencies. Well, sometimes there are reasons, but I
> agree that it'd be better to avoid them.
Your points, of course, are well-taken. But I was arguing against an
inherent requirement that using packages necessarily involved a
separate file for the package definition. For simple cases, there is
no such requirement, although for complex ones there might be one, as
you rightly point out.
> One may want to define "module" in such a way. 1) & 2) clearly impose
> a 1:1 correspondance between modules and files.
I would rather stick with packages and defsystem then.
Holger
--
--- http://www.coling.uni-freiburg.de/~schauer/ ---
Fachbegriffe der Informatik - Einfach erkl�rt
135: Druckertreiber
pure virtual Araber (Martin Neumann)