From: Peter Seibel
Subject: First draft of chapter 17 of Practical Common Lisp on web
Date:
Message-ID: <m3bril1804.fsf@javamonkey.com>
The topic is packages and as always, I'm interested in any feedback
folks have, good, bad, or ugly. This chapter is at:
<http://www.gigamonkeys.com/book/programming-in-the-large-packages-and-symbols.html>
while the main entry point is at:
<http://www.gigamonkeys.com/book/>
While I'm happy to get reports of typos and such, the best kinds of
comments are the ones about things that confused you or that were
particularly helpful. Also if there are topics you feel weren't
covered sufficiently I'd love to hear about that too. If you are
already an experienced Lisper feel free to try to channel some smart
but not-yet-Lisp-aware programmer you know. Or send them the URL to
look at themselves if they have any interest in Lisp. This goes for
all the chapters I've marked as "Ready for review".
-Peter
--
Peter Seibel ·····@javamonkey.com
Lisp is the red pill. -- John Fraser, comp.lang.lisp
Peter,
Very nice chapter. Packages are a confusing topic for many folks. I
thought you described shadowing very well. I understood it before,
but now it makes even more sense.
First, the type-o's:
First paragraph: type-o
"need only discuss on of the more immediate"
on -> one
End of fourth paragraph: missing word
"to map the name to symbol" -> "to map the name to a symbol"
First paragraph of "Standard Packages" section: duplicate word
"COMMON-LISP exports the the names of all"
Second paragraph of "Package Mechanics": duplicate word
"package must be defined before before we"
Towards the end: sentence ends prematurely
"... such as ASDF which is discussed in Appendix TK to manage"
Now, a few ideas:
It may be worth covering the non-declarative approach to packages (ie.
#'import, #'export, #'find-package, #'make-package, #'use-package).
The REPL is so powerful and I know I spend a lot of time just sitting
at the REPL experimenting with things. In those situations it is
really useful to be able to manually import, export (and unexport)
symbols. I would say use-package is especially useful from the REPL
as a way of experimenting with other packages.
How about #'intern and #'unintern. I especially find #'unintern
useful at the REPL when I've accidentally interned some symbol and now
I can't #'use-package a package.
I really do think the non-declarative approach is useful just for it's
value at the REPL.
Great book. I look forward to owning a copy.
Justin Dubs
From: Raistlin Magere
Subject: Re: First draft of chapter 17 of Practical Common Lisp on web
Date:
Message-ID: <cd07e1$dfe$1@news.ox.ac.uk>
Peter Seibel wrote:
> While I'm happy to get reports of typos and such, the best kinds of
> comments are the ones about things that confused you or that were
> particularly helpful. Also if there are topics you feel weren't
> covered sufficiently I'd love to hear about that too.
Peter, I really enjoyed reading this chapter and I think I will now try to
actually use packages in my code - as I feel that not only the help in
avoiding name conflicts but in clarifing to the programmer the relationships
between different parts of his code. I am using acl so I have only been
(in-package :cg-user) and in the program list I just made sure that all my
file got loaded in the right order.
I think it would be helpful if you could cover in a bit more detailed the
relationship between clos and packages.
For example if you create a new class within a package and want to make it
accessable outside the package can you just export the class name or do you
need to export all the accessors you want to make available?
Similarly when dealing with generic functions is there a way to export only
certain methods or does the entire generic function need to be exported;
e.g. let's say I generate the class box, parallelogram and circle and the gf
area which works on all of them (or possibly I have a gf which works on
combinations of them) then I could want to:
a) export box, parallelogram and circle but only the area methods for
parallelogram and circle
b) export parallelogram and circle, not box, and still only want to export
the gf area only for those two classes.
And another point: is there a way to export a function with a different name
than the one used within the package (let's say I use v. non-descriptive
names within the package but want to provide the users of the package with
better names) or to import it with a different name.
Also you might want to mention the nicknames option as you do say that
"common-lisp" can be reffered to as "cl" and that might incentivate lazy
typist to use long-meaningful package names as they now they can also
provide short forms for them.
On Tue, 13 Jul 2004, Raistlin Magere wrote:
> For example if you create a new class within a package and want to make it
> accessable outside the package can you just export the class name or do you
> need to export all the accessors you want to make available?
>
> Similarly when dealing with generic functions is there a way to export only
> certain methods or does the entire generic function need to be exported;
> [...]
You are misunderstanding the purpose of packages. Packages, in Common
Lisp, are for namespacing symbols and ONLY symbols. The package system
has nothing to do with the functions or variables which may be named by
symbols. If you understand this, then you will know why your question is
meaningless.
From: Raistlin Magere
Subject: Re: First draft of chapter 17 of Practical Common Lisp on web
Date:
Message-ID: <cd13l5$oou$1@news.ox.ac.uk>
Matthew Danish wrote:
>> Similarly when dealing with generic functions is there a way to
>> export only certain methods or does the entire generic function need
>> to be exported; [...]
>
> You are misunderstanding the purpose of packages. Packages, in Common
> Lisp, are for namespacing symbols and ONLY symbols. The package
> system has nothing to do with the functions or variables which may be
> named by symbols. If you understand this, then you will know why
> your question is meaningless.
I do appreciate that because you export the symbol `area' you are exporting
the
whole thing essentially and there is no way for the package tool to even
attempt to
understand which methods compose it and therefore as you correctly say it is
kind
of meaningless to ask that (like I expect it is kind of meaningless to hope
for a way
to export a class and all its accessor in one go - so in order to be able
create new
object belonging to class foo do I need to export the function:
(defun make-foo (blah1 blah2)
(make-instance 'foo :blah1 blah1 :blah2 blah2)).
However Peter asked for "the best kinds of comments are the ones about
things
that confused you or that were particularly helpful." and I personally felt
that although
the chapter explained quite well the general concept behind packages it
would have
benefited from a section about "Packages and CLOS".
I just kind of feel that unless most things get well explained about
packages newbie
(and I include myself in there) will not use them.
Raistlin Magere wrote:
> Matthew Danish wrote:
>
>>>Similarly when dealing with generic functions is there a way to
>>>export only certain methods or does the entire generic function need
>>>to be exported; [...]
>>
>>You are misunderstanding the purpose of packages. Packages, in Common
>>Lisp, are for namespacing symbols and ONLY symbols. The package
>>system has nothing to do with the functions or variables which may be
>>named by symbols. If you understand this, then you will know why
>>your question is meaningless.
>
>
> I do appreciate that because you export the symbol `area' you are exporting
> the
> whole thing essentially and there is no way for the package tool to even
> attempt to
> understand which methods compose it and therefore as you correctly say it is
> kind
> of meaningless to ask that
More like it's meaningless to ask it because there are no methods which
'compose' the symbol 'area'; there is just the symbol. Or, more
accurately, there is some symbol SOME-PACKAGE::AREA (and maybe there are
other symbols ANOTHER-PACKAGE::AREA and YET-ANOTHER-PACKAGE::AREA).
These symbols are not 'composed of' methods or classes or anything else;
they are just symbols.
It so happens that symbols are used in Common Lisp to name things,
including functions and classes and methods, but it is the symbols that
are collected in packages, not the classes or the methods. Packages
collect symbols and ONLY symbols. You cannot export a class or a method
at all (and likewise there is no such thing as a private or internal
class or method); you can only export (or decline to export) symbols.
From: Raistlin Magere
Subject: Re: First draft of chapter 17 of Practical Common Lisp on web
Date:
Message-ID: <cd1bu7$rjc$1@news.ox.ac.uk>
mikel wrote:
> These symbols are not 'composed of' methods or classes or anything
> else; they are just symbols.
>
> It so happens that symbols are used in Common Lisp to name things,
> including functions and classes and methods, but it is the symbols
> that are collected in packages, not the classes or the methods.
> Packages collect symbols and ONLY symbols. You cannot export a class
> or a method at all (and likewise there is no such thing as a private
> or internal class or method); you can only export (or decline to
> export) symbols.
Ok I think I get it - though I suspect that I will need to play around with
it
(even just a little bit) to fully grasp its import.
What I find quite interesting is that you used the idea of private/internal
class
vs public/external class (which I guess comes from Java and the like) and
that
seems to be closer to what I kind of thought was going on.
However I have never programmed in anything else but Common Lisp (ok
when I was v.young I played around with TurboPascal 3.0 a little) I just
never
happend to need to use packages so what I am wondering is only if for once
Lisp has chosen a not as natural way of implementing packages (at least with
respect to CLOS) than other languages? Were packages developed before clos?
Sorry for the rambling - they are just some random thoughts.
Raistlin Magere wrote:
> mikel wrote:
>
>>These symbols are not 'composed of' methods or classes or anything
>>else; they are just symbols.
>>
>>It so happens that symbols are used in Common Lisp to name things,
>>including functions and classes and methods, but it is the symbols
>>that are collected in packages, not the classes or the methods.
>>Packages collect symbols and ONLY symbols. You cannot export a class
>>or a method at all (and likewise there is no such thing as a private
>>or internal class or method); you can only export (or decline to
>>export) symbols.
>
>
> Ok I think I get it - though I suspect that I will need to play around with
> it
> (even just a little bit) to fully grasp its import.
> What I find quite interesting is that you used the idea of private/internal
> class
> vs public/external class (which I guess comes from Java and the like) and
> that
> seems to be closer to what I kind of thought was going on.
> However I have never programmed in anything else but Common Lisp (ok
> when I was v.young I played around with TurboPascal 3.0 a little) I just
> never
> happend to need to use packages so what I am wondering is only if for once
> Lisp has chosen a not as natural way of implementing packages (at least with
> respect to CLOS) than other languages? Were packages developed before clos?
Yes.
As a whole, Common Lisp doesn't try very hard to provide facilities for
walling off parts of a program from others. In part I think this is a
feature of its history as an exploratory language. Walling off part of a
program inhibits your ability to explore--to refactor and change things
dynamically as the program runs.
This was a subject of debate during the design of the Dylan language at
Apple; the informal committee of people contributing to the design
included basically two camps: Lisp people and Smalltalk people. The
Smalltalk folks really wanted some way to seal off implementation
details. The result of those debates was the Dylan module system, which
provides an arguably cleaner architecture for modularity than the Common
Lisp package system.
The package system is extremely simple, though, once you get what it
actually does: it makes separate namespaces for *symbols*, period.
From: Raistlin Magere
Subject: Re: First draft of chapter 17 of Practical Common Lisp on web
Date:
Message-ID: <cd5gkv$d9g$1@news.ox.ac.uk>
"mikel" <·····@evins.net> wrote in message
··························@newssvr29.news.prodigy.com...
>
> The package system is extremely simple, though, once you get what it
> actually does: it makes separate namespaces for *symbols*, period.
Thanks for your reply, I actually got the idea across when I relized that if
I have
a symbol with both a function value and a variable value attached to it I
can only
export the both of them or better I can only export the `name' and
everything that
is called that way.
I guess this implies that when you want to use packages as libraries rather
than
just separate clean environment you have to think harder how you name things
instead of hoping to sort out inconsistences and desires through a careful
package
definition.
I found a good example of Packages and CLOS is in the cl-http library.
In it, in the w4 directory in class.lisp you'll see classes that are
subclassed from classes in another package, namely the multi-threaded
queues, and in walker.lisp you'll see a defmethod that was originally
set up in another package, tq:task-queue-execute-task.
"Raistlin Magere" <·······@*the-mail-that-burns*.com> wrote:
>Matthew Danish wrote:
>>> Similarly when dealing with generic functions is there a way to
>>> export only certain methods or does the entire generic function need
>>> to be exported; [...]
>>
>> You are misunderstanding the purpose of packages. Packages, in Common
>> Lisp, are for namespacing symbols and ONLY symbols. The package
>> system has nothing to do with the functions or variables which may be
>> named by symbols. If you understand this, then you will know why
>> your question is meaningless.
>
>I do appreciate that because you export the symbol `area' you are exporting
>the
>whole thing essentially and there is no way for the package tool to even
>attempt to
>understand which methods compose it and therefore as you correctly say it is
>kind
>of meaningless to ask that (like I expect it is kind of meaningless to hope
>for a way
>to export a class and all its accessor in one go - so in order to be able
>create new
>object belonging to class foo do I need to export the function:
>(defun make-foo (blah1 blah2)
> (make-instance 'foo :blah1 blah1 :blah2 blah2)).
>However Peter asked for "the best kinds of comments are the ones about
>things
>that confused you or that were particularly helpful." and I personally felt
>that although
>the chapter explained quite well the general concept behind packages it
>would have
>benefited from a section about "Packages and CLOS".
>I just kind of feel that unless most things get well explained about
>packages newbie
>(and I include myself in there) will not use them.
>
From: Raistlin Magere
Subject: Re: First draft of chapter 17 of Practical Common Lisp on web
Date:
Message-ID: <cd5gn4$dab$1@news.ox.ac.uk>
"Ralph Richard Cook" <······@bellsouth.net> wrote in message
·····················@newsgroups.bellsouth.net...
> I found a good example of Packages and CLOS is in the cl-http library.
>
> In it, in the w4 directory in class.lisp you'll see classes that are
> subclassed from classes in another package, namely the multi-threaded
> queues, and in walker.lisp you'll see a defmethod that was originally
> set up in another package, tq:task-queue-execute-task.
>
Thanks for the hint, I will have a look at it.
In article <··············@javamonkey.com>, Peter Seibel wrote:
> The topic is packages and as always, I'm interested in any feedback
> folks have, good, bad, or ugly. This chapter is at:
>
> <http://www.gigamonkeys.com/book/programming-in-the-large-packages-and-symbols.html>
>
> while the main entry point is at:
>
> <http://www.gigamonkeys.com/book/>
>
> While I'm happy to get reports of typos and such, the best kinds of
> comments are the ones about things that confused you or that were
> particularly helpful. Also if there are topics you feel weren't
> covered sufficiently I'd love to hear about that too. If you are
> already an experienced Lisper feel free to try to channel some smart
> but not-yet-Lisp-aware programmer you know. Or send them the URL to
> look at themselves if they have any interest in Lisp. This goes for
> all the chapters I've marked as "Ready for review".
>
> -Peter
>
You may want to emphasise the differences between Lisp packages
and package or module systems in other languages. In particular
since a symbol can be used in several contexts (variable,
function, symbol eq-ness, etc.), you can't export only one "use".
One often-recurring piece of advice in this newsgroup is that
packages weren't designed for access control. I think this deserves
being spelled out.
Also you should consider printing out examples of package-qualified names,
for example COM.ACME.TEXT:SAVE. This may seem silly but I think it would
help the learning process.
Eric Daniel
Peter Seibel <·····@javamonkey.com> wrote in message news:<··············@javamonkey.com>...
> The topic is packages and as always, I'm interested in any feedback
> folks have, good, bad, or ugly. This chapter is at:
>
> <http://www.gigamonkeys.com/book/programming-in-the-large-packages-and-symbols.html>
>
> while the main entry point is at:
>
> <http://www.gigamonkeys.com/book/>
>
> While I'm happy to get reports of typos and such, the best kinds of
> comments are the ones about things that confused you or that were
> particularly helpful. Also if there are topics you feel weren't
> covered sufficiently I'd love to hear about that too. If you are
> already an experienced Lisper feel free to try to channel some smart
> but not-yet-Lisp-aware programmer you know. Or send them the URL to
> look at themselves if they have any interest in Lisp. This goes for
> all the chapters I've marked as "Ready for review".
>
> -Peter
The information which is there in the chapter is excellent no doubt.
But I have some comments, please consider if they are worthful.
Is this is the only chapter which covers the symbols?
If yes, It seems that here you explained less about symbols.
As the symbols are unique feature of lisp, it is reasonable to have
more explanation about symbols.
It is good if you provide information about how to iterate through
symbols of package and the packages in the environment.
--Venkat
I was surprised to read:
Note, however, that the packages do not exist
until we LOAD the package definitions, either the
source or the .fasl files produced by
COMPILE-FILE. Thus if we are compiling everything
we must still LOAD all the package definitions
before we can before we can COMPILE-FILE any
files that are to be read in the packages.
The hyperspec dictionary entry for defpackage says:
If a defpackage form appears as a top level form, all of
the actions normally performed by this macro at load
time must also be performed at compile time.
I take this to mean that we have merely to compile our files
in the correct order, so that the compiler sees the
defpackage before the in-package. There is no need to use
load on the package definitions when compiling.
Alan Crowe
Edinburgh
Scotland
From: Peter Seibel
Subject: Re: First draft of chapter 17 of Practical Common Lisp on web
Date:
Message-ID: <m3fz7pje1o.fsf@javamonkey.com>
Alan Crowe <····@cawtech.freeserve.co.uk> writes:
> I was surprised to read:
>
> Note, however, that the packages do not exist
> until we LOAD the package definitions, either the
> source or the .fasl files produced by
> COMPILE-FILE. Thus if we are compiling everything
> we must still LOAD all the package definitions
> before we can before we can COMPILE-FILE any
> files that are to be read in the packages.
>
> The hyperspec dictionary entry for defpackage says:
>
> If a defpackage form appears as a top level form, all of
> the actions normally performed by this macro at load
> time must also be performed at compile time.
>
> I take this to mean that we have merely to compile our files in the
> correct order, so that the compiler sees the defpackage before the
> in-package. There is no need to use load on the package definitions
> when compiling.
As I understand it that depends on whether in your implementation the
compile-time environment is the same as the runtime environment. The
actions of a DEFPACKAGE that occur at compile time are guaranteed to
affect the compile time environment. Thus if within a single file you
have a (defpackage :foo ...) and a subsequent (in-package :foo) it
will work. But if you have a (defpackage :foo) in one file and an
(in-package :foo) in another there is no guarantee that the
compile-time effects of the DEFPACKAGE will affect the environment in
which you LOAD or COMPILE-FILE the second file. (In other words each
call to COMPILE-FILE may get its own "compile-time environment".) But
in implementations where the compile-time and runtime environments are
the same then it will work. Or I'm confused; always a possibility.
-Peter
--
Peter Seibel ·····@javamonkey.com
Lisp is the red pill. -- John Fraser, comp.lang.lisp
Peter Seibel wrote:
> But if you have a (defpackage :foo) in one file and an
> (in-package :foo) in another there is no guarantee that
> the compile-time effects of the DEFPACKAGE will affect the
> environment in which you LOAD or COMPILE-FILE the second
> file. (In other words each call to COMPILE-FILE may get
> its own "compile-time environment".)
Surely not. If each call to COMPILE-FILE got its own
compile-time environment, you would have to repeat your
macro definitions in each file that needed them.
Aaarrrgghh! The horror:
3.2.3.1.1 Processing of Defining Macros
In particular, the information stored by the defining macros
at compile time might or might not be available ... during
subsequent calls to the compiler. For example, the following
code is nonportable ...
A line from 3.2.1 Compiler Terminology that has long puzzled
me is:
The compilation environment inherits from the
evaluation environment,...
Hmm, if you cannot rely on
(compile-file "source1")
(compile-file "source2")
passing on a macro definition from source1 to source2
what do you do?
Presumably:
(compile-file "source1")
(load-file "source1")
(compile-file "source2")
then the compilation of source2 will use the macro
definition in source1 because the compilation environment
inherits from the evaluation environment.
Have I understood it correctly now?
Alan Crowe
Edinburgh
Scotland
From: Duane Rettig
Subject: Extent of definitions in environments (Was: First draft ...)
Date:
Message-ID: <4658kj9so.fsf_-_@franz.com>
Alan Crowe <····@cawtech.freeserve.co.uk> writes:
> Peter Seibel wrote:
> > But if you have a (defpackage :foo) in one file and an
> > (in-package :foo) in another there is no guarantee that
> > the compile-time effects of the DEFPACKAGE will affect the
> > environment in which you LOAD or COMPILE-FILE the second
> > file. (In other words each call to COMPILE-FILE may get
> > its own "compile-time environment".)
>
> Surely not. If each call to COMPILE-FILE got its own
> compile-time environment, you would have to repeat your
> macro definitions in each file that needed them.
>
> Aaarrrgghh! The horror:
>
> 3.2.3.1.1 Processing of Defining Macros
>
> In particular, the information stored by the defining macros
> at compile time might or might not be available ... during
> subsequent calls to the compiler. For example, the following
> code is nonportable ...
Yes, this is a problem - there are many intuitive responses
that are broken in each programmers mind, and since there is
no portable way to make any guarantees, it is hard to reset
ones intuition to match that which can't be guaranteed.
> (compile-file "source1")
> (compile-file "source2")
Another intuition that is not covered by the spec is that
since compile-file acts as if it is surrounded by a
with-compilation-unit form, it might make sense to extend
the actions for compile-file to whatever the outermost
non-overridden w-c-u is in effect, so that something like
(with-compilation-unit
(compile-file "source1")
(compile-file "source2"))
would "do the wim" thing and keep definitions contained but
availaible to the whole compilation unit.
However, whatever such guarantees you want in this area are
not given by the spec, and would have to come as extensions by
individual vendors (and would thus only make code be portable
between vendors which give the same guarantees).
> passing on a macro definition from source1 to source2
> what do you do?
>
> Presumably:
>
> (compile-file "source1")
> (load-file "source1")
> (compile-file "source2")
>
> then the compilation of source2 will use the macro
> definition in source1 because the compilation environment
> inherits from the evaluation environment.
>
> Have I understood it correctly now?
Close, but not quite. The macro in source1 is probably defined to
return something like:
'(progn
(eval-when (:compile-toplevel)
...)
(eval-when (:load-toplevel :execute)
...))
and so while source1 is being compiled, the definition is made
available in the compilation environment, _and_ the compiler arranges
that when the file is loaded it becomes part of the global environment
of whatever lisp loads it. So although you can't guarantee what the
extent of definitions in the compilation environment is, you can
guarantee that the definition exists by loading the file and thus making
it defined in the global environment.
In Allegro CL, we've always made a slight distinction between these
two styles of definition (at top-level, of course):
`(progn
(eval-when (:compile-toplevel)
,defining-form-body)
(eval-when (:load-toplevel :execute)
,defining-form-body))
and
`(progn
(eval-when (:compile-toplevel :load-toplevel :execute)
,defining-form-body))
in that the first style explicitly keeps the definition in the
compile-file-environment, which makes it only available while the
compile-file is in effect, and the second places the definiton
directly into the global-environment, which makes it available
both to the compile-file and to the lisp afterwards. Version 7.0
also retains this behavior, and I think I've documented it as a
guarantee. (it is actually the combination of :compile-toplevel
or compile with :load-toplevel or load, in the same eval-when
form, which makes the switch to persistence). I also changed most
of the defining macros in 3.2.3.1.1 (except for defpackage) to be
the "split" definititions with :compile-toplevel and :load-toplevel
not in the same eval-when form. So unadorned, a defining form will
have the restricted scope of the compile-file that is required,
and when the defining form is wrapped with an eval-when form that
contains at least :compile-toplevel and :load-toplevel (or the
shorter equivalents), as in
(eval-when (compile load)
(defmacro foo ...))
then the definition will be persistent.
The environments-access extension that I'm working on (and which
I intend to eventually release as open-source) makes this kind of
guarantee easy, because of the way the environments are set up.
So other vendors who then use that package would be easily able,
with concern only with backward compatibility, to offer the same
guarantee (as an extension, of course).
--
Duane Rettig ·····@franz.com Franz Inc. http://www.franz.com/
555 12th St., Suite 1450 http://www.555citycenter.com/
Oakland, Ca. 94607 Phone: (510) 452-2000; Fax: (510) 452-0182
From: Duane Rettig
Subject: Re: Extent of definitions in environments (Was: First draft ...)
Date:
Message-ID: <41xj8j939.fsf@franz.com>
[ Reading over my own post, it will be easy to misunderstand
what it is that I'm disagreeing with, and where I'm going,
so I'm answering myself to extend the explanation]
Duane Rettig <·····@franz.com> writes:
> Alan Crowe <····@cawtech.freeserve.co.uk> writes:
>
> > Presumably:
> >
> > (compile-file "source1")
> > (load-file "source1")
> > (compile-file "source2")
> >
> > then the compilation of source2 will use the macro
> > definition in source1 because the compilation environment
> > inherits from the evaluation environment.
Note here that there is an explicit CL definition for "evaluation
environment", one which is slightly narrower than a description
that might be assumed by construction.
> >
> > Have I understood it correctly now?
>
> Close, but not quite. The macro in source1 is probably defined to
> return something like:
>
> '(progn
> (eval-when (:compile-toplevel)
> ...)
> (eval-when (:load-toplevel :execute)
> ...))
>
> and so while source1 is being compiled, the definition is made
> available in the compilation environment, _and_ the compiler arranges
> that when the file is loaded it becomes part of the global environment
> of whatever lisp loads it. So although you can't guarantee what the
> extent of definitions in the compilation environment is, you can
> guarantee that the definition exists by loading the file and thus making
> it defined in the global environment.
Note that the evaluation environment is one which the compiler
initiates, but by loading the file that has already been compiled,
you are circumventing the compile altogether to ensure that the
definition is present. Thus, the compiler environment (for the
compilation of source2) inherits, not from an evaluation environment,
but from the global environment, which has been augmented by loading
a file.
--
Duane Rettig ·····@franz.com Franz Inc. http://www.franz.com/
555 12th St., Suite 1450 http://www.555citycenter.com/
Oakland, Ca. 94607 Phone: (510) 452-2000; Fax: (510) 452-0182
Thank you for your long and thorough replies. I must confess
that they are going somewhat over my head. I have never
written a macro that expands to an eval-when. My
understanding of when I need to do this is as follows:
If I'm coding, all in one file, I can do
(defmacro do-twice(form)
(list 'progn form form))
(do-twice (print "Ha "))
The compiler creates an evaluation environment with FORM
bound to (print "Ha "). CL:LIST is in the start up
environment, so it is inherited by the evaluation
environment, so macroexpander is able to find the function
it needs to compute the macroexpansion, and everything is OK.
If I decide that do-twice is best written with a helper
function
(defun duplicate (form)
(list form form))
(defmacro do-twice(form)
(cons 'progn (duplicate form)))
(do-twice "Oh dear ")
things will not go well. The compiler will put object code
for DUPLICATE into the output file, but not add it to the
evaluation environment
;;; Compiling file single-file-fancy.lisp
; While compiling (:TOP-LEVEL-FORM "single-file-fancy.lisp" 106):
Error: attempt to call `DUPLICATE' which is an undefined function.
The minimal fix is to use eval-when
(eval-when (:compile-toplevel)
(defun duplicate (form)
(list form form)))
(defmacro do-twice(form)
(cons 'progn (duplicate form)))
(do-twice (print "Oh dear "))
Compiling works because the definition of DUPLICATE went
into the compilers evaluation environment and was available
for expanding the macro.
Loading worked because DUPLICATE wasn't needed at run time,
since compilation guarantees to fully expand all macros.
Alternatively I can start breaking this program up into
multiple files, with
two-file-macro.lisp
(defun duplicate (form)
(list form form))
(defmacro do-twice(form)
(cons 'progn (duplicate form)))
two-file-func.lisp
(do-twice (print "Oh dear "))
Now my build looks like this
(compile-file "two-file-macro.lisp")
(load "two-file-macro")
(compile-file "two-file-func.lisp")
culminating in (load "two-file-func")
My latest understanding is that the second compile-file
works because
1) The load of two-file-macro loaded definitions of
do-twice and duplicate into the Lisp image
2) Thus they are in the startup environment of the second
compile-file
3) The evaluation environment inherits from the startup
environment.
4) The compilation environment inherits from the evaluation
environment.
The big conflict I'm left with is this:
On the one hand, I'm going to be loading my files with my
package definitions and my macro definitions before I
compile the rest of my code. I can forget about
eval-when. Loading takes care of evaluation for me.
On the other hand Duane Rettig is expecting my macros to
expand to
> '(progn
> (eval-when (:compile-toplevel)
> ...)
> (eval-when (:load-toplevel :execute)
> ...))
which makes me worry that I'm still missing a vital part of
the story.
Is the approach I've sketched out adequate for gaining
experience with multi-file fancy-macro Lisp programs, or
will I quickly come to grief for lack of eval-whens?
Alan Crowe
Edinburgh
Scotland
From: Duane Rettig
Subject: Re: Extent of definitions in environments (Was: First draft ...)
Date:
Message-ID: <4wu0zixrh.fsf@franz.com>
Alan Crowe <····@cawtech.freeserve.co.uk> writes:
> Thank you for your long and thorough replies. I must confess
> that they are going somewhat over my head. I have never
> written a macro that expands to an eval-when.
Are you sure?
> My
> understanding of when I need to do this is as follows:
>
> If I'm coding, all in one file, I can do
>
> (defmacro do-twice(form)
> (list 'progn form form))
>
> (do-twice (print "Ha "))
Before we go any farther, try this:
(pprint (macroexpand '(defmacro do-twice(form)
(list 'progn form form))))
What do you see? Are there any defmacro forms that you
can create which have no eval-when forms in them?
(note that a form with no eval-when form surrounding it is
similar in some respects to a form surrounded by
(eval-when (load eval) ...) or by
(eval-when (:load-toplevel :execute) ... )
and so you may actually only see those eval-when forms that
have to do with compilation)
> The compiler creates an evaluation environment with FORM
> bound to (print "Ha "). CL:LIST is in the start up
> environment, so it is inherited by the evaluation
> environment, so macroexpander is able to find the function
> it needs to compute the macroexpansion, and everything is OK.
>
> If I decide that do-twice is best written with a helper
> function
>
> (defun duplicate (form)
> (list form form))
Try the same (pprint (macroexpand ...)) trick on the defun form,
as well. What is missing? What can you add to get the behavior
you want at compile-time?
> (defmacro do-twice(form)
> (cons 'progn (duplicate form)))
>
> (do-twice "Oh dear ")
>
> things will not go well. The compiler will put object code
> for DUPLICATE into the output file, but not add it to the
> evaluation environment
Let's get past this part before we go on...
--
Duane Rettig ·····@franz.com Franz Inc. http://www.franz.com/
555 12th St., Suite 1450 http://www.555citycenter.com/
Oakland, Ca. 94607 Phone: (510) 452-2000; Fax: (510) 452-0182
Duane Rettig asked socratically:
> > If I decide that do-twice is best written with a helper
> > function
> >
> > (defun duplicate (form)
> > (list form form))
>
> Try the same (pprint (macroexpand ...)) trick on the defun form,
> as well. What is missing? What can you add to get the behavior
> you want at compile-time?
In the expansion of
(defmacro do-twice (form)
(list 'progn form form))
the form (LIST 'PROGN FORM FORM) occurrs twice, once within
an eval-when(:compile-toplevel) within which
COMPILER-PUTPROP stores it away for later use, and once
without an eval-when, to be stored as the DO-TWICE's
macro-function at load time.
The expansion for (defun duplicate ...) differs in only
having one copy of the code being specified. The lambda
expression containing it is compiled at compile time (at
least minimally), but the setf that sets the fdefinition is
not in an eval-when so the function does not become
available until the file is loaded.
The expansion for (defun duplicate ...) does have an
eval-when(:compile-toplevel) but it doesn't retain the code
that defined DUPLICATE in the compilers evaluation
environment. It confines itself to checking that DUPLICATE
has not been defined before, and making a note that it has
been now.
To answer the final question I propose Plan B.
The example code I've used before goes in one file. I have a
prelude in another file
plan-b-prelude.lisp
(defmacro define-macro-helper-function (&rest stuff)
`(progn (eval-when(:compile-toplevel ; the point
;; if I want the helper
;; available outside the file that
;; defines it, some implementation
;; will require me to load
;; load the file, so I need
:load-toplevel)
(defun ,@stuff))))
plan-b-main.lisp
(define-macro-helper-function duplicate (form)
(list form form))
(defmacro do-twice(form)
(cons 'progn (duplicate form)))
(do-twice (print "Oh dear "))
Now I can build my software with
(load "plan-b-prelude.lisp")
(compile-file "plan-b-main.lisp")
and run it with (load "plan-b-main").
At last! I've written a macro that expands to an eval-when.
Hmm. I'm not too sure what I've achieved here. In both
Allegro and CMUCL, the compile-file makes the function
DUPLICATE available in the Lisp image. However reading
3.2.3.1 Processing of Top Level Forms:
Evaluate: evaluate the body in the dynamic execution
context of the compiler, using the evaluation
environment as the global environment and the lexical
environment in which the eval-when appears.
so I don't think this is guaranteed. In particular, if I
think that if I want to extend my program portably with a
third file that uses the DO-TWICE macro, I need to load
plan-b-main.fasl before I compile the third file.
From: Duane Rettig
Subject: Re: Extent of definitions in environments (Was: First draft ...)
Date:
Message-ID: <4ekn6em6w.fsf@franz.com>
Alan Crowe <····@cawtech.freeserve.co.uk> writes:
> Duane Rettig asked socratically:
> > > If I decide that do-twice is best written with a helper
> > > function
> > >
> > > (defun duplicate (form)
> > > (list form form))
> >
> > Try the same (pprint (macroexpand ...)) trick on the defun form,
> > as well. What is missing? What can you add to get the behavior
> > you want at compile-time?
>
> In the expansion of
>
> (defmacro do-twice (form)
> (list 'progn form form))
>
> the form (LIST 'PROGN FORM FORM) occurrs twice, once within
> an eval-when(:compile-toplevel) within which
> COMPILER-PUTPROP stores it away for later use, and once
> without an eval-when, to be stored as the DO-TWICE's
> macro-function at load time.
>
> The expansion for (defun duplicate ...) differs in only
> having one copy of the code being specified. The lambda
> expression containing it is compiled at compile time (at
> least minimally), but the setf that sets the fdefinition is
> not in an eval-when so the function does not become
> available until the file is loaded.
Note that the explicit lack of an eval-when invites the wrapping
of one, unencumbered, in open code - there are a couple of rules
that take effect when eval-when forms are nested (still at toplevel),
and those rules need to be understood per the table at 3.2.3.1,
though most of them tend to be intuitive. But because your
defining forms have an eval-when-less component, and because
eval-when forms don't themselves change the top-level-ness of a
form, you can wrap your own eval-when around the entire defining
form, as you have effectively done, below.
> The expansion for (defun duplicate ...) does have an
> eval-when(:compile-toplevel) but it doesn't retain the code
> that defined DUPLICATE in the compilers evaluation
> environment. It confines itself to checking that DUPLICATE
> has not been defined before, and making a note that it has
> been now.
Yes; a defining form may have several eval-when components that
have nothing to do with the actual definition, but which instead
take care of some of the bookkeeping for such tasks as source-file
information, cross-reference info, and redefinition warnings.
> To answer the final question I propose Plan B.
> The example code I've used before goes in one file. I have a
> prelude in another file
>
> plan-b-prelude.lisp
>
> (defmacro define-macro-helper-function (&rest stuff)
> `(progn (eval-when(:compile-toplevel ; the point
> ;; if I want the helper
> ;; available outside the file that
> ;; defines it, some implementation
> ;; will require me to load
> ;; load the file, so I need
> :load-toplevel)
I usually tend to go ahead and add the :evaluate - defensive
programming against the user who loads my file interpreted...
> (defun ,@stuff))))
>
> plan-b-main.lisp
>
> (define-macro-helper-function duplicate (form)
> (list form form))
Yes, this works, but it might have been simpler to just do
it inline:
(eval-when (:compile-toplevel :load-toplevel :evaluate)
(defun duplicate (form)
(list form form)))
And since it is then in the same file, it avoids the non-guaranteee that
you noticed, as long as do-twice is only used in the same file as
it is defined.
> (defmacro do-twice(form)
> (cons 'progn (duplicate form)))
>
> (do-twice (print "Oh dear "))
>
> Now I can build my software with
>
> (load "plan-b-prelude.lisp")
> (compile-file "plan-b-main.lisp")
>
> and run it with (load "plan-b-main").
>
> At last! I've written a macro that expands to an eval-when.
Good job! The trick, as you step back and look at what you
did, was to wrap an eval-when to get the behavior that you
wanted. You could also wrap the macro, as well.
> Hmm. I'm not too sure what I've achieved here. In both
> Allegro and CMUCL, the compile-file makes the function
> DUPLICATE available in the Lisp image. However reading
> 3.2.3.1 Processing of Top Level Forms:
>
> Evaluate: evaluate the body in the dynamic execution
> context of the compiler, using the evaluation
> environment as the global environment and the lexical
> environment in which the eval-when appears.
> so I don't think this is guaranteed.
And more specifically, 3.2.3.1.1 gives this non-guarantee in the last
textual paragraph.
> In particular, if I
> think that if I want to extend my program portably with a
> third file that uses the DO-TWICE macro, I need to load
> plan-b-main.fasl before I compile the third file.
Correct, unless the implementation subscribes to the
eval-when-compile/eval-when-compile-load distinction that
I have proposed codifying earlier in this thread.
--
Duane Rettig ·····@franz.com Franz Inc. http://www.franz.com/
555 12th St., Suite 1450 http://www.555citycenter.com/
Oakland, Ca. 94607 Phone: (510) 452-2000; Fax: (510) 452-0182
Duane Rettig <·····@franz.com> wrote:
+---------------
| I usually tend to go ahead and add the :evaluate - defensive
| programming against the user who loads my file interpreted...
...
| (eval-when (:compile-toplevel :load-toplevel :evaluate)
| (defun duplicate (form)
| (list form form)))
+---------------
:EVALUATE? I know about :EXECUTE. Is :EVALUATE an ACL thing?
-Rob
-----
Rob Warnock <····@rpw3.org>
627 26th Avenue <URL:http://rpw3.org/>
San Mateo, CA 94403 (650)572-2607
From: Duane Rettig
Subject: Re: Extent of definitions in environments (Was: First draft ...)
Date:
Message-ID: <43c3llpne.fsf@franz.com>
····@rpw3.org (Rob Warnock) writes:
> Duane Rettig <·····@franz.com> wrote:
> +---------------
> | I usually tend to go ahead and add the :evaluate - defensive
> | programming against the user who loads my file interpreted...
> ...
> | (eval-when (:compile-toplevel :load-toplevel :evaluate)
> | (defun duplicate (form)
> | (list form form)))
> +---------------
>
> :EVALUATE? I know about :EXECUTE. Is :EVALUATE an ACL thing?
No, sorry; it was a brain-o
--
Duane Rettig ·····@franz.com Franz Inc. http://www.franz.com/
555 12th St., Suite 1450 http://www.555citycenter.com/
Oakland, Ca. 94607 Phone: (510) 452-2000; Fax: (510) 452-0182
Alan Crowe <····@cawtech.freeserve.co.uk> wrote:
+---------------
| ...things will not go well. The compiler will put object code
| for DUPLICATE into the output file, but not add it to the
| evaluation environment
|
| ;;; Compiling file single-file-fancy.lisp
| ; While compiling (:TOP-LEVEL-FORM "single-file-fancy.lisp" 106):
| Error: attempt to call `DUPLICATE' which is an undefined function.
|
| The minimal fix is to use eval-when
|
| (eval-when (:compile-toplevel)
| (defun duplicate (form)
| (list form form)))
|
| (defmacro do-twice(form)
| (cons 'progn (duplicate form)))
|
| (do-twice (print "Oh dear "))
|
| Compiling works because the definition of DUPLICATE went
| into the compilers evaluation environment and was available
| for expanding the macro.
+---------------
Yes, but interactive development is now broken!! [1]
> (eval-when (:compile-toplevel)
(defun duplicate (form)
(list form form)))
NIL
> (defmacro do-twice(form)
(cons 'progn (duplicate form)))
DO-TWICE
> (do-twice (print "Oh dear "))
;
; Warning: This function is undefined:
; DUPLICATE
Error in KERNEL:%COERCE-TO-FUNCTION: the function DUPLICATE is undefined.
[Condition of type UNDEFINED-FUNCTION]
Restarts:
0: [ABORT] Return to Top-Level.
I think you have no choice but to use:
> (eval-when (:compile-toplevel :load-topleve :execute)
(defun duplicate (form)
(list form form)))
-Rob
[1] At least, in implementations that don't automatically compile
everything.
-----
Rob Warnock <····@rpw3.org>
627 26th Avenue <URL:http://rpw3.org/>
San Mateo, CA 94403 (650)572-2607
Rob Warnock advised:
| (eval-when (:compile-toplevel)
| (defun duplicate (form)
| (list form form)))
|
| (defmacro do-twice(form)
| (cons 'progn (duplicate form)))
|
| (do-twice (print "Oh dear "))
|
| Compiling works because the definition of DUPLICATE went
| into the compilers evaluation environment and was available
| for expanding the macro.
+---------------
Yes, but interactive development is now broken!! [1]
I agree, but the interactive development environment broke
the minute I compiled code that used macros that were still
under development. If I change DUPLICATE I have to recompile
all the code that uses DO-TWICE.
Alan Crowe
Edinburgh
Scotland
Duane Rettig <·····@franz.com> writes:
> In Allegro CL, we've always made a slight distinction between these
> two styles of definition (at top-level, of course):
>
> `(progn
> (eval-when (:compile-toplevel)
> ,defining-form-body)
> (eval-when (:load-toplevel :execute)
> ,defining-form-body))
>
> and
>
> `(progn
> (eval-when (:compile-toplevel :load-toplevel :execute)
> ,defining-form-body))
>
> in that the first style explicitly keeps the definition in the
> compile-file-environment, which makes it only available while the
> compile-file is in effect, and the second places the definiton
> directly into the global-environment, which makes it available both
> to the compile-file and to the lisp afterwards.
Huh? You totally lost me there. Can you explain that in more detail or
point me to the docs that do. I sure wouldn't have expected the
behavior of EVAL-WHEN to depend on the grouping.
-Peter
--
Peter Seibel ·····@javamonkey.com
Lisp is the red pill. -- John Fraser, comp.lang.lisp
From: Duane Rettig
Subject: Re: Extent of definitions in environments
Date:
Message-ID: <4smbniw1j.fsf@franz.com>
Peter Seibel <·····@javamonkey.com> writes:
> Duane Rettig <·····@franz.com> writes:
>
> > In Allegro CL, we've always made a slight distinction between these
> > two styles of definition (at top-level, of course):
> >
> > `(progn
> > (eval-when (:compile-toplevel)
> > ,defining-form-body)
> > (eval-when (:load-toplevel :execute)
> > ,defining-form-body))
> >
> > and
> >
> > `(progn
> > (eval-when (:compile-toplevel :load-toplevel :execute)
> > ,defining-form-body))
> >
> > in that the first style explicitly keeps the definition in the
> > compile-file-environment, which makes it only available while the
> > compile-file is in effect, and the second places the definiton
> > directly into the global-environment, which makes it available both
> > to the compile-file and to the lisp afterwards.
>
> Huh? You totally lost me there. Can you explain that in more detail or
> point me to the docs that do.
It is not yet documented, and is intended to become a guarantee in
7.0. The fact that it's always worked that way has always been
something that a user couldn't count on, because the spec requires
that the user not make those assumptions.
> I sure wouldn't have expected the
> behavior of EVAL-WHEN to depend on the grouping.
I admit it's not pretty from a consistency point-of-view. And the
desire to supply the extra power really has little to do with whether
we confiscate a pair of undefined usages or define the extension in
some other way. However, what would be the alternative to allow this
power to be added with an extension?
One alternative would be to add yet another eval-when keyword: say,
:compile-extended-toplevel, which would describe such indefinite-extent
behavior. There would be two problems with such an addition, besides
the fact that the implementation would still have to restrict the
:compile-toplevel behavior to _not_ retain the definition:
1. The name is longer and not really even descriptive enough as
it is; it should be something like :compile-toplevel-indefinite-extent.
If this seems too trivial a problem to consider, note that the Ansi
committee added :compile-toplevel as a preferred name for compile,
and :load-toplevel as a preferred name for load, and :execute in
place of eval, in order to make the intention of the situation clearer.
2. A new eval-when keyword would tend to cause programs that use it
to break harder in lisps that don't recognize the new keyword - you
might not even be able to compile your app. At least, the way I've
described it, the behavior falls completely within CL standard,
and doesn't resort to nonstandard extensions to even allow the basic
compilation to take place. And since the failure that might occur
is one which can usually be worked around by loading the compiled
file, it seems like a much less harsh change to make (indeed, in
Allegro CL the only change would be the guarantee that becomes
documented).
--
Duane Rettig ·····@franz.com Franz Inc. http://www.franz.com/
555 12th St., Suite 1450 http://www.555citycenter.com/
Oakland, Ca. 94607 Phone: (510) 452-2000; Fax: (510) 452-0182