Hi!
I'm a little confused about the relationship between top-level forms,
eval-with and compile-file. I'm listing a mix of thoughts, questions
and tentative answers below.
** What is a top-level form?
I have found three approaches to answer this question: the first two
are complementary, the third contradicts the others but I think this
is not a problem because the motivations for the definition are quite
different; anyway, I will comment it.
1) According to section 3.2.3.1 in the Hyperspec, a top-level form is:
a) a form at the top level of a file, or
b) a form in the macro expansion of a top-level macro
c) a form in a top-level progn, locally, macrolet or
symbol-macrolet, or
d) a form in a top-level eval-when that "process" its body
(note that I'm using the expression "top level form" in
the simple sintactical sense of not being nested as opposed
to top-level form, which is what is being defined.
So b),c),d) are recursive and a) is the base case)
2) A form for which the situations :compile-toplevel and
:load-toplevel are taken into account. For other forms
(subforms) these situations are ignored.
While 1) is a procedure to determine if a form is top-level,
2) gives us the implications of this form being top-level.
(Is this the only pragmatic importance of the concept???)
3) According to CLTL 5.3:
A top-level form is specifically designed to be convenient for
use at top level rather than embedded within other forms.
These forms are (at least because they are described in
subsections of 5.3):
a) defun (5.3.1)
b) defvar, defparameter, defconstant (5.3.2)
c) eval-when (5.3.3)
Following 1), defun can be top-level (for example, when it is
top level) but can also be non-top-level (for example, when it
is nested into another defun or a let). So 3) contradicts 1).
I will ignore this definition for the remaining of the post.
** What is the motivation for introducing top-level forms?
Definition 2) above is an answer but brings an obvious new question:
Why should the use of :compile-toplevel and :load-toplevel
be restricted?
Consider the following example:
((defun make-n-adder (n)
(defun n-adder (a)
(+ a n))))
Of course, the nested defun can't be evaluated at compile or load
time but only during execution (caused by the invokation of
make-n-adder), so:
((defun make-n-adder (n)
(eval-when (:compile-toplevel)
(defun n-adder (a)
(+ a n)))))
((defun make-n-adder (n)
(eval-when (:load-toplevel)
(defun n-adder (a)
(+ a n)))))
should be banned. Consider then the following situation:
(let (n 5)
(defun n-adder (a)
(+ a n)))
The nested defun can't be evaluated at compile time again but now
it could make sense to be able to evaluate it at load time (I mean
load-compiled-code time; instead load-source-code time will be the
:execute situation in this context). I intentionally said "it could
make sense": in a logical way, I'm not sure if it could be of
practical interest.
Then K. Pittman's equation [1] comes into play:
:compile-toplevel + :load-toplevel = :execute
But the :compile-toplevel "term" could be outside the let form, in a
legal top-level form. So, again, I can't see a (logical) reason to
avoid :load-toplevel here. Obviously, permiting this would be complex:
some forms would be top-level, another will be "semi-top-level"
(supporting any of (:load-toplevel :execute), (:execute) or
(:load-toplevel)), another will be subforms.
Thank you,
CarlosP
References:
[1] From: Kent M Pitman (······@world.std.com)
Subject: Re: empty LET vs. PROGN
Newsgroups: comp.lang.lisp
Date: 2001-01-06 19:36:03 PST
·······@yahoo.com.ar (Carlos P.) writes:
> Hi!
>
> I'm a little confused about the relationship between top-level forms,
> eval-with and compile-file. I'm listing a mix of thoughts, questions
> and tentative answers below.
>
> ** What is a top-level form?
>
> I have found three approaches to answer this question: the first two
> are complementary, the third contradicts the others but I think this
> is not a problem because the motivations for the definition are quite
> different; anyway, I will comment it.
>
> 1) According to section 3.2.3.1 in the Hyperspec, a top-level form is:
>
> a) a form at the top level of a file, or
> b) a form in the macro expansion of a top-level macro
> c) a form in a top-level progn, locally, macrolet or
> symbol-macrolet, or
> d) a form in a top-level eval-when that "process" its body
>
>
> (note that I'm using the expression "top level form" in
> the simple sintactical sense of not being nested as opposed
> to top-level form, which is what is being defined.
> So b),c),d) are recursive and a) is the base case)
>
> 2) A form for which the situations :compile-toplevel and
> :load-toplevel are taken into account. For other forms
> (subforms) these situations are ignored.
> While 1) is a procedure to determine if a form is top-level,
> 2) gives us the implications of this form being top-level.
> (Is this the only pragmatic importance of the concept???)
>
> 3) According to CLTL 5.3:
[Which is not a valid reference for ANSI CL. We made incompatible
changes, some compatible, some incompatible, in the langauge between
CLTL and ANSI CL. This is one that is largely "compatible", but may
still confuse you if you use an old reference.]
> A top-level form is specifically designed to be convenient for
> use at top level rather than embedded within other forms.
> These forms are (at least because they are described in
> subsections of 5.3):
>
> a) defun (5.3.1)
> b) defvar, defparameter, defconstant (5.3.2)
> c) eval-when (5.3.3)
>
> Following 1), defun can be top-level (for example, when it is
> top level) but can also be non-top-level (for example, when it
> is nested into another defun or a let). So 3) contradicts 1).
> I will ignore this definition for the remaining of the post.
>
> ** What is the motivation for introducing top-level forms?
Certain actions happen immediately upon compilation. e.g., the compiler
takes note of the arguments given a function so it can complain about the
function being used wrong. Or information can be added needed for correct
compilation. e.g., defconstant can't work if you don't know the value to
inline as a constant--it might as well be defvar otherwise.
Certain forms that encapsulate others can impede the effectiveness of forms
having this effect. e.g.,
(setq x #'(lambda () (defun f (x) x)))
does not mean that a subsequent call to
(f 3 4)
is syntactically in error since you don't know whether (funcall x) will
have been done.
Because of the "halting problem", there is a gray area in which you can't
just look at a function and symbolically determine without executing it
whether it will or won't have compile-time side-effect, so we don't leave
it to execution semantics to tell us whether those effects will or won't
happen at compile time. Rather, we define a specific set of syntactic
contexts in which the eval-when definitely will run and won't, in a way that
is simple enough that the halting problem is not involved.
> Definition 2) above is an answer but brings an obvious new question:
>
> Why should the use of :compile-toplevel and :load-toplevel
> be restricted?
Because they are specifically and only for the purpose of accomodating
the raising of a loop invariant from the loop part (load-toplevel, since
you are in a loop, loading, using lisp, quitting, loading again, etc.)
to the compile-time part (the non-loop part that is done only once).
In the case you cite below, the simple raising rotocol these use won't work.
> Consider the following example:
>
> ((defun make-n-adder (n)
> (defun n-adder (a)
> (+ a n))))
>
> Of course, the nested defun can't be evaluated at compile or load
> time but only during execution (caused by the invokation of
> make-n-adder), so:
>
> ((defun make-n-adder (n)
> (eval-when (:compile-toplevel)
> (defun n-adder (a)
> (+ a n)))))
>
> ((defun make-n-adder (n)
> (eval-when (:load-toplevel)
> (defun n-adder (a)
> (+ a n)))))
>
> should be banned.
The capability isn't banned. You're just using the wrong keyword.
The reason we changed EVAL to :EXECUTE is that :EXECUTE shoudl do all
the work that the combination of :COMPILE-TOPLEVEL + :LOAD-TOPLEVEL
would do, but in the case where both occur at runtime, so not in a split
way.
> Consider then the following situation:
>
> (let (n 5)
> (defun n-adder (a)
> (+ a n)))
> The nested defun can't be evaluated at compile time again but now
> it could make sense to be able to evaluate it at load time (I mean
> load-compiled-code time; instead load-source-code time will be the
> :execute situation in this context). I intentionally said "it could
> make sense": in a logical way, I'm not sure if it could be of
> practical interest.
Use :EXECUTE to handle this case. Or use special variables because
they do not force the code to be a closure, and consequently do not
make a complication for the compiler.
> Then K. Pittman's equation [1] comes into play:
[Pitman]
> :compile-toplevel + :load-toplevel = :execute
>
> But the :compile-toplevel "term" could be outside the let form, in a
> legal top-level form. So, again, I can't see a (logical) reason to
> avoid :load-toplevel here. Obviously, permiting this would be complex:
> some forms would be top-level, another will be "semi-top-level"
> (supporting any of (:load-toplevel :execute), (:execute) or
> (:load-toplevel)), another will be subforms.
This wasn't followed by a question so I'm not sure what it's here for.
I agree with it though. ;)
I have been thinking a little more about the subject and I
still can't see the motivations for some of the restrictions
and liberties in the use of eval-when.
For example, this code will break the compilation because
dynamic variable x will have no value at that time:
(defvar x 3)
(defmacro f () x)
This is similar to your (Kent's) example [1]:
(let ((x 3))
(defmacro f () x))
which aimed to show why :compile-toplevel part should be
avoided for non top-level forms.
When you say [1]:
The compile-toplevel and load-toplevel only run in the
null lexical environment because they wouldn't have access
to the lexical environment at the time they run anyway.
For the compile-toplevel part this null lexical environment is
the evaluation environment which is inherited by the compilation
environment, isn't it? But it is not the null lexical environment
of the file while being load'ed. So the first example seems as
unsafe as the second: is this really a question of lexical vs
global (null) environment? I know that the problem is obviously
in the difference between compilation(/evaluation) environment
and run-time environment (as defined in "3.2.1 Compiler Terminology")
and that the intention is to solve it by syntactical means and also
that some simplification must be done because of the "halting problem".
So, does this mean that macros should be written with this
convention in mind? (That is, that this mechanism won't work with
macros that were written without taking care about their intended
use: as top-level macros or as non top-level macros.) I'm asking this
because eval'ing the :compile-toplevel part of a top-level form seems
to be unsafe in the general case even when fully applying the restrictions
(according to the examples).
Then, for the "restrictions part" of the question: I haven't thought
a lot about this but if a macro is written as "top-level safe" wouldn't
it be reasonable to let its :compile-toplevel part be eval'ed even
when the macro isn't top-level and this particular condition holds: the
macro is used inside nested forms that only bind names in the lexical
environment (let, let*, flet, labels, macrolet,...)?. Here I'm
avoiding forms whose body could be potentially eval'ed at execution
time but is not necessarilly eval'ed at load time (as the
:compile-toplevel part of macros nested in these forms could be
prematurely eval'ed); for example, defun. Note that this part depends
on the (probably wrong) conclusion of the "liberties part".
Thank you again.
Carlos
References:
[1] From: Kent M Pitman (······@world.std.com)
Subject: Re: empty LET vs. PROGN
Newsgroups: comp.lang.lisp
Date: 2001-01-06 19:36:03 PST
> > Then K. Pittman's equation [1] comes into play:
>
> [Pitman]
>
Sorry.
·······@yahoo.com.ar (Carlos P.) writes:
> Then, for the "restrictions part" of the question: I haven't thought
> a lot about this
That's ok. It's a hard problem to think about.... Some of us have been
burned before, so for us it's a "memory exercise" while for you it's a
"problem solving exercise". You're at a disadvantage in churning out
examples because they are not seared into your wetware with a strong link
to some associated memory of pain...
> but if a macro is written as "top-level safe" wouldn't
> it be reasonable to let its :compile-toplevel part be eval'ed even
> when the macro isn't top-level and this particular condition holds: the
> macro is used inside nested forms that only bind names in the lexical
> environment (let, let*, flet, labels, macrolet,...)?
No, I don't think so... Just off the top of my head, this below is what
comes to mind. I didn't think a lot either. ;-)
In part because there's already a condition that will fire here,
:execute. You should never bein a case where more than one condition
applies because you can't communicate among them to say "I handled
this one" and there's not a mechanism for saying "prefer this one".
And in part because you'd effectively be moving side-effects to compile
time that belong at load time. Consider:
(let ((x (make-package "FOO"))) ;has a persistent side-effect at runtime
(defmacro foo () ... use x...))
Doing this at compile time moves the package making to compile time
and does not arrange for it to be recreated at runtime. And what
about
(defvar *foo* 0)
(defun new-foo () `(foo ,(incf *foo*)))
(let ((x (new-foo))) ;<-- BAD
(defmacro foo () x))
where the side-effect needs to be a runtime effect.
Consider also
(let ((x (foo)))
(defmacro foo () x) ;<-- BAD
(defun foo () x))
where FOO isn't defined until runtime because it's in another package, but
it's apparently needed both at compile time and runtime.
And also consider
(let ((x (foo)))
(defmacro foo () 'foo)
x)
where you would presumably NOT move the side-effect back to compile time
versus
(let ((x (foo)))
(defmacro foo () x) ;<-- BAD
'foo)
where you would.
> (let ((x (make-package "FOO"))) ;has a persistent side-effect
> at runtime
> (defmacro foo () ... use x...))
>
> Doing this at compile time moves the package making to compile
> time and does not arrange for it to be recreated at runtime.
> And what about
I'm not saying that the (let ...) form in the example should
be eval'ed at compilation time but only the :compile-toplevel
part of defmacro (of course, I understand that it will be an
error in this example). The question was very related to the
first part of my post. The chain of reasoning is this (I hope
to be clearer this time):
> (defvar x 3)
> (defmacro f () x)
> (let ((x 3))
> (defmacro f () x))
As these two examples present the same problem if the
:compile-toplevel part of defmacro is effectively eval'ed at
compilation time, here -at least- safety is not a matter of
being top-level or non top-level. So the :compile-toplevel
part of an eval-when is not safely eval'ed at compilation
time by the mere fact that it appears as a top-level form.
So that idea of "top-level safe" had to do with avoiding
dependencies on an environment inexistent at compilation
time and had nothing to do with "top-level'ness" except for
that matter of a possible convention that relates both things:
> So, does this mean that macros should be written with this
> convention in mind? (That is, that this mechanism won't work with
> macros that were written without taking care about their intended
> use: as top-level macros or as non top-level macros.)
(everytime I talk about macros in this context I'm thinking
about its expansion into a progn containing eval-when forms
with :compile-toplevel parts, :load-toplevel parts and
:execute parts)
So instead of being safe by virtue of being top-level perhaps
the macro itself is safe (I will not say "top-level safe" anymore)
or perhaps indications about safe use of the macro should be
given...for example the use of defmacro in
(defvar x 3) (defmacro f () x) isn't safe ... (why is this
allowed at top-level? Could it be because this unsafe use will
be uncommon at top-level?)). Quoting myself again:
> When you say [1]:
>
> The compile-toplevel and load-toplevel only run in the
> null lexical environment because they wouldn't have access
> to the lexical environment at the time they run anyway.
>
> For the compile-toplevel part this null lexical environment is
> the evaluation environment which is inherited by the compilation
> environment, isn't it? But it is not the null lexical environment
> of the file while being load'ed. So the first example seems as
> unsafe as the second: is this really a question of lexical vs
> global (null) environment?
But there is -at least- another problem: some forms won't be
executed until runtime. I'm excluding these ones as I've explained
in the previous post. So if a macro doesn't depend on an
inexistent environment and it's not nested in a defun-like form
it seems like it would be safe to eval its :compile-toplevel part
(and its :load-toplevel part at load time as I'm excluding
defun-like forms). I'm not implying that the compiler would be able
to detect this condition but perhaps that eval-when could flag
it someway: for example with new situations. I'm involved with
this mainly because I'm working in a CL implementation and when
the day comes to implement this feature I would like to provide
some compatible extensions if possible. That's the reason why
I'm allowing myself to hypothesize some things.