Hi,
I've been meaning to ask for a long time about why LET was chosen NOT
to evaluate its body forms as top-level forms if the LET is used at
top-level. I guess there must be a good reason for this, and maybe
it's a silly question, given that something like
(let ((x 3))
(defun f (y)
(* x y)))
evaluates a function in some lexical environment, and so how can it
make sense that it also be a top-level form? But I'll ask anyway,
since my current understanding of top-level forms is that they are
handled specially by the file compiler.
After writing a recent macro that was fairly largish, I would have
preferred LET and LET* as evaluating each of their body forms at
top-level.
dave
David Bakhash <·····@alum.mit.edu> writes:
>
> Hi,
>
> I've been meaning to ask for a long time about why LET was chosen NOT
> to evaluate its body forms as top-level forms if the LET is used at
> top-level. I guess there must be a good reason for this, and maybe
> it's a silly question, given that something like
>
> (let ((x 3))
> (defun f (y)
> (* x y)))
>
> evaluates a function in some lexical environment, and so how can it
> make sense that it also be a top-level form? But I'll ask anyway,
> since my current understanding of top-level forms is that they are
> handled specially by the file compiler.
>
> After writing a recent macro that was fairly largish, I would have
> preferred LET and LET* as evaluating each of their body forms at
> top-level.
Well, to understand this you have to understand what the whole purpose
of the "toplevel" form distinction is. I'm going to make some remarks
that should be interpreted as a sketch--that is, I will omit and gloss
some details to make a point. Don't take this as a tutorial, but rather
as a possible crude model that you can build some conceptual intuitions
around.
EVAL-WHEN is about observing that sometimes there are actions which are
done wholly at execute time and sometimes there are actions that if you
could wrench them in half, you could most effectively do one "half" of the
computation just once at compile time and then do the other "half" at runtime.
The hope is that by splitting it this way, you'll maximize the runtime
speed by factoring out "invariants".
I like to think of it that the actions of
compile-toplevel + load-toplevel = execute
although sometimes in practice there are things that must be redundantly
done in both the compile-toplevel and load-toplevel parts for bookkeeping
reasons.
Now, if I have a form x which depends on its lexical environment, then
it looks like
(f (g))
where (f ...) is something that establishes a lexical environment and
(g) is the form that will expand into an eval-when.
supposing that the expansion of (g) is
(progn (eval-when (:compile-toplevel) ... first "half" of the execute ...)
(eval-when (:load-toplevel) ... second "half" of the execute ...)
(eval-when (:execute) ... the whole thing ...))
Now there are two questions:
- Why doesn't load-toplevel happen at all when there is a lexical
environment
and
- Why doesn't load-toplevel see the lexical environment when it does
execute
The first is answered by thinking about
(defun foo () (defun bar () ...))
If you assume DEFUN will expand into something that does some compile-time
and/or load-time setup, it's pretty clear that FOO can do such setup but
BAR must not. That's because FOO may never be called, and so BAR shouldn't
have its stuff done "prematurely" either at compile time or at load time.
So we simply define that if it's not at toplevel, the execute part is the
only method that runs. Only that version can see the lexical environment.
That goes back to the first question. The compile-toplevel and load-toplevel
only run in the null lexical environment because they wouldn't have access
to the lexical environment at the time they run anyway.
Consider the problem in
(let ((x 3))
(defmacro f () x))
Think about what the expansion would be in EVAL-WHEN's. Probably something
like
(let ((x 3))
(progn
(eval-when (:compile-toplevel)
(system::record-macro-for-compilation
'f #'(lambda () x))) ; x will be free since no x exists at compile time
(eval-when (:load-toplevel :execute)
(setf (macro-function 'f) #'(lambda () x)))))
The problem is that the :compile-toplevel thing would screw up badly if you
started trying to grab the code and use it in the compiler. User code has
no way to detect it's going to lose here and so if you go back to my
equation
compile-toplevel + load-toplevel = execute
the compiler can tell the compile-toplevel is going to lose and so it just
opts not to use it. The execute method will win, and so that gets used.
Hmm. I feel as if I'm not being as coherent as I could here, but I hope this
helps a little.
The formula can only really safely be used, and the compile+load pair
can only be substituted for the execute, in the case where there is no lexical
environment to be lost.
Thanks, kent. This helps. The pseudo-formula you used:
> compile-toplevel + load-toplevel = execute
made the point pretty clear. It's now easier for me to see how
:compile-toplevel forms must be in the null lexical environment.
thanks,
dave
Another reason for the body of LET not being at top level is that if
it was, then you'd need to invent some other PROGN-like construct
which stopped things from being at top level. This may sound stupid,
and it's obviously a lesser reason than Kent's, but it can happen.
An example is if you have a macro which expands to
(eval-when (:compile-toplevel ...) ...)
the evaluation of whose expansion you want to delay until load time.
Because of the semantics of EVAL-WHEN it is *not* enough to say this:
(eval-when (:load-toplevel) (macro ...))
because this expands to
(eval-when (:load-toplevel) (eval-when (:compile-toplevel ...) ...))
and because the body of EVAL-WHEN at toplevel is also at toplevel this
will in fact cause the expansion to be evaluated at compile time.
You can't use PROGN to stop this because it preserves top-levelness,
but you *can* use LET, which does not:
(eval-when (:load-toplevel) (let () (macro ...)))
works.
Now whether this use of LET was ever thought of I'm not sure. I
rather think it wasn't and that EVAL-WHEN should really have some
mechanism of avoiding this problem. But it's certainly useful that
it's there, because when you need it you really need it!
--tim
Tim Bradshaw <···@cley.com> writes:
> Now whether this use of LET was ever thought of I'm not sure. I
> rather think it wasn't and that EVAL-WHEN should really have some
> mechanism of avoiding this problem. But it's certainly useful that
> it's there, because when you need it you really need it!
When you would need this I have no idea. I do see how you can use LET
to get you out of top-level if you were using some macro or special
operator that had special top-level semantics that you wanted to avoid,
then you could use this LET trick. But when such a thing would be
useful I literally have no idea. It's almost always the opposite case,
where you suddenly wish you were at top-level again, at least in my
experience.
Can you please give (or cook up) an example of where one might want to
avoid being at top-level and therefore use LET?
dave
* David Bakhash wrote:
> Can you please give (or cook up) an example of where one might want to
> avoid being at top-level and therefore use LET?
My conduits system defines a version of DEFPACKAGE. DEFPACKAGE needs
to have compile-time effect, as you'd expect. After defining the
variant DEFPACKAGE, it uses it to bootstrap the CL/CONDUITS package
(which is a conduit package like CL but with some of the package
macros & functions replaced by the conduit variants). It's essential
that this invocation of the macro does *not* have effect until load
time, because at compile time most of the conduits system simply isn't
there. So you need to say:
(eval-when (:load-toplevel :execute)
(let ()
(defpackage ... ; this is CONDUITS::DEFPACKAGE
)))
It was while writing this that I discovered how EVAL-WHEN really
works. I thought at the time that EVAL-WHEN should perhaps be changed
so it doesn't have this odd behaviour, but I'm not sure I still think
that.
It's also worth noting that not every implementation gets this right.
Of implementations I had then, CLISP gets it wrong, a prerelease CMUCL
18c gets it right (but has other bugs in EVAL-WHEN), and ACL 5.0.1
gets EVAL-WHEN completely right I think.
--tim
Tim Bradshaw <···@cley.com> writes:
> * David Bakhash wrote:
>
> > Can you please give (or cook up) an example of where one might want to
> > avoid being at top-level and therefore use LET?
>
> My conduits system defines a version of DEFPACKAGE. DEFPACKAGE needs
> to have compile-time effect, as you'd expect. After defining the
> variant DEFPACKAGE, it uses it to bootstrap the CL/CONDUITS package
> (which is a conduit package like CL but with some of the package
> macros & functions replaced by the conduit variants). It's essential
> that this invocation of the macro does *not* have effect until load
> time, because at compile time most of the conduits system simply isn't
> there. So you need to say:
>
> (eval-when (:load-toplevel :execute)
> (let ()
> (defpackage ... ; this is CONDUITS::DEFPACKAGE
> )))
Now I see what you're talking about, and here's how I feel about it.
First off, having read the conduits package, and afterwards having
implemented something similar (maybe more of what I was inquiring about
when I first posted about safe packages), I think that it is possible to
work it so that your macro expands into a DEFPACKAGE at toplevel.
Nevertheless, it's good to know how the function `make-package' works as
well, and then have your defpackage macro expand into using make-package
instead.
dave
* David Bakhash wrote:
> First off, having read the conduits package, and afterwards having
> implemented something similar (maybe more of what I was inquiring about
> when I first posted about safe packages), I think that it is possible to
> work it so that your macro expands into a DEFPACKAGE at toplevel.
> Nevertheless, it's good to know how the function `make-package' works as
> well, and then have your defpackage macro expand into using make-package
> instead.
It is possible to expand into a DEFPACKAGE, and the original system
did that. *But* the form you end up with is large -- often including
something like (:export <almost-every-symbol-in-CL>), so invoking it
from the editor is painful and it bloats code seriously. So I changed
to doing it the way I do now, so M-C-x is faster in emacs, basically!
--tim