From: Jimmy Miller
Subject: Sneakily EVALing code in a non-null lexical environment
Date: 
Message-ID: <0d36616a-5868-4f7c-b949-3ac386d96481@x3g2000yqa.googlegroups.com>
Why does the following code snippet:

(setf x (read-from-string "(+ y 1)"))
(defmacro id (x) x)
(let ((y 3))
  (id x))

Produce the value (+ Y 1) while this code:

(defmacro test () (list '+ 'y '1))
(let ((y 3))
  (test))

Produces the number 4?  I've confirmed that x and (list '+ 'y '1) are
both lists, and EQUAL says that they're equal to each other, so why
does Lisp treat them differently when they're macro return values?

As the thread title implies, I thought that the ID macro would be a
clever way of calling EVAL on code in a non-null lexical environment
in an implementation-independent way, simply by expanding to the code
it was passed, but obviously I was wrong.  Can anyone explain why?

From: Kaz Kylheku
Subject: Re: Sneakily EVALing code in a non-null lexical environment
Date: 
Message-ID: <20090718111734.867@gmail.com>
On 2009-07-06, Jimmy Miller <··············@gmail.com> wrote:
>Subject: Re: Sneakily EVALing code in a non-null lexical environment

What the subject line implies is impossible (in portable ANSI CL code).
EVAL does not evaluate in a non-null lexical environment. Such an EVAL
may be provided as an extension.

> Why does the following code snippet:
>
> (setf x (read-from-string "(+ y 1)"))

Be aware that if a variable called X does not exist, SETF is not required to
create it. The behavior is undefined. Many Lisps do respond by doing
something related to the symbol X, in their own peculiar ways.

> (defmacro id (x) x)

Note that the second X here refers to the first one. The dynamic
variable X is not involved.  You might want to call this:

 (defmacro id (form) form)

to make it clearer that this macro is completely unrelated to the
X which supposedly holds (+ Y 1).

> (let ((y 3))
>   (id x))

Here (id x) is equivalent to x; the macro does nothing other than
convert expressions of the form (id <expr>) to <expr>.

So you've basically written:

  (let ((y 3))
    x)

The macro receives the literal form X (the symbol X itself) as its
argument, and since all it does is substitute its argument form,
we just end up with X in its place.

There is no opportunity here for X to be reduced to its value (+ y 1);
that would require an extra round of evaluation, which has to be
deliberately coded in the macro.

> Produce the value (+ Y 1) while this code:
> (defmacro test () (list '+ 'y '1))
> (let ((y 3))
>   (test))

Here the macro produces the object (+ y 1) as its expansion. Its job,
as defined, is to transform occurences of the form (TEST) to (+ Y 1).
So your LET block is, effectively:

 (let ((y 3))
   (+ y 1))

> Produces the number 4?  I've confirmed that x and (list '+ 'y '1) are
> both lists, and EQUAL says that they're equal to each other, so why
> does Lisp treat them differently when they're macro return values?

Ah, that brings us back to an earlier point, namely that you should
rewrite

 (defmacro id (x) x)

as

 (defmacro id (form) form)

so that you don't confuse yourself into thinking that it has anything
to do with the global variable X.

The X that the macro returns refers to its own parameter. It is 
reduced to the value of that parameter, and not to the value of
the global variable.

In compiled code, macros cannot even /see/ variables defined in that
code (unless the definitions are wrapped with an appropriate EVAL-WHEN).

> As the thread title implies, I thought that the ID macro would be a
> clever way of calling EVAL on code in a non-null lexical environment
> in an implementation-independent way

Macros let you stick source code into other source code, so then when that
source code becomes a lexical environment, the code which was stuck into it
turns into translate code which can see that environment.

So for instance (test) sticks the source code (+ y 1) into the (let ((y 3))
...) source code. The result is that the (+ y 1) accesses the binding of
Y that has value 3.

We could pull the inserted code from a variable like this:

 ;; ensure that our macro can use this variable in all situations:
 ;; - when the macro is loaded as source code and then used
 ;; - when the macro is loaded as compiled code and then used
 ;; - when the macro is loaded during compilation and used
 ;;   in code that is being compiled

 (eval-when (:execute :load-toplevel :compile-toplevel)
   (defconstant x '(+ y 1)))


 ;; Expand to the piece of code named by variable VAR
 (defmacro insert-named-code (var)
   (eval var)) ;; macro-expansion-time eval

  (let ((y 1))
    (insert-named-code x))

Now the LET should yield two. Note that we have not tricked EVAL into
using a lexical environment.  Our EVAL reduces the variable X to its
value (+ Y 1), which comes from the toplevel dynamic environment. 
Our EVAL knows nothing about Y. What resolves Y is that the macro
sticks the source code (+ Y 1) in the proper place.

All this happens long before the environment set up by the LET even exists at
all.

In fact this code perfectly demonstrates that the evaluation is not lexically
transparent. The DEFCONSTANT form which which binds X to (+ Y 1) is clearly
outside of the scope of the LET; it is not in that lexical environment at all.
Yet, (+ Y 1) ends up seeing the (Y 1) binding. We are simply dicing up bits of
source code in an unhygienic way.
From: Jimmy Miller
Subject: Re: Sneakily EVALing code in a non-null lexical environment
Date: 
Message-ID: <12acc0de-a13d-4287-8b44-3e31cd731999@h2g2000yqg.googlegroups.com>
On Jul 6, 1:41 am, Kaz Kylheku <········@gmail.com> wrote:
> On 2009-07-06, Jimmy Miller <··············@gmail.com> wrote:
>
> >Subject: Re: Sneakily EVALing code in a non-null lexical environment
>
> What the subject line implies is impossible (in portable ANSI CL code).
> EVAL does not evaluate in a non-null lexical environment. Such an EVAL
> may be provided as an extension.

Yeah, I've decided to just use extensions based on the implementation
that my code runs on, it seems that most provide this facility anyway.

> > Why does the following code snippet:
>
> > (setf x (read-from-string "(+ y 1)"))
>
> Be aware that if a variable called X does not exist, SETF is not required to
> create it. The behavior is undefined. Many Lisps do respond by doing
> something related to the symbol X, in their own peculiar ways.

Yes, I'm well aware of that; I just typed in a snippet off the top of
my head, I don't actually set variables like that in my code.

> > (defmacro id (x) x)
>
> Note that the second X here refers to the first one. The dynamic
> variable X is not involved.  You might want to call this:
>
>  (defmacro id (form) form)
>
> to make it clearer that this macro is completely unrelated to the
> X which supposedly holds (+ Y 1).

Again, I know.

> There is no opportunity here for X to be reduced to its value (+ y 1);
> that would require an extra round of evaluation, which has to be
> deliberately coded in the macro.

Yeah, this occurred to me while I was thinking about the problem
earlier this morning.  I should think longer before I post a
question. :)

> > Produces the number 4?  I've confirmed that x and (list '+ 'y '1) are
> > both lists, and EQUAL says that they're equal to each other, so why
> > does Lisp treat them differently when they're macro return values?
>
> Ah, that brings us back to an earlier point, namely that you should
> rewrite
>
>  (defmacro id (x) x)
>
> as
>
>  (defmacro id (form) form)
>
> so that you don't confuse yourself into thinking that it has anything
> to do with the global variable X.

I never did confuse myself in this way.

But thanks for taking the time to write your explanation, I appreciate
it.
From: http://public.xdi.org/=pf
Subject: Re: Sneakily EVALing code in a non-null lexical environment
Date: 
Message-ID: <m27hymh5ss.fsf@wyoming.home>
Jimmy Miller <··············@gmail.com> writes:

> Why does the following code snippet:
>
> (setf x (read-from-string "(+ y 1)"))
> (defmacro id (x) x)
> (let ((y 3))
>   (id x))
>
> Produce the value (+ Y 1) while this code:
>
> (defmacro test () (list '+ 'y '1))
> (let ((y 3))
>   (test))
>
> Produces the number 4?  I've confirmed that x and (list '+ 'y '1) are
> both lists, and EQUAL says that they're equal to each other, so why
> does Lisp treat them differently when they're macro return values?

Try comparing the result of (macroexpand '(id x)) to that of
(macroexpand '(test))
From: Alex Mizrahi
Subject: Re: Sneakily EVALing code in a non-null lexical environment
Date: 
Message-ID: <4a5269fa$0$48238$14726298@news.sunsite.dk>
 JM> Can anyone explain why?

You do not have a good idea about how CL works and how macros work.

Basically, lexical variables and macro DO NOT EXIST when you run your
code. They exist in the source code and they exist while program is 
transformed,
but once program (or function or a code snippet) is compiled, they cease
to exist.

If you know C (or C++, or any "compiled" language), it is absolutely like in 
C -- 
variables and macros exist in your source code, but then program is 
compiled,
and variables/macros are not directly present in the machine code.

Dynamic (special) variables, however, DO exist in runtime. Thus, use dynamic
variables if you want them to exist in runtime, when you do EVAL. That's it.