From: Aneil Mallavarapu
Subject: Determining lexically bound variables inside a macro
Date: 
Message-ID: <393eea.0408030807.6a342c4b@posting.google.com>
I would like to figure out within a macro whether a variable is
lexically bound.  In particular, I would like to be able to use a
global value for a symbol if a lexical binding isn't available.

Here is the macro I'd like to write:

(defmacro use-lexical-then global (sym &environment e)
  (if (lexical-binding-exists sym e) sym
     `(symbol-value ',sym))

How should we write lexical-binding-exists?  In Lispworks, the
environment object is a struct which has a slot (VARIABLES) containing
an assoc list.  For example, when A is lexically bound, the
environment object is:

#S(LEXICAL::ENVIRONMENT 
      LEXICAL::VARIABLES ((A . #A)) 
      LEXICAL::FUNCTIONS NIL
      LEXICAL::REMOTE-ENVIRONMENT NIL)

So, I could solve the problem with 

(defun lexical-binding-exists (sym e)
  (assoc 'sym (slot-value e 'lexical::variables))).  

The problem is that the implementation of the environment object is
unspecified.  So, is there a correct(& portable) way to write
lexical-binding-exists?

* * * *
Why this approach?  

I don't want to use dynamic variables because of the debugging
problems they can introduce, so defvar, defparameter, declaim/proclaim
special won't work for me.  Defconstant won't work because it will
prevent me from reusing symbols in lexical bindings.  I have a large
number of symbols I'd like to define in the global environment.
* * * *

Example:

(setf a 'global)

(defun lexical ()
  (let ((a 'lexical))
     (use-lexical-then-global a)))

(defun global ()
   (use-lexical-then-global a))

(defun unlike-dynamic ()
  (let ((a 'lexical)) ;; if A had been defined as a dynamic var
     (global))) ;; then, (global) => 'lexical

a         => 'global
(lexical) => 'lexical
(global) => 'global
(unlike-dynamic) => 'global 


Thanks for any help,

Aneil Mallavarapu

From: Pascal Costanza
Subject: Re: Determining lexically bound variables inside a macro
Date: 
Message-ID: <ceofa8$bl2$1@newsreader2.netcologne.de>
Aneil Mallavarapu wrote:

> I don't want to use dynamic variables because of the debugging
> problems they can introduce, so defvar, defparameter, declaim/proclaim
> special won't work for me.  Defconstant won't work because it will
> prevent me from reusing symbols in lexical bindings.  I have a large
> number of symbols I'd like to define in the global environment.

I don't see what debugging problems special variables may introduce, but 
what you actually want is a global lexical variables. The standard way 
to simulate this in Common Lisp is by using a symbol macro, like this:

(defvar *some-var* some-value)
(define-symbol-macro some-var *some-var*)

Symbol macros are lexically scoped, so this should do what you want. 
Sometimes, this is offered as a macro. Google for deflexical, deflex, 
defglobal, or some such.


Pascal

-- 
Tyler: "How's that working out for you?"
Jack: "Great."
Tyler: "Keep it up, then."
From: Aneil Mallavarapu
Subject: Re: Determining lexically bound variables inside a macro
Date: 
Message-ID: <393eea.0408031909.38d5ad30@posting.google.com>
Hi Pascal,
Symbol macros do the trick.
*
The 'debugging' problem I alluded to is the inadvertent binding of a
dynamic var.  This is normally not a problem.  But, in a multi-user
system with many dynamic vars, you can imagine a situation in which
one user writes code which inadvertently binds a dynamic var defined
elsewhere (by another user), and then calls functions which intend to
refer to the global-level binding.  Of course, one could stick to the
*VAR* naming convention, but I'd like to avoid this for aesthetic
reasons.
*
Thanks for the help,
Aneil

Pascal Costanza <········@web.de> wrote in message news:<············@newsreader2.netcologne.de>...
> Aneil Mallavarapu wrote:
> 
> > I don't want to use dynamic variables because of the debugging
> > problems they can introduce, so defvar, defparameter, declaim/proclaim
> > special won't work for me.  Defconstant won't work because it will
> > prevent me from reusing symbols in lexical bindings.  I have a large
> > number of symbols I'd like to define in the global environment.
> 
> I don't see what debugging problems special variables may introduce, but 
> what you actually want is a global lexical variables. The standard way 
> to simulate this in Common Lisp is by using a symbol macro, like this:
> 
> (defvar *some-var* some-value)
> (define-symbol-macro some-var *some-var*)
> 
> Symbol macros are lexically scoped, so this should do what you want. 
> Sometimes, this is offered as a macro. Google for deflexical, deflex, 
> defglobal, or some such.
> 
> 
> Pascal
From: Rob Warnock
Subject: Re: Determining lexically bound variables inside a macro
Date: 
Message-ID: <X9qdnQyd2LGL8o3cRVn-jw@speakeasy.net>
Aneil Mallavarapu <·················@gmail.com> wrote:
+---------------
| Of course, one could stick to the *VAR* naming convention, but I'd
| like to avoid this for aesthetic reasons.
+---------------

Please don't avoid it, at least, not for any variables that are truly
dynamically bound *anywhere*. It will only make your code hard to maintain
by others. The *VAR* convention is, at this point, so pervasive in the
community that it's almost mandatory (as is +NAME+ for global constants).

As Kent Pitman is well-known for saying[1], a language is really defined
as much by its community as by the standard (if any), a point that should
not be ignored when extending said language. The jury is still out on
whether "global lexicals" really have a proper place in Common Lisp,
and if so, what the best way to implement them is.[2] I say this as
one who has more than once tried to find "the right way" to fit
"global lexicals" into Common Lisp (since, coming from Scheme, it
initially seemed like a good idea). But I'm still not sure there is
one that doesn't violate the community expectations of what is implied
by the normal variable naming conventions.

Yes, I have my own personal DEFLEX macro, and yes, I use it a lot
when doing exploratory programming interactively at the REPL, but I
don't use it at *all* in actual source code I write. Besides a few
subtle bugs[3], it just doesn't seem to feel stylistically correct
for wide use in Common Lisp (at least not yet).

In summary, I would suggest *not* changing all your globals to
"lexical globals" just because you find *VAR* "unaesthetic".
Try it out for a while in some corner of the code (which nevertheless
gets edited by more than one person) to see whether it passes the
dual tests of robustness and the principle of least astonishment
to your fellow coders.


-Rob

[1] <http://www.nhplace.com/kent/PS/Lambda.html>

[2] Pascal's simple suggestion, while fine for discussion, doesn't
    actually suffice in practice. E.g., what happens if you have
    said (DEFINE-SYMBOL-MACRO MY-VAR *MY-VAR*) and somebody elsewhere
    in the code also does a (DEFVAR *MY-VAR* xxx) and then SETQs it?
    Oops. You need to do this with some sort of DEFLEX macro that
    automatically munges the underlying variable name so that such
    collisions don't happen [or are at least *very* unlikely].

[3] It didn't always do quite the right thing under CMUCL when used
    in some "place expressions" (SETF, INCF, etc.). I haven't yet
    figured out whether the problem is my macro or in CMUCL's handling
    of symbol macros in place expressions. ("Film at 11...")

-----
Rob Warnock			<····@rpw3.org>
627 26th Avenue			<URL:http://rpw3.org/>
San Mateo, CA 94403		(650)572-2607
From: Aneil Mallavarapu
Subject: Re: Determining lexically bound variables inside a macro
Date: 
Message-ID: <393eea.0408040943.12d790cb@posting.google.com>
····@rpw3.org (Rob Warnock) wrote in message news:<······················@speakeasy.net>...
> Aneil Mallavarapu <·················@gmail.com> wrote:
> +---------------
> | Of course, one could stick to the *VAR* naming convention, but I'd
> | like to avoid this for aesthetic reasons.
> +---------------
> 
> Please don't avoid it, at least, not for any variables that are truly
> dynamically bound *anywhere*. It will only make your code hard to maintain
> by others. The *VAR* convention is, at this point, so pervasive in the
> community that it's almost mandatory (as is +NAME+ for global constants).
> 
> As Kent Pitman is well-known for saying[1], a language is really defined
> as much by its community as by the standard (if any), a point that should
> not be ignored when extending said language. The jury is still out on
> whether "global lexicals" really have a proper place in Common Lisp,
> and if so, what the best way to implement them is.[2] I say this as
> one who has more than once tried to find "the right way" to fit
> "global lexicals" into Common Lisp (since, coming from Scheme, it
> initially seemed like a good idea). But I'm still not sure there is
> one that doesn't violate the community expectations of what is implied
> by the normal variable naming conventions.
> 
> Yes, I have my own personal DEFLEX macro, and yes, I use it a lot
> when doing exploratory programming interactively at the REPL, but I
> don't use it at *all* in actual source code I write. Besides a few
> subtle bugs[3], it just doesn't seem to feel stylistically correct
> for wide use in Common Lisp (at least not yet).

It's a shame that CL hasn't defined such a thing.  I agree with your
comments.  However, I'm at the experimental stages of building an
embedded language, which I don't anticipate will be much used by
LISPers.  Also, I anticipate reading and writing these global-lexical
symbols a LOT... so I'm willing to experiment.

> 
> In summary, I would suggest *not* changing all your globals to
> "lexical globals" just because you find *VAR* "unaesthetic".
> Try it out for a while in some corner of the code (which nevertheless
> gets edited by more than one person) to see whether it passes the
> dual tests of robustness and the principle of least astonishment
> to your fellow coders.
> 
> 
> -Rob
> 
> [1] <http://www.nhplace.com/kent/PS/Lambda.html>
> 
> [2] Pascal's simple suggestion, while fine for discussion, doesn't
>     actually suffice in practice. E.g., what happens if you have
>     said (DEFINE-SYMBOL-MACRO MY-VAR *MY-VAR*) and somebody elsewhere
>     in the code also does a (DEFVAR *MY-VAR* xxx) and then SETQs it?
>     Oops. You need to do this with some sort of DEFLEX macro that
>     automatically munges the underlying variable name so that such
>     collisions don't happen [or are at least *very* unlikely].

Yes, I ended up doing something like the following:

(defmacro defglobalconst (s val)
  (let ((hidden-sym     (make-hidden-symbol s)))
  `(eval-when (:compile-toplevel :load-toplevel :execute)
     (defconstant ,hidden-sym ,v)
     (define-symbol-macro ,s ,hidden-sym))

(defun make-hidden-symbol (s)
  (intern (mkstr (symbol-package s) ":" (symbol-name s))
          "HIDDEN-SYMBOL"))


> 
> [3] It didn't always do quite the right thing under CMUCL when used
>     in some "place expressions" (SETF, INCF, etc.). I haven't yet
>     figured out whether the problem is my macro or in CMUCL's handling
>     of symbol macros in place expressions. ("Film at 11...")

> 
> -----
> Rob Warnock			<····@rpw3.org>
> 627 26th Avenue			<URL:http://rpw3.org/>
> San Mateo, CA 94403		(650)572-2607
From: Rob Warnock
Subject: Re: Determining lexically bound variables inside a macro
Date: 
Message-ID: <6fWdnQF6DtTfjI_cRVn-qw@speakeasy.net>
Aneil Mallavarapu <·················@gmail.com> wrote:
+---------------
| ····@rpw3.org (Rob Warnock) wrote:
| > Yes, I have my own personal DEFLEX macro, and yes, I use it a lot
| > when doing exploratory programming interactively at the REPL, but I
| > don't use it at *all* in actual source code I write. Besides a few
| > subtle bugs[3], it just doesn't seem to feel stylistically correct
| > for wide use in Common Lisp (at least not yet).
| 
| It's a shame that CL hasn't defined such a thing.  I agree with your
| comments.  However, I'm at the experimental stages of building an
| embedded language, which I don't anticipate will be much used by
| LISPers.  Also, I anticipate reading and writing these global-lexical
| symbols a LOT... so I'm willing to experiment.
...
| >     You need to do this with some sort of DEFLEX macro that
| >     automatically munges the underlying variable name so that such
| >     collisions don't happen [or are at least *very* unlikely].
| 
| Yes, I ended up doing something like the following:
| 
| (defmacro defglobalconst (s val)
|   (let ((hidden-sym     (make-hidden-symbol s)))
|   `(eval-when (:compile-toplevel :load-toplevel :execute)
|      (defconstant ,hidden-sym ,v)
|      (define-symbol-macro ,s ,hidden-sym))
| 
| (defun make-hidden-symbol (s)
|   (intern (mkstr (symbol-package s) ":" (symbol-name s))
|           "HIDDEN-SYMBOL"))
+---------------

Well, for what it's worth, here's my current version:

    (defmacro deflex (var val &optional (doc nil docp))    
      (let* ((prefix (load-time-value (symbol-name '#:*storage-for-deflex-)))
	     (name (symbol-name var))
	     (backing-var (intern (concatenate 'string prefix name "*"))))
	`(progn
	   (defparameter ,backing-var ,val ,@(when docp `(,doc)))
	   ,@(when docp
	      `((setf (documentation ',var 'variable) ,doc)))
	   (define-symbol-macro ,var ,backing-var))))	; Must be last!

Note that it uses DEFPARAMETER, since I was trying to copy the
top-level behavior of Scheme's DEFINE (that is, unlike DEFVAR, later
ones override earlier ones), and it also handles documentation strings.


-Rob

-----
Rob Warnock			<····@rpw3.org>
627 26th Avenue			<URL:http://rpw3.org/>
San Mateo, CA 94403		(650)572-2607
From: Pascal Costanza
Subject: Re: Determining lexically bound variables inside a macro
Date: 
Message-ID: <cesubi$i5u$1@newsreader2.netcologne.de>
Aneil Mallavarapu wrote:

> (defmacro defglobalconst (s val)
>   (let ((hidden-sym     (make-hidden-symbol s)))
>   `(eval-when (:compile-toplevel :load-toplevel :execute)
>      (defconstant ,hidden-sym ,v)
>      (define-symbol-macro ,s ,hidden-sym))
> 
> (defun make-hidden-symbol (s)
>   (intern (mkstr (symbol-package s) ":" (symbol-name s))
>           "HIDDEN-SYMBOL"))

This is overkill. If you want a global constant, use the +CONSTANT+ 
naming convention, and you are effectively in a different namespace so 
you won't get any nameclashes.


Pascal

-- 
Tyler: "How's that working out for you?"
Jack: "Great."
Tyler: "Keep it up, then."
From: Pascal Costanza
Subject: Re: Determining lexically bound variables inside a macro
Date: 
Message-ID: <ceq7nh$c00$1@newsreader2.netcologne.de>
Rob Warnock wrote:

> [2] Pascal's simple suggestion, while fine for discussion, doesn't
>     actually suffice in practice. E.g., what happens if you have
>     said (DEFINE-SYMBOL-MACRO MY-VAR *MY-VAR*) and somebody elsewhere
>     in the code also does a (DEFVAR *MY-VAR* xxx) and then SETQs it?
>     Oops. You need to do this with some sort of DEFLEX macro that
>     automatically munges the underlying variable name so that such
>     collisions don't happen [or are at least *very* unlikely].

Thanks for spotting this, and thanks for your other excellent remarks.

In fact, my preliminary conclusions from working on dynamic scoping for 
functions are that _all_ global definitions should be dynamically scoped 
whereas all local definitions should be lexically scoped...



Pascal

-- 
Tyler: "How's that working out for you?"
Jack: "Great."
Tyler: "Keep it up, then."
From: Mario S. Mommer
Subject: Re: Determining lexically bound variables inside a macro
Date: 
Message-ID: <fzsmb3t8ef.fsf@germany.igpm.rwth-aachen.de>
Pascal Costanza <········@web.de> writes:
> My preliminary conclusions from working on dynamic scoping for
> functions are that _all_ global definitions should be dynamically
> scoped whereas all local definitions should be lexically scoped...

I don't agree. I think there should be non-dynamic globals, at least
in an interactive environment. And in fact, that this should be the
default. I am very thankfull that cmucl has
ext:*top-level-auto-declare*, and that setting it to NIL on startup
lets me do

  (setf x 1)

on toplevel without declaring a special variable.

If it were a special, then

(defun foo (x)
    #'(lambda (y) (+ x y)))

would start working in a funny way.

I don't know what that top level x it is (whether a lexical global or
a swarm of fish), but in practice, I don't really care either :-)
From: Rob Warnock
Subject: Re: Determining lexically bound variables inside a macro
Date: 
Message-ID: <m6idnecru-Vxe43cRVn-pQ@speakeasy.net>
Mario S. Mommer  <········@yahoo.com> wrote:
+---------------
| I am very thankfull that cmucl has ext:*top-level-auto-declare*,
| and that setting it to NIL on startup lets me do
| 
|   (setf x 1)
| 
| on toplevel without declaring a special variable.
+---------------

But you'll still get a warning:

    > (setf ext:*top-level-auto-declare* nil)

    NIL
    > (setf x 123)
    ; 

    ; Warning: This variable is undefined:
    ;   X
    ; 
    123
    > 


-Rob

-----
Rob Warnock			<····@rpw3.org>
627 26th Avenue			<URL:http://rpw3.org/>
San Mateo, CA 94403		(650)572-2607
From: Pascal Costanza
Subject: dynamic/non-dynamic globals, was Re: Determining lexically bound variables inside a macro
Date: 
Message-ID: <ceql8l$q7c$1@f1node01.rhrz.uni-bonn.de>
Mario S. Mommer wrote:

> Pascal Costanza <········@web.de> writes:
> 
>>My preliminary conclusions from working on dynamic scoping for
>>functions are that _all_ global definitions should be dynamically
>>scoped whereas all local definitions should be lexically scoped...
> 
> I don't agree. I think there should be non-dynamic globals, at least
> in an interactive environment.

Yes, the listener has different requirements. Global definitions in a 
source code make up the "ontology" of a program or a library - they are 
the entries that you use from the outside to get your own stuff going. I 
think it is very meaningful to be able to say "in my context, I want 
these things to behave slightly differently", and dynamic scoping is a 
very useful mechanism to make this work in a controlled way.

Temporary variables that you introduce in a listener don't have that 
quality. They are actually purely "local" to the task that you are 
concerned about.

Does this make sense?


Pascal

-- 
Pascal Costanza               University of Bonn
···············@web.de        Institute of Computer Science III
http://www.pascalcostanza.de  R�merstr. 164, D-53117 Bonn (Germany)
From: Kaz Kylheku
Subject: Re: Determining lexically bound variables inside a macro
Date: 
Message-ID: <cf333042.0408050921.6343fa32@posting.google.com>
·················@gmail.com (Aneil Mallavarapu) wrote in message news:<··························@posting.google.com>...
> I would like to figure out within a macro whether a variable is
> lexically bound.  In particular, I would like to be able to use a
> global value for a symbol if a lexical binding isn't available.
> 
> Here is the macro I'd like to write:
> 
> (defmacro use-lexical-then global (sym &environment e)
>   (if (lexical-binding-exists sym e) sym
>      `(symbol-value ',sym))

What does this achieve over:

  (defmacro use-lexical-then-global (sym)
    sym)

Normal evaluation of a symbol already does what you want, just ``throw
it into the path of the evaluator'', as it is sometimes said.

That is to say, the expression X basically means (SYMBOL-VALUE 'X)
when there is no lexical binding for X.

Whatever problem you are having likely has nothing to do with getting
the above macro to work.

> I don't want to use dynamic variables because of the debugging
> problems they can introduce, so defvar, defparameter, declaim/proclaim
> special won't work for me.  Defconstant won't work because it will
> prevent me from reusing symbols in lexical bindings.  I have a large
> number of symbols I'd like to define in the global environment.

Aha, so your real issue is that you want a global environment that can
be lexically shadowed when you want it to, kind of like definitions in
Scheme or other languages. Which is different from the Lisp behavior
of special variables, whose local rebinding keeps them special.

But your ``use-lexical-then-global'' macro won't do it, regardless of
the quality of information you have in the environment parameter.

The trick is to use globally defined symbol macros:
DEFINE-SYMBOL-MACRO. These can expand to some ``hidden'' variable that
holds the actual value. When you rebind the symbol used as the macro
name, the symbol is nicely reused in a lexical way, and the hidden
variable is unaffected.

  (defvar *xyz* 42)
  (define-symbol-macro xyz *xyz*)

The *xyz* variable, our ``hidden'' variable, is by convention nicely
named with asterisks so that it's clear to everyone that it's special,
and nobody expects to be able to make lexical bindings for that
symbol.

The binding of XYZ is to a macro; wherever XYZ occurs, it expands to
*XYZ*. This macro binding can be lexically shadowed. So effectively,
such macros constitute that global lexical environment that can be
locally shadowed.

> * * * *
> 
> Example:
> 
> (setf a 'global)

This is wrong since you don't have a binding for A to assign to. You
need that DEFPARAMETER or DEFVAR for this SETF to be well-defined
according to ANSI Lisp.

> (defun lexical ()
>   (let ((a 'lexical))
>      (use-lexical-then-global a)))

Once you have set up A as a special variable, nothing you can do in
the macro will convince Lisp that A is a lexical variable! The
expression A simply means (SYMBOL-VALUE 'A) and that's that, so the
two possible expansions coming out of the macro are equivalent.

A good ``reality check'' is to try to write a piece of code manually
in place of the macro, which will give you the behavior you are
looking for out of that macro. If *you*, the human programmer, cannot
find a way to do to that, you won't be able to make your macro do
that.  :)
From: Kalle Olavi Niemitalo
Subject: Re: Determining lexically bound variables inside a macro
Date: 
Message-ID: <87hdrgxaiv.fsf@Astalo.kon.iki.fi>
···@ashi.footprints.net (Kaz Kylheku) writes:

> That is to say, the expression X basically means (SYMBOL-VALUE 'X)
> when there is no lexical binding for X.

That is a nonstandard extension, if X has not been declared special.