From: Eli Bendersky
Subject: defvar affecting captured closure variables ?
Date: 
Message-ID: <1191603060.718152.144820@d55g2000hsg.googlegroups.com>
Hello,

Consider this code:

(defun printer (val)
  (lambda () (format t "~a~%" val)))

(setq printer-of-10 (printer 10))
(funcall printer-of-10)

It prints "10" as expected. However, consider this code:

(defun printer (val)
  (lambda () (format t "~a~%" val)))

(setq printer-of-10 (printer 10))
(defvar val 12)
(funcall printer-of-10)

It prints 12 ! This defies my understanding of how the dynamic &
lexical scoping rules of CL interact.

How does this work ? defvar seems to be very dangerous (I found it out
the hard way - this small example is a summary of code I was pouring
over for the past 2 hours - such a bug is horribly difficult to
find!).

Thanks in advance,
Eli

From: Ari Johnson
Subject: Re: defvar affecting captured closure variables ?
Date: 
Message-ID: <m2tzp5qz4e.fsf@nibbler.theari.com>
Eli Bendersky <······@gmail.com> writes:

> Hello,
>
> Consider this code:
>
> (defun printer (val)
>   (lambda () (format t "~a~%" val)))
>
> (setq printer-of-10 (printer 10))
> (funcall printer-of-10)
>
> It prints "10" as expected. However, consider this code:
>
> (defun printer (val)
>   (lambda () (format t "~a~%" val)))
>
> (setq printer-of-10 (printer 10))
> (defvar val 12)
> (funcall printer-of-10)
>
> It prints 12 ! This defies my understanding of how the dynamic &
> lexical scoping rules of CL interact.
>
> How does this work ? defvar seems to be very dangerous (I found it out
> the hard way - this small example is a summary of code I was pouring
> over for the past 2 hours - such a bug is horribly difficult to
> find!).
>
> Thanks in advance,
> Eli

OpenMCL and SBCL both give me 10.  Which implementation are you using?
From: Eli Bendersky
Subject: Re: defvar affecting captured closure variables ?
Date: 
Message-ID: <1191603643.801567.316820@w3g2000hsg.googlegroups.com>
On Oct 5, 6:56 pm, Ari Johnson <·········@gmail.com> wrote:
> Eli Bendersky <······@gmail.com> writes:
> > Hello,
>
> > Consider this code:
>
> > (defun printer (val)
> >   (lambda () (format t "~a~%" val)))
>
> > (setq printer-of-10 (printer 10))
> > (funcall printer-of-10)
>
> > It prints "10" as expected. However, consider this code:
>
> > (defun printer (val)
> >   (lambda () (format t "~a~%" val)))
>
> > (setq printer-of-10 (printer 10))
> > (defvar val 12)
> > (funcall printer-of-10)
>
> > It prints 12 ! This defies my understanding of how the dynamic &
> > lexical scoping rules of CL interact.
>
> > How does this work ? defvar seems to be very dangerous (I found it out
> > the hard way - this small example is a summary of code I was pouring
> > over for the past 2 hours - such a bug is horribly difficult to
> > find!).
>
> > Thanks in advance,
> > Eli
>
> OpenMCL and SBCL both give me 10.  Which implementation are you using?

CLISP v2.41 (on Windows)
From: Raffael Cavallaro
Subject: Re: defvar affecting captured closure variables ?
Date: 
Message-ID: <2007100513072850878-raffaelcavallaro@pasdespamsilvousplaitmaccom>
On 2007-10-05 13:00:43 -0400, Eli Bendersky <······@gmail.com> said:

> CLISP v2.41 (on Windows)

Interpreted or compiled?
From: Eli Bendersky
Subject: Re: defvar affecting captured closure variables ?
Date: 
Message-ID: <1191604154.481790.65920@50g2000hsm.googlegroups.com>
On Oct 5, 7:07 pm, Raffael Cavallaro <················@pas-d'espam-
s'il-vous-plait-mac.com> wrote:
> On 2007-10-05 13:00:43 -0400, Eli Bendersky <······@gmail.com> said:
>
> > CLISP v2.41 (on Windows)
>
> Interpreted or compiled?


Interpreted
From: Rainer Joswig
Subject: Re: defvar affecting captured closure variables ?
Date: 
Message-ID: <joswig-2884CA.19130505102007@news-europe.giganews.com>
In article <························@w3g2000hsg.googlegroups.com>,
 Eli Bendersky <······@gmail.com> wrote:

> On Oct 5, 6:56 pm, Ari Johnson <·········@gmail.com> wrote:
> > Eli Bendersky <······@gmail.com> writes:
> > > Hello,
> >
> > > Consider this code:
> >
> > > (defun printer (val)
> > >   (lambda () (format t "~a~%" val)))
> >
> > > (setq printer-of-10 (printer 10))
> > > (funcall printer-of-10)
> >
> > > It prints "10" as expected. However, consider this code:
> >
> > > (defun printer (val)
> > >   (lambda () (format t "~a~%" val)))
> >
> > > (setq printer-of-10 (printer 10))
> > > (defvar val 12)
> > > (funcall printer-of-10)
> >
> > > It prints 12 ! This defies my understanding of how the dynamic &
> > > lexical scoping rules of CL interact.
> >
> > > How does this work ? defvar seems to be very dangerous (I found it out
> > > the hard way - this small example is a summary of code I was pouring
> > > over for the past 2 hours - such a bug is horribly difficult to
> > > find!).
> >
> > > Thanks in advance,
> > > Eli
> >
> > OpenMCL and SBCL both give me 10.  Which implementation are you using?
> 
> CLISP v2.41 (on Windows)

That's an undesirable behavior of CLISP.
It shows a difference in compiled and interpreted code.

Try (compile 'printer) after you have defined printer.
Then you get 10, as expected.

Maybe you want to complain about it on the CLISP mailing list...

Btw., always use the *foo* naming convention on toplevel variables.
Especially those defined by DEFVAR and DEFPARAMETER.

-- 
http://lispm.dyndns.org
From: Eli Bendersky
Subject: Re: defvar affecting captured closure variables ?
Date: 
Message-ID: <1191606327.371756.301770@r29g2000hsg.googlegroups.com>
> That's an undesirable behavior of CLISP.
> It shows a difference in compiled and interpreted code.
>
> Try (compile 'printer) after you have defined printer.
> Then you get 10, as expected.
>
> Maybe you want to complain about it on the CLISP mailing list...

It's not the only issue in which CLISP's compiled and interpreted code
behaves differently. One notable example is tail-call optimization.
They're enabled in compiled but not interpreted mode in CLISP.
However, this is different - while optimizing tail-calls is left by
the standard to the consideration of implementations, it appears that
the behavior in the snippet I presented is plain wrong and goes
against the Common Lisp standard. So it's a bug in CLISP ?

Eli
From: Rainer Joswig
Subject: Re: defvar affecting captured closure variables ?
Date: 
Message-ID: <joswig-F85318.20152305102007@news-europe.giganews.com>
In article <························@r29g2000hsg.googlegroups.com>,
 Eli Bendersky <······@gmail.com> wrote:

> > That's an undesirable behavior of CLISP.
> > It shows a difference in compiled and interpreted code.
> >
> > Try (compile 'printer) after you have defined printer.
> > Then you get 10, as expected.
> >
> > Maybe you want to complain about it on the CLISP mailing list...
> 
> It's not the only issue in which CLISP's compiled and interpreted code
> behaves differently. One notable example is tail-call optimization.
> They're enabled in compiled but not interpreted mode in CLISP.
> However, this is different - while optimizing tail-calls is left by
> the standard to the consideration of implementations,

Not really. In ANSI CL are many things mentioned where
the implementation has freedom to use different implementation
strategies.

But ANSI CL says nothing about optimizing tail-calls. Nada. Nichts.
It doesn't even say that it doesn't say anything. ;-)

> it appears that
> the behavior in the snippet I presented is plain wrong and goes
> against the Common Lisp standard. So it's a bug in CLISP ?

Don't know. Maybe Kent has an opinion?

Even if it is not a bug, it is undesirable, IMHO. It is desirable
that there are as little differences between interpreted
and compiled code.

> 
> Eli

-- 
http://lispm.dyndns.org
From: Michael Livshin
Subject: Re: defvar affecting captured closure variables ?
Date: 
Message-ID: <878x6hzag4.fsf@colinux.kakpryg.net.cmm>
Eli Bendersky <······@gmail.com> writes:

> it appears that the behavior in the snippet I presented is plain
> wrong and goes against the Common Lisp standard. So it's a bug in
> CLISP ?

no, it does not go against the standard.  the valid ways of
implementing an interpreter are many; including, for instance,
recompiling every function before calling it.

there also are other ways to thwart an interpreter.  like, for
instance, interestingly redefining macros which are used by
interpreted functions.

interpreters are thwartable.  it's a feature.

there is a reason why the more well-behaved sub-species of special
variables evolved those cute *earmuffs* on their names, and some
implementations warn you about the less well-behaved sub-species.
feel free to do business with the *earmuffed* specials, citizen, and
report any violations to the authorities.

cheers,
--m
From: Duane Rettig
Subject: Re: defvar affecting captured closure variables ?
Date: 
Message-ID: <o0r6k988qp.fsf@gemini.franz.com>
Michael Livshin <······@cmm.kakpryg.net> writes:

> Eli Bendersky <······@gmail.com> writes:
>
>> it appears that the behavior in the snippet I presented is plain
>> wrong and goes against the Common Lisp standard. So it's a bug in
>> CLISP ?
>
> no, it does not go against the standard.  the valid ways of
> implementing an interpreter are many; including, for instance,
> recompiling every function before calling it.

This isn't a question of compiled vs interpreted; it is a question of
when a variable is closed over.  Section 3.1.4 of the spec gives the
clear example: the variable is closed over (that is, its binding is
captured) when the example function two-funs is entered (and not when
the resultant closure is funcalled). Note that in the OP's example the
equivalent of two-funs is printer, and when it is entered, val not
declared special (it is only declared (globally) special via the
defvar after the call to printer).  Any attempt to interpret var as
special after this point is changing the binding, which is supposed to
have been captured.

So the demonstrated behavior is a bug. Here's an example of a lisp
with an interpreter that does the correct thing (Allegro CL, of
course):

CL-USER(1): (defun printer (val)
              (lambda () (format t "~a~%" val)))
PRINTER
CL-USER(2): (setq printer-of-10 (printer 10))
#<Interpreted Closure (:INTERNAL PRINTER) @ #x30730692>
CL-USER(3): (defvar val 12)
VAL
CL-USER(4): (funcall printer-of-10)
10
NIL
CL-USER(5): (setq fresh-printer-of-10 (printer 10))
#<Interpreted Closure (:INTERNAL PRINTER) @ #x3073146a>
CL-USER(6): (funcall fresh-printer-of-10)
12
NIL
CL-USER(7): 

Note that the second closure from the call to printer-of-10 is
created _after_ the defvar, and so here it demonstrates the difference
between the interpreter and the compiler, which gives this result:

CL-USER(1): (defun printer (val)
              (lambda () (format t "~a~%" val)))
PRINTER
CL-USER(2): (compile *)
PRINTER
NIL
NIL
CL-USER(3): (setq printer-of-10 (printer 10))
#<Closure (:INTERNAL PRINTER 0) @ #x307357da>
CL-USER(4): (defvar val 12)
VAL
CL-USER(5): (funcall printer-of-10)
10
NIL
CL-USER(6): (setq fresh-printer-of-10 (printer 10))
#<Closure (:INTERNAL PRINTER 0) @ #x30735b7a>
CL-USER(7): (funcall fresh-printer-of-10)
10
NIL
CL-USER(8): 

-- 
Duane Rettig    ·····@franz.com    Franz Inc.  http://www.franz.com/
555 12th St., Suite 1450               http://www.555citycenter.com/
Oakland, Ca. 94607        Phone: (510) 452-2000; Fax: (510) 452-0182   
From: Eli Bendersky
Subject: Re: defvar affecting captured closure variables ?
Date: 
Message-ID: <1191606541.321598.318740@19g2000hsx.googlegroups.com>
> Btw., always use the *foo* naming convention on toplevel variables.
> Especially those defined by DEFVAR and DEFPARAMETER.

It appears that defvar has some gotchas because it gives the variable
dynamic scope. Isn't it a better idea to avoid it completely if I have
no intentions to use dynamic scope[*] ?

As I replacement I can use a "setf". I.e. on the top level:

(setf some-var 5)

Declares a variable some-var and assigns 5 to it, but the scope of
some-var is not dymic. Right ?

[*] Which seems to be 99.99% of the time.

Eli
From: Rainer Joswig
Subject: Re: defvar affecting captured closure variables ?
Date: 
Message-ID: <joswig-078CF7.20084105102007@news-europe.giganews.com>
In article <························@19g2000hsx.googlegroups.com>,
 Eli Bendersky <······@gmail.com> wrote:

> > Btw., always use the *foo* naming convention on toplevel variables.
> > Especially those defined by DEFVAR and DEFPARAMETER.
> 
> It appears that defvar has some gotchas because it gives the variable
> dynamic scope. Isn't it a better idea to avoid it completely if I have
> no intentions to use dynamic scope[*] ?
> 
> As I replacement I can use a "setf". I.e. on the top level:
> 
> (setf some-var 5)
> 
> Declares a variable some-var and assigns 5 to it, but the scope of
> some-var is not dymic. Right ?

The effect of   (setf some-var 5)  is undefined in ANSI CL.
At least one implementation (CMUCL) used to make those
special, too (which I find ugly).

The typical Lisp style is:

* define top-level variables with DEFVAR or DEFPARAMETER .
  Name them *foo* . Be aware that they are special
  variables.

* Otherwise avoid top-level variables.

> 
> [*] Which seems to be 99.99% of the time.
> 
> Eli

-- 
http://lispm.dyndns.org
From: Chris Russell
Subject: Re: defvar affecting captured closure variables ?
Date: 
Message-ID: <1191672391.362605.262170@19g2000hsx.googlegroups.com>
On 5 Oct, 18:13, Rainer Joswig <······@lisp.de> wrote:
> In article <························@w3g2000hsg.googlegroups.com>,
>  Eli Bendersky <······@gmail.com> wrote:
>
>
>
> > On Oct 5, 6:56 pm, Ari Johnson <·········@gmail.com> wrote:
> > > Eli Bendersky <······@gmail.com> writes:
> > > > Hello,
>
> > > > Consider this code:
>
> > > > (defun printer (val)
> > > >   (lambda () (format t "~a~%" val)))
>
> > > > (setq printer-of-10 (printer 10))
> > > > (funcall printer-of-10)
>
> > > > It prints "10" as expected. However, consider this code:
>
> > > > (defun printer (val)
> > > >   (lambda () (format t "~a~%" val)))
>
> > > > (setq printer-of-10 (printer 10))
> > > > (defvar val 12)
> > > > (funcall printer-of-10)
>
> > > > It prints 12 ! This defies my understanding of how the dynamic &
> > > > lexical scoping rules of CL interact.
>
> > > > How does this work ? defvar seems to be very dangerous (I found it out
> > > > the hard way - this small example is a summary of code I was pouring
> > > > over for the past 2 hours - such a bug is horribly difficult to
> > > > find!).
>
> > > > Thanks in advance,
> > > > Eli
>
> > > OpenMCL and SBCL both give me 10.  Which implementation are you using?
>
> > CLISP v2.41 (on Windows)
>
> That's an undesirable behavior of CLISP.
> It shows a difference in compiled and interpreted code.
>
> Try (compile 'printer) after you have defined printer.
> Then you get 10, as expected.
>
However, if you do this you should get different results when you
redefine and compile it a second time after the defvar has been made,
which can lead to endless fun in the debugger.
From: Kent M Pitman
Subject: Re: defvar affecting captured closure variables ?
Date: 
Message-ID: <u4ph4xi0f.fsf@nhplace.com>
Chris Russell <·····················@gmail.com> writes:

> > Try (compile 'printer) after you have defined printer.
> > Then you get 10, as expected.
> >
> However, if you do this you should get different results when you
> redefine and compile it a second time after the defvar has been made,
> which can lead to endless fun in the debugger.

Right.  I'm glad someone mentioned this, since I was going to point this
out and my posts always run irritatingly longer than nice little summaries
like this.

This is the reason we usually use *'s around the variable.  It signals
your intent to always be special, not just after a certain point.  And
always not using *'s for lexical is equally important.
From: Eli Bendersky
Subject: Re: defvar affecting captured closure variables ?
Date: 
Message-ID: <1191782385.643473.178170@v3g2000hsg.googlegroups.com>
On Oct 5, 7:13 pm, Rainer Joswig <······@lisp.de> wrote:
> In article <························@w3g2000hsg.googlegroups.com>,
>  Eli Bendersky <······@gmail.com> wrote:
>
>
>
> > On Oct 5, 6:56 pm, Ari Johnson <·········@gmail.com> wrote:
> > > Eli Bendersky <······@gmail.com> writes:
> > > > Hello,
>
> > > > Consider this code:
>
> > > > (defun printer (val)
> > > >   (lambda () (format t "~a~%" val)))
>
> > > > (setq printer-of-10 (printer 10))
> > > > (funcall printer-of-10)
>
> > > > It prints "10" as expected. However, consider this code:
>
> > > > (defun printer (val)
> > > >   (lambda () (format t "~a~%" val)))
>
> > > > (setq printer-of-10 (printer 10))
> > > > (defvar val 12)
> > > > (funcall printer-of-10)
>
> > > > It prints 12 ! This defies my understanding of how the dynamic &
> > > > lexical scoping rules of CL interact.
>
> > > > How does this work ? defvar seems to be very dangerous (I found it out
> > > > the hard way - this small example is a summary of code I was pouring
> > > > over for the past 2 hours - such a bug is horribly difficult to
> > > > find!).
>
> > > > Thanks in advance,
> > > > Eli
>
> > > OpenMCL and SBCL both give me 10.  Which implementation are you using?
>
> > CLISP v2.41 (on Windows)
>
> That's an undesirable behavior of CLISP.
> It shows a difference in compiled and interpreted code.
>
> Try (compile 'printer) after you have defined printer.
> Then you get 10, as expected.
>
> Maybe you want to complain about it on the CLISP mailing list...
>

I tried. Re-posted exactly the same message, which:

""
has been rejected by the list moderator.  The moderator gave the
following reason for rejecting your request:

"Your message was rejected: it contains html.
http://clisp.cons.org/impnotes/faq.html#faq-rejected "

Any questions or comments should be directed to the list administrator
at:
""

This isn't friendly. Sigh. Anyone from the clisp maillist here in the
n.g. ?

P.S. HTML ?!?!
From: Raffael Cavallaro
Subject: Re: defvar affecting captured closure variables ?
Date: 
Message-ID: <200710071749178930-raffaelcavallaro@pasdespamsilvousplaitmaccom>
On 2007-10-07 14:39:45 -0400, Eli Bendersky <······@gmail.com> said:

> This isn't friendly. Sigh. Anyone from the clisp maillist here in the
> n.g. ?
> 
> P.S. HTML ?!?!

No, it isn't friendly to send html email to a text-only mailing list. 
Not all mailing list subscribers want to receive html email. Send a 
plain text email.
From: Barry Margolin
Subject: Re: defvar affecting captured closure variables ?
Date: 
Message-ID: <barmar-E457F6.21422005102007@comcast.dca.giganews.com>
In article <························@d55g2000hsg.googlegroups.com>,
 Eli Bendersky <······@gmail.com> wrote:

> Hello,
> 
> Consider this code:
> 
> (defun printer (val)
>   (lambda () (format t "~a~%" val)))
> 
> (setq printer-of-10 (printer 10))
> (funcall printer-of-10)
> 
> It prints "10" as expected. However, consider this code:
> 
> (defun printer (val)
>   (lambda () (format t "~a~%" val)))
> 
> (setq printer-of-10 (printer 10))
> (defvar val 12)
> (funcall printer-of-10)
> 
> It prints 12 ! This defies my understanding of how the dynamic &
> lexical scoping rules of CL interact.
> 
> How does this work ? defvar seems to be very dangerous (I found it out
> the hard way - this small example is a summary of code I was pouring
> over for the past 2 hours - such a bug is horribly difficult to
> find!).

Are you sure you didn't execute the DEFVAR *before* you defined PRINTER?  
That would proclaim VAL special, and this would affect all functions 
defined thereafter.

-- 
Barry Margolin, ······@alum.mit.edu
Arlington, MA
*** PLEASE post questions in newsgroups, not directly to me ***
*** PLEASE don't copy me on replies, I'll read them in the group ***
From: Rainer Joswig
Subject: Re: defvar affecting captured closure variables ?
Date: 
Message-ID: <joswig-2DA2EA.09334506102007@news-europe.giganews.com>
In article <····························@comcast.dca.giganews.com>,
 Barry Margolin <······@alum.mit.edu> wrote:

> In article <························@d55g2000hsg.googlegroups.com>,
>  Eli Bendersky <······@gmail.com> wrote:
> 
> > Hello,
> > 
> > Consider this code:
> > 
> > (defun printer (val)
> >   (lambda () (format t "~a~%" val)))
> > 
> > (setq printer-of-10 (printer 10))
> > (funcall printer-of-10)
> > 
> > It prints "10" as expected. However, consider this code:
> > 
> > (defun printer (val)
> >   (lambda () (format t "~a~%" val)))
> > 
> > (setq printer-of-10 (printer 10))
> > (defvar val 12)
> > (funcall printer-of-10)
> > 
> > It prints 12 ! This defies my understanding of how the dynamic &
> > lexical scoping rules of CL interact.
> > 
> > How does this work ? defvar seems to be very dangerous (I found it out
> > the hard way - this small example is a summary of code I was pouring
> > over for the past 2 hours - such a bug is horribly difficult to
> > find!).
> 
> Are you sure you didn't execute the DEFVAR *before* you defined PRINTER?  
> That would proclaim VAL special, and this would affect all functions 
> defined thereafter.

Fresh CLISP started on my Mac:


RJMBP:~ joswig$ clisp
  i i i i i i i       ooooo    o        ooooooo   ooooo   ooooo
  I I I I I I I      8     8   8           8     8     o  8    8
  I  \ `+' /  I      8         8           8     8        8    8
   \  `-+-'  /       8         8           8      ooooo   8oooo
    `-__|__-'        8         8           8           8  8
        |            8     o   8           8     o     8  8
  ------+------       ooooo    8oooooo  ooo8ooo   ooooo   8

Welcome to GNU CLISP 2.41 (2006-10-13) <http://clisp.cons.org/>

Copyright (c) Bruno Haible, Michael Stoll 1992, 1993
Copyright (c) Bruno Haible, Marcus Daniels 1994-1997
Copyright (c) Bruno Haible, Pierpaolo Bernardi, Sam Steingold 1998
Copyright (c) Bruno Haible, Sam Steingold 1999-2000
Copyright (c) Sam Steingold, Bruno Haible 2001-2006

Type :h and hit Enter for context help.

[1]> (defun printer (val)
  (lambda () (format t "~a~%" val)))
PRINTER
[2]> (setq printer-of-10 (printer 10))
#<FUNCTION :LAMBDA NIL (FORMAT T "~a~%" VAL)>
[3]> (defvar val 12)
VAL
[4]> (funcall printer-of-10)
12
NIL
[5]>

-- 
http://lispm.dyndns.org
From: Barry Margolin
Subject: Re: defvar affecting captured closure variables ?
Date: 
Message-ID: <barmar-B68AC1.23020006102007@comcast.dca.giganews.com>
In article <····························@news-europe.giganews.com>,
 Rainer Joswig <······@lisp.de> wrote:

> [1]> (defun printer (val)
>   (lambda () (format t "~a~%" val)))
> PRINTER
> [2]> (setq printer-of-10 (printer 10))
> #<FUNCTION :LAMBDA NIL (FORMAT T "~a~%" VAL)>
> [3]> (defvar val 12)
> VAL
> [4]> (funcall printer-of-10)
> 12
> NIL
> [5]>

The it's definitely a bug in CLISP.  DEFVAR proclaims a variable 
special, but it isn't supposed to perform time travel.  PRINTER was 
defined before the declaration was in effect, so VAL was a lexical 
variable at the time, and should be closed over properly.

Does this happen if you put everything in a file and compile it?

-- 
Barry Margolin, ······@alum.mit.edu
Arlington, MA
*** PLEASE post questions in newsgroups, not directly to me ***
*** PLEASE don't copy me on replies, I'll read them in the group ***
From: Rainer Joswig
Subject: Re: defvar affecting captured closure variables ?
Date: 
Message-ID: <joswig-9DBD92.10290308102007@news-europe.giganews.com>
In article <····························@comcast.dca.giganews.com>,
 Barry Margolin <······@alum.mit.edu> wrote:

> In article <····························@news-europe.giganews.com>,
>  Rainer Joswig <······@lisp.de> wrote:
> 
> > [1]> (defun printer (val)
> >   (lambda () (format t "~a~%" val)))
> > PRINTER
> > [2]> (setq printer-of-10 (printer 10))
> > #<FUNCTION :LAMBDA NIL (FORMAT T "~a~%" VAL)>
> > [3]> (defvar val 12)
> > VAL
> > [4]> (funcall printer-of-10)
> > 12
> > NIL
> > [5]>
> 
> The it's definitely a bug in CLISP.  DEFVAR proclaims a variable 
> special, but it isn't supposed to perform time travel.  PRINTER was 
> defined before the declaration was in effect, so VAL was a lexical 
> variable at the time, and should be closed over properly.
> 
> Does this happen if you put everything in a file and compile it?

Let's see:

[1]> (compile-file "clisp-test.lisp")
;; Compiling file /private/tmp/clisp-test.lisp ...
WARNING in 5 5 (SETQ PRINTER-OF-10 (PRINTER 10))-2 in line 5 :
PRINTER-OF-10 is neither declared nor bound,
it will be treated as if it were declared SPECIAL.
WARNING in 9 9 (PRINT (FUNCALL PRINTER-OF-10))-4 in line 9 :
PRINTER-OF-10 is neither declared nor bound,
it will be treated as if it were declared SPECIAL.
;; Wrote file /private/tmp/clisp-test.fas
The following special variables were not defined:
 PRINTER-OF-10
0 errors, 2 warnings
#P"/private/tmp/clisp-test.fas" ;
2 ;
2
[2]> (load *)
;; Loading file /private/tmp/clisp-test.fas ...
10

NIL 
;; Loaded file /private/tmp/clisp-test.fas
T

So it gives 10 when compiled...

-- 
http://lispm.dyndns.org
From: Rainer Joswig
Subject: Re: defvar affecting captured closure variables ?
Date: 
Message-ID: <joswig-B5B579.00521911102007@news-europe.giganews.com>
In article <····························@comcast.dca.giganews.com>,
 Barry Margolin <······@alum.mit.edu> wrote:

> In article <····························@news-europe.giganews.com>,
>  Rainer Joswig <······@lisp.de> wrote:
> 
> > [1]> (defun printer (val)
> >   (lambda () (format t "~a~%" val)))
> > PRINTER
> > [2]> (setq printer-of-10 (printer 10))
> > #<FUNCTION :LAMBDA NIL (FORMAT T "~a~%" VAL)>
> > [3]> (defvar val 12)
> > VAL
> > [4]> (funcall printer-of-10)
> > 12
> > NIL
> > [5]>
> 
> The it's definitely a bug in CLISP.

CLISP maintainers think differently:

  http://clisp.podval.org/impnotes/faq.html#faq-scope

>  DEFVAR proclaims a variable 
> special, but it isn't supposed to perform time travel.  PRINTER was 
> defined before the declaration was in effect, so VAL was a lexical 
> variable at the time, and should be closed over properly.
> 
> Does this happen if you put everything in a file and compile it?
From: Duane Rettig
Subject: Re: defvar affecting captured closure variables ?
Date: 
Message-ID: <o0k5puxt6c.fsf@gemini.franz.com>
Rainer Joswig <······@lisp.de> writes:

> In article <····························@comcast.dca.giganews.com>,
>  Barry Margolin <······@alum.mit.edu> wrote:
>
>> In article <····························@news-europe.giganews.com>,
>>  Rainer Joswig <······@lisp.de> wrote:
>> 
>> > [1]> (defun printer (val)
>> >   (lambda () (format t "~a~%" val)))
>> > PRINTER
>> > [2]> (setq printer-of-10 (printer 10))
>> > #<FUNCTION :LAMBDA NIL (FORMAT T "~a~%" VAL)>
>> > [3]> (defvar val 12)
>> > VAL
>> > [4]> (funcall printer-of-10)
>> > 12
>> > NIL
>> > [5]>
>> 
>> The it's definitely a bug in CLISP.
>
> CLISP maintainers think differently:
>
>   http://clisp.podval.org/impnotes/faq.html#faq-scope

This interpretation ignores section 3.1.4, which is very specific
about capturing the binding of the variable, which must occur at the
time the outer function is called.  The explanation is trying to
rationalize the evaluation of a special binding, but because the
binding had to be captured at the time printer is called, when val was
a lexical (and thus the binding that is captured must be lexical)
the interpreter is not conforming.  The attempt to distinguish
compiled code from interpreted by describing the elimination of
the _symbol_ val is a red herring; it is not the _symbol_ that must be
captured in a closure, but the _binding_, and those don't change
(i.e. as to the binding being lexical) between interpreted and
compiled code, whether the variable's symbol remains or not.

Also, the explanation tries to call on section 3.2.2.3 for
justification, but the statement (i.e. "Special proclamations for
dynamic variables must be made in the compilation environment.") is
taken out of context in two ways:

1. The whole of section 3.2.2.3 is labeled at the top as a set of
constraints with the stated purpose of _minimizing_ differences
between the interpreter and the compiler.  It is rather audacious to
call on parts of such a section to justify a difference between the
compiler and the interpreter.

2. The section where the sentence is found describes _compilation_
semantics, not interpreter semantics.  It has nothing whatsoever to do
with the interpreter (see its context: 3.2.2).  Thus, the user code is
not non-conforming for the stated reason (I also believe that the code
is conforming, because no other restrictions are placed on interpreted
code; the rules of 3.2.2.3 are intended to apply to the _compiler_ to
bring it closer to the behavior of the interpreter, not the other way
around.

-- 
Duane Rettig    ·····@franz.com    Franz Inc.  http://www.franz.com/
555 12th St., Suite 1450               http://www.555citycenter.com/
Oakland, Ca. 94607        Phone: (510) 452-2000; Fax: (510) 452-0182   
From: Pascal Costanza
Subject: Re: defvar affecting captured closure variables ?
Date: 
Message-ID: <5mnjhtFe8uhuU1@mid.individual.net>
Eli Bendersky wrote:
> Hello,
> 
> Consider this code:
> 
> (defun printer (val)
>   (lambda () (format t "~a~%" val)))
> 
> (setq printer-of-10 (printer 10))
> (funcall printer-of-10)
> 
> It prints "10" as expected. However, consider this code:
> 
> (defun printer (val)
>   (lambda () (format t "~a~%" val)))
> 
> (setq printer-of-10 (printer 10))
> (defvar val 12)
> (funcall printer-of-10)
> 
> It prints 12 ! This defies my understanding of how the dynamic &
> lexical scoping rules of CL interact.
> 
> How does this work ? defvar seems to be very dangerous 

defvar and defparameter introduce global variables, and such global 
variables are always special variables (i.e., dynamically scoped). 
Common Lisp doesn't have lexical global variables (although there is a 
common idiom to fake them via symbol macros - google for deflex or 
defglobal).

Follow the naming conventions: global special variables are always like 
*val*, never like val - so they, by convention, always start with, and 
end in, an asterisk. Local lexical variables typically don't have a 
special naming convention.

BTW, your (setq printer-of-10 (printer 10)) seems to work on an 
undefined variable. That's undefined behavior in Common Lisp. Better 
introduce such variables properly, with defvar or defparameter.


Pascal

-- 
My website: http://p-cos.net
Common Lisp Document Repository: http://cdr.eurolisp.org
Closer to MOP & ContextL: http://common-lisp.net/project/closer/
From: Eli Bendersky
Subject: Re: defvar affecting captured closure variables ?
Date: 
Message-ID: <1191648559.382303.8150@50g2000hsm.googlegroups.com>
On Oct 5, 10:02 pm, Pascal Costanza <····@p-cos.net> wrote:
> Eli Bendersky wrote:
> > Hello,
>
> > Consider this code:
>
> > (defun printer (val)
> >   (lambda () (format t "~a~%" val)))
>
> > (setq printer-of-10 (printer 10))
> > (funcall printer-of-10)
>
> > It prints "10" as expected. However, consider this code:
>
> > (defun printer (val)
> >   (lambda () (format t "~a~%" val)))
>
> > (setq printer-of-10 (printer 10))
> > (defvar val 12)
> > (funcall printer-of-10)
>
> > It prints 12 ! This defies my understanding of how the dynamic &
> > lexical scoping rules of CL interact.
>
> > How does this work ? defvar seems to be very dangerous
>
> defvar and defparameter introduce global variables, and such global
> variables are always special variables (i.e., dynamically scoped).
> Common Lisp doesn't have lexical global variables (although there is a
> common idiom to fake them via symbol macros - google for deflex or
> defglobal).
>
> Follow the naming conventions: global special variables are always like
> *val*, never like val - so they, by convention, always start with, and
> end in, an asterisk. Local lexical variables typically don't have a
> special naming convention.
>
> BTW, your (setq printer-of-10 (printer 10)) seems to work on an
> undefined variable. That's undefined behavior in Common Lisp. Better
> introduce such variables properly, with defvar or defparameter.
>

I realize this can be solved by following the naming convention.
However, I think you'll have to agree that this is not a clean
solution. Using a naming convention to avoid wrong behavior of the
environment shouldn't be the panacea.

Sometimes then playing with complex code I find it useful to define
many global vars on the top environment. Adding earmuffs to each is
tiresome because these vars are not true globals of my code which I
want to mark, but just temporary places to hold stuff while I'm
testing how the code works.

Eli
From: Geoff Wozniak
Subject: Re: defvar affecting captured closure variables ?
Date: 
Message-ID: <1191653162.183880.101290@19g2000hsx.googlegroups.com>
On Oct 6, 1:29 am, Eli Bendersky <······@gmail.com> wrote:
> I realize this can be solved by following the naming convention.
> However, I think you'll have to agree that this is not a clean
> solution. Using a naming convention to avoid wrong behavior of the
> environment shouldn't be the panacea.
>

I'd wager that fixing this problem without a naming convention would
be (or have been, at the time it was decided on) a lot uglier.

> Sometimes then playing with complex code I find it useful to define
> many global vars on the top environment. Adding earmuffs to each is
> tiresome because these vars are not true globals of my code which I
> want to mark, but just temporary places to hold stuff while I'm
> testing how the code works.
>

Perhaps "global lexicals" are something that you could integrate into
your workflow.

http://www.tfeb.org/lisp/toys.html#GLEX
From: Pascal Costanza
Subject: Re: defvar affecting captured closure variables ?
Date: 
Message-ID: <5mp78fFea4ikU1@mid.individual.net>
Eli Bendersky wrote:
> On Oct 5, 10:02 pm, Pascal Costanza <····@p-cos.net> wrote:

>> Follow the naming conventions: global special variables are always like
>> *val*, never like val - so they, by convention, always start with, and
>> end in, an asterisk. Local lexical variables typically don't have a
>> special naming convention.
> 
> I realize this can be solved by following the naming convention.
> However, I think you'll have to agree that this is not a clean
> solution. Using a naming convention to avoid wrong behavior of the
> environment shouldn't be the panacea.

I agree that this is not the best solution. ISLISP does this better by 
having a clear separation between the dynamic environment and the 
lexical environments. [1] The parameters that you find in some Scheme 
implementations also do this better.

Nevertheless, the naming convention works surprisingly well, such that 
for all practical purposes, this isn't an issue. Except for Common Lisp 
beginners, nobody seems to have any problems with this.

> Sometimes then playing with complex code I find it useful to define
> many global vars on the top environment. Adding earmuffs to each is
> tiresome because these vars are not true globals of my code which I
> want to mark, but just temporary places to hold stuff while I'm
> testing how the code works.

Well, for those you don't need any naming convention, of course, because 
that is throwaway code anyway. You just have to keep in mind that such 
globals are always special. Or, again, use any of those deflex or 
defglobal macros, which give you global lexicals.

Here is a rough sketch:

(defmacro defglobal (name value)
   (let ((var (copy-symbol name)))
     `(progn
        (defvar ,var ,value)
        (define-symbol-macro ,name ,var)
        ',var)))

Untested, but should work.


Pascal

[1] ISLISP is, more or less, a subset of Common Lisp with a few minor 
improvements.

-- 
My website: http://p-cos.net
Common Lisp Document Repository: http://cdr.eurolisp.org
Closer to MOP & ContextL: http://common-lisp.net/project/closer/
From: Kent M Pitman
Subject: Re: defvar affecting captured closure variables ?
Date: 
Message-ID: <u8x6gxi2o.fsf@nhplace.com>
Pascal Costanza <··@p-cos.net> writes:

> Here is a rough sketch:
> 
> (defmacro defglobal (name value)
>    (let ((var (copy-symbol name)))
>      `(progn
>         (defvar ,var ,value)
>         (define-symbol-macro ,name ,var)
>         ',var)))

Cosmetic, but I think this should return:   ',name 

> Untested, but should work.

Yeah, I didn't test it either, but it looks like one possible way to
do it.  (I might have added a property on name which pointed to the
copied symbol so it was available for debugging, but that's a detail.)
In short, though, it looks to me, too, like it will do what you intend.

BUT I thought I had heard him suggest he wanted to have things be
special, but not pervasively special.  My response (in another post)
went to that question.

Both are things people commonly ask for.  Both are trivial to achieve.
So if the global non-pervasive special is what he wants, he should see
my other message nearby in this conversation tree.
From: Pascal Costanza
Subject: Re: defvar affecting captured closure variables ?
Date: 
Message-ID: <5mqcodFeoq3uU1@mid.individual.net>
Kent M Pitman wrote:
> Pascal Costanza <··@p-cos.net> writes:
> 
>> Here is a rough sketch:
>>
>> (defmacro defglobal (name value)
>>    (let ((var (copy-symbol name)))
>>      `(progn
>>         (defvar ,var ,value)
>>         (define-symbol-macro ,name ,var)
>>         ',var)))
> 
> Cosmetic, but I think this should return:   ',name 

Thanks, you're right.

>> Untested, but should work.
> 
> Yeah, I didn't test it either, but it looks like one possible way to
> do it.  (I might have added a property on name which pointed to the
> copied symbol so it was available for debugging, but that's a detail.)

That's not necessary, AFAICT, because you can always do a macroexpand on 
the name.

> In short, though, it looks to me, too, like it will do what you intend.
> 
> BUT I thought I had heard him suggest he wanted to have things be
> special, but not pervasively special.  My response (in another post)
> went to that question.

Right.


Pascal

-- 
My website: http://p-cos.net
Common Lisp Document Repository: http://cdr.eurolisp.org
Closer to MOP & ContextL: http://common-lisp.net/project/closer/
From: Rob Warnock
Subject: Re: defvar affecting captured closure variables ?
Date: 
Message-ID: <ULudnU2GXMc515XanZ2dnUVZ_hqdnZ2d@speakeasy.net>
Pascal Costanza  <··@p-cos.net> wrote:
+---------------
| Or, again, use any of those deflex or defglobal macros,
| which give you global lexicals. Here is a rough sketch:
|   (defmacro defglobal (name value)
|      (let ((var (copy-symbol name)))
|        `(progn
|           (defvar ,var ,value)
|           (define-symbol-macro ,name ,var)
|           ',var)))
| Untested, but should work.
+---------------

It does, but I (slightly) prefer my version[1] of DEFLEX
because it doesn't leave uninterned symbols dangling about,
the backing symbol is in the same package as the symbol macro,
and like other DEFxxx forms it returns the symbol being defined:

    > (defglobal foo 13)

    #:FOO
    > (deflex bar 47)

    BAR
    > (describe 'foo)

    FOO is an internal symbol in the COMMON-LISP-USER package.
    It is a symbol macro with expansion: #:FOO.
    > (describe 'bar)

    BAR is an internal symbol in the COMMON-LISP-USER package.
    It is a symbol macro with expansion: *STORAGE-FOR-DEFLEX-VAR-BAR*.
    > 

But as a tutorial proof of concept, your DEFGLOBAL seems fine.


-Rob

[1] My current version:

(defmacro deflex (var val &optional (doc nil docp))    
  "Define a top level (global) lexical VAR with initial value VAL,
  which is assigned unconditionally as with DEFPARAMETER. If a DOC
  string is provided, it is attached to both the name |VAR| and the
  name *STORAGE-FOR-DEFLEX-VAR-|VAR|* as a documentation string of
  kind 'VARIABLE. The new VAR will have lexical scope and thus may be
  shadowed by LET bindings without affecting its dynamic (global) value."
  (let* ((s0 (symbol-name '#:*storage-for-deflex-var-))
         (s1 (symbol-name var))
         (s2 (symbol-name '#:*))
         (s3 (symbol-package var))
         (backing-var (intern (concatenate 'string s0 s1 s2) s3)))
    ;; Note: The DEFINE-SYMBOL-MACRO must be the last thing we do so
    ;; that the value of the form is the symbol VAR.
    (if docp
      `(progn
         (defparameter ,backing-var ,val ,doc)
         (setf (documentation ',var 'variable) ,doc)
         (define-symbol-macro ,var ,backing-var))
      `(progn
         (defparameter ,backing-var ,val)
         (define-symbol-macro ,var ,backing-var)))))

-----
Rob Warnock			<····@rpw3.org>
627 26th Avenue			<URL:http://rpw3.org/>
San Mateo, CA 94403		(650)572-2607
From: Pascal Costanza
Subject: Re: defvar affecting captured closure variables ?
Date: 
Message-ID: <5mrp1sFertocU1@mid.individual.net>
Rob Warnock wrote:
> Pascal Costanza  <··@p-cos.net> wrote:
> +---------------
> | Or, again, use any of those deflex or defglobal macros,
> | which give you global lexicals. Here is a rough sketch:
> |   (defmacro defglobal (name value)
> |      (let ((var (copy-symbol name)))
> |        `(progn
> |           (defvar ,var ,value)
> |           (define-symbol-macro ,name ,var)
> |           ',var)))
> | Untested, but should work.
> +---------------
> 
> It does, but I (slightly) prefer my version[1] of DEFLEX
> because it doesn't leave uninterned symbols dangling about,
> the backing symbol is in the same package as the symbol macro,
> and like other DEFxxx forms it returns the symbol being defined:

The latter was a bug in my version that Kent already noticed. Of course, 
the return value should be ',name, not ',var.

Why are the first two characteristics (no uninterned symbols, same 
package as the symbol macro) important?


Pascal

-- 
My website: http://p-cos.net
Common Lisp Document Repository: http://cdr.eurolisp.org
Closer to MOP & ContextL: http://common-lisp.net/project/closer/
From: Kent M Pitman
Subject: Re: defvar affecting captured closure variables ?
Date: 
Message-ID: <uwsty29h2.fsf@nhplace.com>
Pascal Costanza <··@p-cos.net> writes:

> Why are the first two characteristics (no uninterned symbols, same
> package as the symbol macro) important?

I never use uninterned symbols either.  They have issues with becoming
external data in file compilation when you've macroexpanded them.  If
all your uses are in one file, you may not see it, because I think
implementations try to eq-ify gensyms across a target stream, but if 
you were to write them to a separate file, there is no way for those
two files, upon reloading, to realize they need to coalesce--their
symbol identities will, of necessity, diverge and they will no longer
be cooperating.
From: Thomas F. Burdick
Subject: Re: defvar affecting captured closure variables ?
Date: 
Message-ID: <1191776500.394673.110380@k79g2000hse.googlegroups.com>
On Oct 7, 6:08 pm, Kent M Pitman <······@nhplace.com> wrote:
> Pascal Costanza <····@p-cos.net> writes:
> > Why are the first two characteristics (no uninterned symbols, same
> > package as the symbol macro) important?
>
> I never use uninterned symbols either.  They have issues with becoming
> external data in file compilation when you've macroexpanded them.  If
> all your uses are in one file, you may not see it, because I think
> implementations try to eq-ify gensyms across a target stream, but if
> you were to write them to a separate file, there is no way for those
> two files, upon reloading, to realize they need to coalesce--their
> symbol identities will, of necessity, diverge and they will no longer
> be cooperating.

Also, simply re-evaluating the deflex form should probably not make
all references from that point on be disjoint from all references
compiled up to that point.
From: Rob Warnock
Subject: Re: defvar affecting captured closure variables ?
Date: 
Message-ID: <-YSdncWJO4sYTpTanZ2dnUVZ_rOpnZ2d@speakeasy.net>
Thomas F. Burdick <········@gmail.com> wrote:
+---------------
| Kent M Pitman <······@nhplace.com> wrote:
| > Pascal Costanza <····@p-cos.net> writes:
| > > Why are the first two characteristics (no uninterned symbols, same
| > > package as the symbol macro) important?
| >
| > I never use uninterned symbols either.  They have issues with becoming
| > external data in file compilation when you've macroexpanded them.  If
| > all your uses are in one file, you may not see it, because I think
| > implementations try to eq-ify gensyms across a target stream, but if
| > you were to write them to a separate file, there is no way for those
| > two files, upon reloading, to realize they need to coalesce--their
| > symbol identities will, of necessity, diverge and they will no longer
| > be cooperating.
| 
| Also, simply re-evaluating the deflex form should probably not make
| all references from that point on be disjoint from all references
| compiled up to that point.
+---------------

Yup, that's precisely the main reason I don't like uninterned
symbols as "backing variables" for lexicals. In what I use DEFLEX
for, it is *very* common for the DEFLEX form to be re-evaluated
or even changed slightly and *then* re-evaluated [as when editing
a source file and re-loading it into the running image]. The latter
is also why my DEFLEX expands into a DEFPARAMETER of the backing
variable rather than a DEFVAR. I *want* the (re)evaluation of a
(possibly-modified) DEFLEX to be able to change the variable and
all extant references to it.

To answer Pascal's other question, having the same package as
the symbol macro has two benefits: (1) If you're *not* using
uninterned symbols at backing variables, having the backing
variable match the packages of the symbol macro avoids collisions
between "DEFLEX FOO" forms in different packages. [This was a bug
in an earlier version of my DEFLEX which Adam Warner helpfully
pointed out in mid-2005.]  And, (2) if you *do* use uninterned
symbols at backing variables, then when debugging you sometimes
don't know to *which* DEFLEX form a "#:FOO" refers (which package
the symbol macro came from), since all you have is the debugger
error message that's complaining about something in the expansion,
"#:FOO".


-Rob

-----
Rob Warnock			<····@rpw3.org>
627 26th Avenue			<URL:http://rpw3.org/>
San Mateo, CA 94403		(650)572-2607
From: Pascal Costanza
Subject: Re: defvar affecting captured closure variables ?
Date: 
Message-ID: <5muh9uFfe6b7U1@mid.individual.net>
Rob Warnock wrote:
> Thomas F. Burdick <········@gmail.com> wrote:
> +---------------
> | Kent M Pitman <······@nhplace.com> wrote:
> | > Pascal Costanza <····@p-cos.net> writes:
> | > > Why are the first two characteristics (no uninterned symbols, same
> | > > package as the symbol macro) important?
> | >
> | > I never use uninterned symbols either.  They have issues with becoming
> | > external data in file compilation when you've macroexpanded them.  If
> | > all your uses are in one file, you may not see it, because I think
> | > implementations try to eq-ify gensyms across a target stream, but if
> | > you were to write them to a separate file, there is no way for those
> | > two files, upon reloading, to realize they need to coalesce--their
> | > symbol identities will, of necessity, diverge and they will no longer
> | > be cooperating.
> | 
> | Also, simply re-evaluating the deflex form should probably not make
> | all references from that point on be disjoint from all references
> | compiled up to that point.
> +---------------
> 
> Yup, that's precisely the main reason I don't like uninterned
> symbols as "backing variables" for lexicals. In what I use DEFLEX
> for, it is *very* common for the DEFLEX form to be re-evaluated
> or even changed slightly and *then* re-evaluated [as when editing
> a source file and re-loading it into the running image]. The latter
> is also why my DEFLEX expands into a DEFPARAMETER of the backing
> variable rather than a DEFVAR. I *want* the (re)evaluation of a
> (possibly-modified) DEFLEX to be able to change the variable and
> all extant references to it.

OK

> To answer Pascal's other question, having the same package as
> the symbol macro has two benefits: (1) If you're *not* using
> uninterned symbols at backing variables, having the backing
> variable match the packages of the symbol macro avoids collisions
> between "DEFLEX FOO" forms in different packages. [This was a bug
> in an earlier version of my DEFLEX which Adam Warner helpfully
> pointed out in mid-2005.]  And, (2) if you *do* use uninterned
> symbols at backing variables, then when debugging you sometimes
> don't know to *which* DEFLEX form a "#:FOO" refers (which package
> the symbol macro came from), since all you have is the debugger
> error message that's complaining about something in the expansion,
> "#:FOO".

OK, I understand this better now.

However, I am not totally convinced yet. Putting such generated symbols 
in the same package has the potential of leading to other kinds of 
nameclashes, so I don't like that.

What I have come up in another (related) setting was that I interned 
such generated symbols all in a dedicated package for generated symbols, 
but the symbol names thereof are composed of their symbol package names 
and their symbol names. So, say, for a symbol P1::S1 you get something 
like SPECIAL-PACKAGE::|P1::S1|.

There is still the possibility of a nameclash - the name of the special 
package could accidentally clash with some other package. But that's a 
single point of failure which is easy to detect and to workaround. At 
least that's how I see it at the moment.

Do you agree that that's a viable approach? Too complicated? Still too 
simple?


Pascal

-- 
My website: http://p-cos.net
Common Lisp Document Repository: http://cdr.eurolisp.org
Closer to MOP & ContextL: http://common-lisp.net/project/closer/
From: Rob Warnock
Subject: Re: defvar affecting captured closure variables ?
Date: 
Message-ID: <ENidnbyE_8g2bpfanZ2dnUVZ_q6hnZ2d@speakeasy.net>
Pascal Costanza  <··@p-cos.net> wrote:
+---------------
| Rob Warnock wrote:
| > To answer Pascal's other question, having the same package as
| > the symbol macro has two benefits: (1) If you're *not* using
| > uninterned symbols at backing variables, having the backing
| > variable match the packages of the symbol macro avoids collisions
| > between "DEFLEX FOO" forms in different packages. [This was a bug
| > in an earlier version of my DEFLEX which Adam Warner helpfully
| > pointed out in mid-2005.]  ...
| 
| OK, I understand this better now.
| 
| However, I am not totally convinced yet. Putting such generated symbols 
| in the same package has the potential of leading to other kinds of 
| nameclashes, so I don't like that.
+---------------

Ahhh, but you should note *what* the symbols look like that
my version of DEFLEX generates:

    > (defpackage :bar)

    #<The BAR package, 0/9 internal, 0/2 external>
    > (deflex foo 13)

    FOO
    > (deflex bar::foo 57)

    BAR::FOO
    > (describe 'foo)

    FOO is an internal symbol in the COMMON-LISP-USER package.
    It is a symbol macro with expansion: *STORAGE-FOR-DEFLEX-VAR-FOO*.
    > (describe 'bar::foo)

    FOO is an internal symbol in the BAR package.
    It is a symbol macro with expansion: BAR::*STORAGE-FOR-DEFLEX-VAR-FOO*.
    >

Using this scheme, conflicts are (IMHO) *very* unlikely!!  ;-}  ;-}

+---------------
| What I have come up in another (related) setting was that I interned 
| such generated symbols all in a dedicated package for generated symbols, 
| but the symbol names thereof are composed of their symbol package names 
| and their symbol names. So, say, for a symbol P1::S1 you get something 
| like SPECIAL-PACKAGE::|P1::S1|.
...
| Do you agree that that's a viable approach? Too complicated?
| Still too simple?
+---------------

Certainly viable, but too complicated.
Mine is simpler and less ugly. (IMHO. YMMV.)


-Rob

-----
Rob Warnock			<····@rpw3.org>
627 26th Avenue			<URL:http://rpw3.org/>
San Mateo, CA 94403		(650)572-2607
From: Pascal Costanza
Subject: Re: defvar affecting captured closure variables ?
Date: 
Message-ID: <5n3chkFffkk3U1@mid.individual.net>
Rob Warnock wrote:
> Pascal Costanza  <··@p-cos.net> wrote:
> +---------------
> | Rob Warnock wrote:
> | > To answer Pascal's other question, having the same package as
> | > the symbol macro has two benefits: (1) If you're *not* using
> | > uninterned symbols at backing variables, having the backing
> | > variable match the packages of the symbol macro avoids collisions
> | > between "DEFLEX FOO" forms in different packages. [This was a bug
> | > in an earlier version of my DEFLEX which Adam Warner helpfully
> | > pointed out in mid-2005.]  ...
> | 
> | OK, I understand this better now.
> | 
> | However, I am not totally convinced yet. Putting such generated symbols 
> | in the same package has the potential of leading to other kinds of 
> | nameclashes, so I don't like that.
> +---------------
> 
> Ahhh, but you should note *what* the symbols look like that
> my version of DEFLEX generates:
> 
>     > (defpackage :bar)
> 
>     #<The BAR package, 0/9 internal, 0/2 external>
>     > (deflex foo 13)
> 
>     FOO
>     > (deflex bar::foo 57)
> 
>     BAR::FOO
>     > (describe 'foo)
> 
>     FOO is an internal symbol in the COMMON-LISP-USER package.
>     It is a symbol macro with expansion: *STORAGE-FOR-DEFLEX-VAR-FOO*.
>     > (describe 'bar::foo)
> 
>     FOO is an internal symbol in the BAR package.
>     It is a symbol macro with expansion: BAR::*STORAGE-FOR-DEFLEX-VAR-FOO*.
>     >
> 
> Using this scheme, conflicts are (IMHO) *very* unlikely!!  ;-}  ;-}

OK, thank you very much. This makes sense.



Pascal

-- 
My website: http://p-cos.net
Common Lisp Document Repository: http://cdr.eurolisp.org
Closer to MOP & ContextL: http://common-lisp.net/project/closer/
From: Pascal Costanza
Subject: Re: defvar affecting captured closure variables ?
Date: 
Message-ID: <5mugvmFf1ks0U1@mid.individual.net>
Kent M Pitman wrote:
> Pascal Costanza <··@p-cos.net> writes:
> 
>> Why are the first two characteristics (no uninterned symbols, same
>> package as the symbol macro) important?
> 
> I never use uninterned symbols either.  They have issues with becoming
> external data in file compilation when you've macroexpanded them.  If
> all your uses are in one file, you may not see it, because I think
> implementations try to eq-ify gensyms across a target stream, but if 
> you were to write them to a separate file, there is no way for those
> two files, upon reloading, to realize they need to coalesce--their
> symbol identities will, of necessity, diverge and they will no longer
> be cooperating.

OK, right. I actually had that issue before, and had to come up with a 
scheme where I use interned symbols as well. Thanks for reminding me.


Pascal

-- 
My website: http://p-cos.net
Common Lisp Document Repository: http://cdr.eurolisp.org
Closer to MOP & ContextL: http://common-lisp.net/project/closer/
From: Scott Burson
Subject: Re: defvar affecting captured closure variables ?
Date: 
Message-ID: <1192033205.347462.161580@50g2000hsm.googlegroups.com>
On Oct 7, 9:08 am, Kent M Pitman <······@nhplace.com> wrote:
> Pascal Costanza <····@p-cos.net> writes:
> > Why are the first two characteristics (no uninterned symbols, same
> > package as the symbol macro) important?
>
> I never use uninterned symbols either.  They have issues with becoming
> external data in file compilation when you've macroexpanded them.  If
> all your uses are in one file, you may not see it, because I think
> implementations try to eq-ify gensyms across a target stream, but if
> you were to write them to a separate file, there is no way for those
> two files, upon reloading, to realize they need to coalesce--their
> symbol identities will, of necessity, diverge and they will no longer
> be cooperating.

CLISP seems not to maintain identity of uninterned symbols even within
a file -- at least, I worked around this (arguable) bug in it at one
point.

-- Scott
From: Eli Bendersky
Subject: Re: defvar affecting captured closure variables ?
Date: 
Message-ID: <1191783668.013021.173800@g4g2000hsf.googlegroups.com>
> [1] My current version:
>
> (defmacro deflex (var val &optional (doc nil docp))
>   "Define a top level (global) lexical VAR with initial value VAL,
>   which is assigned unconditionally as with DEFPARAMETER. If a DOC
>   string is provided, it is attached to both the name |VAR| and the
>   name *STORAGE-FOR-DEFLEX-VAR-|VAR|* as a documentation string of
>   kind 'VARIABLE. The new VAR will have lexical scope and thus may be
>   shadowed by LET bindings without affecting its dynamic (global) value."
>   (let* ((s0 (symbol-name '#:*storage-for-deflex-var-))
>          (s1 (symbol-name var))
>          (s2 (symbol-name '#:*))
>          (s3 (symbol-package var))
>          (backing-var (intern (concatenate 'string s0 s1 s2) s3)))
>     ;; Note: The DEFINE-SYMBOL-MACRO must be the last thing we do so
>     ;; that the value of the form is the symbol VAR.
>     (if docp
>       `(progn
>          (defparameter ,backing-var ,val ,doc)
>          (setf (documentation ',var 'variable) ,doc)
>          (define-symbol-macro ,var ,backing-var))
>       `(progn
>          (defparameter ,backing-var ,val)
>          (define-symbol-macro ,var ,backing-var)))))
>

Rob, this looks interesting. Can I consider it as a viable replacement
for defparameter at the top level for defining lexical globals ? Is it
portable across all implementation ?

Eli
From: Rob Warnock
Subject: Re: defvar affecting captured closure variables ?
Date: 
Message-ID: <NbudnTvbLv4Sf5TanZ2dnUVZ_judnZ2d@speakeasy.net>
Eli Bendersky  <······@gmail.com> wrote:
+---------------
| > [1] My current version:
| >
| > (defmacro deflex (var val &optional (doc nil docp))
...[trimmed]...
| 
| Rob, this looks interesting. Can I consider it as a viable replacement
| for defparameter at the top level for defining lexical globals ?
+---------------

I certainly use it that way quite liberally in my own code,
especially for variables that are going to be set or referenced
at the REPL.

But I also still use DEFPARAMETER (with traditional *earmuffs*
on the variable) for variables that are ever going to be dynamically
rebound, since DEFLEX is *NOT* compatible with the DEFVAR/SHADOWABLE
approach[1] mentioned elsewhere in this thread. That is, if you do
(DEFVAR/SHADOWABLE FOO 13) or (DEFINE-SYMBOL-MACRO FOO (SYMBOL-VALUE 'FOO)),
then you can both LET-bind FOO lexically *and* reference the global
value using (LOCALLY (DECLARE (SPECIAL FOO)) FOO), e.g.:

    > (defvar/shadowable foo 13)

    FOO
    > (defun foo () foo)

    FOO
    > (let ((foo 27))
	(locally (declare (special foo)) (setf foo 53))
	(list foo (foo) (locally (declare (special foo)) foo)))

    (27 53 53)
    > foo

    53
    > (locally (declare (special foo)) foo)

    53
    > 

But if you (DEFLEX FOO 13), then (LOCALLY (DECLARE (SPECIAL FOO)) FOO)
is going to get the value cell for the symbol FOO itself, *not*
the global value of the backing variable for FOO. This can lead
to considerable confusion:

    > (deflex foo 13)

    FOO
    > (defun foo () foo)

    FOO
    > (let ((foo 27))
	(locally (declare (special foo)) (setf foo 53))
	(list foo (foo) (locally (declare (special foo)) foo)))

    (27 13 53)
    > foo

    13
    > (locally (declare (special foo)) foo)

    53
    > 

+---------------
| Is it portable across all implementation ?
+---------------

As far as I know, yes, it should be. It uses only ANSI CL constructs.

However, be aware that in CMUCL-19d and earlier there was a bug
in macro expansion that in some cases [e.g., "INCF FOO", but not
just "FOO" or "SETF FOO"] caused references to symbol macros that
had been shadowed by lexical variables to expand to the symbol macro
expansion instead of refering to the lexical variable. [This would
affect all symbol macros, including both DEFLEX and DEFVAR/SHADOWABLE.]
This bug was fixed in the CVS for "code/eval.lisp" as of Revision
1.43 2006-12-19, but to my knowledge there is not yet an official
"cmucl-19d-patch-NNN" for this. I posted a patch here in article
<·········································@speakeasy.net> on 2006-12-18
[which also works in CMUCL-19c], or you can just extract the
routine MACROEXPAND-1 from the CVS tree version of "eval.lisp" at
<http://common-lisp.net/cgi-bin/viewcvs.cgi/*checkout*/src/
code/eval.lisp?rev=1.43&root=cmucl>.


-Rob

[1] In case you missed it, I was proposing this if one wanted
to use the "shadowable global special" approach:

    (defmacro defvar/shadowable (var &optional value (doc nil docp))
      `(progn
	 (setf (symbol-value ',var) ,value)
	 ,@(when docp
	     (list `(setf (documentation ',var 'variable) ,doc)))
	 (define-symbol-macro ,var (symbol-value ',var))))

-----
Rob Warnock			<····@rpw3.org>
627 26th Avenue			<URL:http://rpw3.org/>
San Mateo, CA 94403		(650)572-2607
From: Kent M Pitman
Subject: Re: defvar affecting captured closure variables ?
Date: 
Message-ID: <uhcl4xjla.fsf@nhplace.com>
Eli Bendersky <······@gmail.com> writes:

> I realize this can be solved by following the naming convention.
> However, I think you'll have to agree that this is not a clean
> solution. Using a naming convention to avoid wrong behavior of the
> environment shouldn't be the panacea.

It's probably hard to see, but global specials used to be the DEFAULT
in Lisp and lexical variables were on a VERY short list of one of the
most controversial choices that Common Lisp designers did, risking
alienating the entire community.  It was one of the few cases where
those of us on the CLTL design team basically said something very close
to "look, we don't care that many users are afraid of lexicals, they
are still the right thing and we're going to do this transition because
it will make programs more correct".  Globally pervasive specials were
an accommodation to the fact that this is exactly how SPECIAL used to
work in older lisps, and we needed to preserve a body of legacy programs.

At the time, the *foo* naming convention was not widely used in most
legacy applications, so we couldn't depend on it.

I think the reason we didn't ultimately repair it to work differently
is that, in fact, true reasons for new specials (as opposed to the
many accidental reasons for legacy specials) are scarce, and making an
elaborate new facility didn't seem worth it.

> Sometimes then playing with complex code I find it useful to define
> many global vars on the top environment. Adding earmuffs to each is
> tiresome because these vars are not true globals of my code which I
> want to mark, but just temporary places to hold stuff while I'm
> testing how the code works.

But it makes your code more readable to others, and many compilers are
willing to warn you about mismatches between notation and
binding/setting/reference.  

In practice, I agree it's baroque in notational style, and it's not
presented very clearly in many teaching texts. (I don't count ANSI CL,
nor CLHS, as a teaching text.)

I don't agree it's broken, in the sense that it has a clear design and
the design serves a specific purpose that was not subject to debate.

If it were new design without such constraints, I'd certainly do it
differently.

But among the things worth worrying about in the language, it's hardly
worth mentioning.  Once one learns how to do it, it's pretty easy to
reliably use correctly.

Incidentally, ignore the advice about global lexicals.  For better or
worse, I'm pretty sure all you want is this:

 (define-symbol-macro foo (symbol-value 'foo))

Don't forget to use explicit special bindings when you really mean to 
bind these globals, since otherwise you'll get a lexical foo when you
bind foo.
From: Pascal Costanza
Subject: Re: defvar affecting captured closure variables ?
Date: 
Message-ID: <5mptrhFemg3vU1@mid.individual.net>
Kent M Pitman wrote:

>  (define-symbol-macro foo (symbol-value 'foo))

That's sweet!


Pascal

-- 
My website: http://p-cos.net
Common Lisp Document Repository: http://cdr.eurolisp.org
Closer to MOP & ContextL: http://common-lisp.net/project/closer/
From: Alex Mizrahi
Subject: Re: defvar affecting captured closure variables ?
Date: 
Message-ID: <4707d152$0$90268$14726298@news.sunsite.dk>
(message (Hello 'Kent)
(you :wrote  :on '(06 Oct 2007 13:02:09 -0400))
(

 KMP> Incidentally, ignore the advice about global lexicals.  For better or
 KMP> worse, I'm pretty sure all you want is this:

 KMP>  (define-symbol-macro foo (symbol-value 'foo))

by the way, why global lexicals by standard should be defined in such 
non-trivial way (i find it quite interesting that symbol-macro can be 
shadowed by LET, but works fine with SETQ), while all implementation (i 
know/tested) actually support more straightforward and easy way?

i mean:
CG-USER(1): (setf (symbol-value 'foo) 5)
5
CG-USER(2): foo
5
CG-USER(4): (funcall (let ((foo 6)) (lambda () foo)))
6
CG-USER(5): foo
5

clearly that's a global lexical, right?
or another way: (locally (declare (special foo)) (setq foo 5))

this also arises quite interesting question about unbound variables.
is that a well-define thing according to standard? if i write some symbol 
that is not declared as variable (lexical or special one), conforming 
implementation should complain, right?

seems pretty clear, but.. after you do perfectly legal (setf (symbol-value 
'foo) 5), all implementations i know define FOO as some magical variable --  
it's not lexical or special, nevertheless it's definitely a variable, since 
it's bound, and indeed there are no complains from implementations when such 
variables are used.

so, all implementations implement some "magical" kind of variables that are 
not described by standard, and those variables are not defined by some 
magical construct, but by a perfectly legal code according to standard.

i believe that this means that all Common Lisp implementations are not 
conformant to the Common Lisp standard.
possibly some language lawyers can find a way how they are conformant (i.e. 
find a paragraph which allows implementations to introduce some magical 
variables at their wish?), but still clearly it's very unpleasant situation, 
even language basics are ill-defined.

there are some other arising controversies:
Function BOUNDP
Returns true if symbol is bound; otherwise, returns false.
bound adj., v.t. 1. adj. having an associated denotation in a binding. ``The 
variables named by a let are bound within its body.'' See unbound. 2. adj. 
having a local binding which shadows[2] another.

so clearly symbol is bound when it is a (special, or lexical?) variable.
but, with aforementioned experiments:

* (boundp 'foo)
T

so, (setf (symbol-value 'foo) 5) creates a binding, but it's not specified 
in standard?

and certainly there is a well-known issues with SETQ -- behaviour is 
undefined if it's given a non-variable. maybe most implementations behave 
like if is set-quote (they set symbol-function), but other do not..

so, i wonder is there some rationale for introducing non-trivial 
symbol-macro rules instead of legalizing behaviour that is already present 
in all (most) implementations, and resolving many controversies at same 
time?

)
(With-best-regards '(Alex Mizrahi) :aka 'killer_storm)
"Hanging In The Balance Of Deceit And Blasphemy") 
From: Juho Snellman
Subject: Re: defvar affecting captured closure variables ?
Date: 
Message-ID: <slrnfgfm7m.pba.jsnell@vasara.proghammer.com>
On 2007-10-06, Alex Mizrahi <········@users.sourceforge.net> wrote:
> i mean:
> CG-USER(1): (setf (symbol-value 'foo) 5)
> 5
> CG-USER(2): foo
> 5
> CG-USER(4): (funcall (let ((foo 6)) (lambda () foo)))
> 6
> CG-USER(5): foo
> 5
>
> clearly that's a global lexical, right?

No, it clearly isn't:

(setf (symbol-value 'foo) 1)

(defun bar ()
  foo)

(let ((foo 2))
  (declare (special foo))
  ;; Oops, returns 2.
  (bar))

This also shows why the simple symbol-macro from FOO => (SYMBOL-VALUE 'FOO)
isn't a good way of defining "global lexicals". Better to do expand store
the value in a different symbol than the one naming the global lexical.
Something along the lines of:

(progn
  (defvar #1=#:gensym1234)
  (define-symbol-macro foo (symbol-value '#1#)))

-- 
Juho Snellman
From: Rainer Joswig
Subject: Re: defvar affecting captured closure variables ?
Date: 
Message-ID: <joswig-0F7FB2.20314006102007@news-europe.giganews.com>
In article <·························@news.sunsite.dk>,
 "Alex Mizrahi" <········@users.sourceforge.net> wrote:

> (message (Hello 'Kent)
> (you :wrote  :on '(06 Oct 2007 13:02:09 -0400))
> (
> 

...

> so, i wonder is there some rationale for introducing non-trivial 
> symbol-macro rules instead of legalizing behaviour that is already present 
> in all (most) implementations, and resolving many controversies at same 
> time?

Good point.

> )
> (With-best-regards '(Alex Mizrahi) :aka 'killer_storm)
> "Hanging In The Balance Of Deceit And Blasphemy")

-- 
http://lispm.dyndns.org
From: ·········@math.ufl.edu
Subject: Re: defvar affecting captured closure variables ?
Date: 
Message-ID: <1191722019.321161.216470@57g2000hsv.googlegroups.com>
    It has already been remarked that this can be confusing to
beginners.  (I count myself in their number.)  Here is one
additional data-point of confusion, for me.

In a fresh CLISP, with interpreted code, I have inserted a
`(setq val ... )' before the `defvar':


    /--------------------------------------------------------\

    MacOS$   clisp -norc
     ...

    [1]> (defun printer (val)   (lambda () (format t "~a~%" val)))
    PRINTER

    [2]> (setq printer-of-10 (printer 10))
    #<FUNCTION :LAMBDA NIL (FORMAT T "~a~%" VAL)>

    [3]> (funcall printer-of-10)
    10
    NIL

    [4]> (setq val
314159265358979323846264338327950288419716939937510)
    314159265358979323846264338327950288419716939937510

    [5]> (funcall printer-of-10)
    10
    NIL

    [6]> (defvar val 12)
    VAL

    [7]> (funcall printer-of-10)
    314159265358979323846264338327950288419716939937510
    NIL

    [8]>

    \________________________________________________________/

If the (setq val 31415...)   has been a coupla-hundred
lines of code back, and the `(funcall printer-of-10)' had
been returning 10, and I then executed `(defvar val 12)'
and now saw `(funcall printer-of-10)'  returning

   314159265358979323846264338327950288419716939937510

I would have been bewildered.  Is it
reasonable/possible/politically-possible to have a second ANSI
standard on top of the first?  It would not contradict the first.
However, places where the first standard left behavior undefined
could be emended by the second standard to: "A conforming
installation MUST signal an error/warning when ... "

--
Prof. Jonathan LF King	 Mathematics dept, Univ. of Florida
            <http://www.math.ufl.edu/~squash/teaching.html>
From: Thomas Bakketun
Subject: Re: defvar affecting captured closure variables ?
Date: 
Message-ID: <5mr0r2Fep3s7U1@mid.individual.net>
 ·········@math.ufl.edu wrote:

>     [1]> (defun printer (val)   (lambda () (format t "~a~%" val)))
>     PRINTER
> 
>     [2]> (setq printer-of-10 (printer 10))
>     #<FUNCTION :LAMBDA NIL (FORMAT T "~a~%" VAL)>
> 
>     [3]> (funcall printer-of-10)
>     10
>     NIL
> 
>     [4]> (setq val
> 314159265358979323846264338327950288419716939937510)
>     314159265358979323846264338327950288419716939937510

This is illegal code, but CLISP (like almost any other implementation)
silently threats is as (setq (symbol-value 'val) ...) when not compiling.
 
>     [5]> (funcall printer-of-10)
>     10
>     NIL
> 
>     [6]> (defvar val 12)
>     VAL

Here defvar will not change the value of val since it already been bound by
the previous setq, but it will proclaim val as globally special, making all
use of val from now on threated as a dynamic variable.

>     [7]> (funcall printer-of-10)
>     314159265358979323846264338327950288419716939937510
>     NIL

To add confusion this is a bug in CLISP, as stated elsewere in this thread.

> Is it
> reasonable/possible/politically-possible to have a second ANSI
> standard on top of the first?  It would not contradict the first.
> However, places where the first standard left behavior undefined
> could be emended by the second standard to: "A conforming
> installation MUST signal an error/warning when ... "

This is more a problem with the implementations than the standard. Uses of
undefined variables can simply be refused or at least always be warned
about. Declaring special variables without *earmuffs* can not be refused,
but issuing a style warning should be ok. 
From: ·········@math.ufl.edu
Subject: Re: defvar affecting captured closure variables ?
Date: 
Message-ID: <1191722032.983594.197830@k79g2000hse.googlegroups.com>
    It has already been remarked that this can be confusing to
beginners.  (I count myself in their number.)  Here is one
additional data-point of confusion, for me.

In a fresh CLISP, with interpreted code, I have inserted a
`(setq val ... )' before the `defvar':


    /--------------------------------------------------------\

    MacOS$   clisp -norc
     ...

    [1]> (defun printer (val)   (lambda () (format t "~a~%" val)))
    PRINTER

    [2]> (setq printer-of-10 (printer 10))
    #<FUNCTION :LAMBDA NIL (FORMAT T "~a~%" VAL)>

    [3]> (funcall printer-of-10)
    10
    NIL

    [4]> (setq val
314159265358979323846264338327950288419716939937510)
    314159265358979323846264338327950288419716939937510

    [5]> (funcall printer-of-10)
    10
    NIL

    [6]> (defvar val 12)
    VAL

    [7]> (funcall printer-of-10)
    314159265358979323846264338327950288419716939937510
    NIL

    [8]>

    \________________________________________________________/

If the (setq val 31415...)   has been a coupla-hundred
lines of code back, and the `(funcall printer-of-10)' had
been returning 10, and I then executed `(defvar val 12)'
and now saw `(funcall printer-of-10)'  returning

   314159265358979323846264338327950288419716939937510

I would have been bewildered.  Is it
reasonable/possible/politically-possible to have a second ANSI
standard on top of the first?  It would not contradict the first.
However, places where the first standard left behavior undefined
could be emended by the second standard to: "A conforming
installation MUST signal an error/warning when ... "

--
Prof. Jonathan LF King	 Mathematics dept, Univ. of Florida
            <http://www.math.ufl.edu/~squash/teaching.html>
From: Kent M Pitman
Subject: Re: defvar affecting captured closure variables ?
Date: 
Message-ID: <umyuvitfi.fsf@nhplace.com>
"Alex Mizrahi" <········@users.sourceforge.net> writes:

> i mean:
> CG-USER(1): (setf (symbol-value 'foo) 5)
> 5
> CG-USER(2): foo
> 5
> CG-USER(4): (funcall (let ((foo 6)) (lambda () foo)))
> 6
> CG-USER(5): foo
> 5
> 
> clearly that's a global lexical, right?

Of course not.  A global lexical is where you get this behavior:

 (define-lexical x 3)
 (defun foo () x)
 (let ((x 4)) (declare (special x)) (foo))
 => 3

CL doesn't have those by default, but they are trivial to implement and
there was code offered in the last day showing it.

You're confusing yourself because you have previously in this implementation
done either (proclaim '(special foo)) 
... or else (defvar foo) or (defparameter foo), which do that implicitly.

Ordinarily, if you do

 (setf (symbol-value 'foo) 5)

you should expect to see 5 and here

 (funcall (let ((foo 6)) (lambda () foo)))
 
you should expect 6.  The 5 result here will come only if you have
proclaimed FOO special.

> this also arises quite interesting question about unbound variables.
> is that a well-define thing according to standard? if i write some symbol 
> that is not declared as variable (lexical or special one), conforming 
> implementation should complain, right?

No, a conforming implementation may do something implementation-defined.

You shouldn't do it without reading what the implementation says will
happen.  The standard is silent to allow implementations to experiment
with possible things here, like global lexicals by default, or by warnings
about having defined nothing, or by making globals default to specials
for debugging.

> seems pretty clear, but.. after you do perfectly legal (setf (symbol-value 
> 'foo) 5), all implementations i know define FOO as some magical variable --  

No implementation I'm familiar with defines foo in any way when you do this.

> it's not lexical or special, nevertheless it's definitely a variable, since 
> it's bound, and indeed there are no complains from implementations when such 
> variables are used.

No, the special variable has been set in all implementations. What you don't
know is whether writing the variable FOO by itself, undeclared, is lexical
or special.  But you definitely know that at toplevel,
(locally (declare (special foo)) foo)
is going to give you 5 after you've done 
(setf (symbol-value 'foo) 5)
in all implementations.
 
> so, all implementations implement some "magical" kind of variables that are 
> not described by standard, and those variables are not defined by some 
> magical construct, but by a perfectly legal code according to standard.

No, all that's happened is that implementations decide which of several 
states you're in, and you just don't know which, partly because they are
not required to agree.

> i believe that this means that all Common Lisp implementations are not 
> conformant to the Common Lisp standard.

I try to avoid telling people they are wrong, but I certainly don't
know of any passage that would make you right.  You'll need to cite
chapter and verse to make me think you're right.  The default state of
the language is not "something is illegal for implementations unless
the spec says otherwise" but in fact just the opposite, that
"implementations are free to do what they want as long as the spec
doesn't tell them otherwise".  The spec is not a thing that constrains
things to only work one way, it is a common area that says that if you
program a certain way, the implementations will try to conform to a
certain meaning.  For things you do outside of what the spec allows
YOU to do, the implementation generally gets freedom to decide how to
treat that.  It's such freedom that allows things like multiprocessing,
and other important features that are not nailed down by the spec.

> possibly some language lawyers can find a way how they are conformant (i.e. 
> find a paragraph which allows implementations to introduce some magical 
> variables at their wish?), but still clearly it's very unpleasant situation, 
> even language basics are ill-defined.

It isn't a language basic.  You aren't supposed to do that.  I don't see
quite what the problem is.  Language basics do not include "doing things
no one has ever told you should work".  They include "doing things you're
told to do".  If there is a text telling you to do this and to expect the
effect of getting a "magic variable", I might think that text needed some
work to keep from misleading anyone.
 
> there are some other arising controversies:
> Function BOUNDP
> Returns true if symbol is bound; otherwise, returns false.
> bound adj., v.t. 1. adj. having an associated denotation in a binding. ``The 
> variables named by a let are bound within its body.'' See unbound. 2. adj. 
> having a local binding which shadows[2] another.
> 
> so clearly symbol is bound when it is a (special, or lexical?) variable.

The symbol FOO is bound (as a special) when it has a value.  But that doesn't
mean it's always accessible by the mere use of its name.  In:

 (let ((x 3))
   (let ((x 4))
     x))

the outer x is bound but not accessible at the point of the x in the
last line, since that variable refers to the inner x.  And in

 (let ((blah 3))
   (declare (special blah))
   (let ((blah 4))
     blah))

the blah on the last line is bound as a special but not directly accessible
in the last line, although it could be if you did
     
 (let ((blah 3))
   (declare (special blah))
   (let ((blah 4))
     (list blah
          (locally (declare (special blah)) blah)
          (symbol-value 'blah))))

which should return (4 3 3).

> but, with aforementioned experiments:
> 
> * (boundp 'foo)
> T
> 
> so, (setf (symbol-value 'foo) 5) creates a binding, but it's not specified 
> in standard?

No, it creates a binding that is specified in the standard.  A special 
binding for FOO. But that doesn't mean it makes the variable FOO available
for use at that particular instant.

> and certainly there is a well-known issues with SETQ -- behaviour is 
> undefined if it's given a non-variable.

The controversial issue is that some implementations auto-define variables
assigned with toplevel SETQ [the undefined case in the standard]
by proclaiming them special.  This confuses a lot of newbies and I recommend
disabling it.

> maybe most implementations behave 
> like if is set-quote (they set symbol-function), but other do not..

They can choose.  To claim something is violating the spec, you have to point
in the spec to the part that says it cannot happen.
 
> so, i wonder is there some rationale for introducing non-trivial 
> symbol-macro rules instead of legalizing behaviour that is already present 
> in all (most) implementations, and resolving many controversies at same 
> time?

Because there was an interest in the community in experimenting, as I 
mentioned, with

 (a) global lexicals
 (b) giving an error if people used toplevel variables free without reference
 (c) having toplevel free references behave defaultly as specials
From: Alex Mizrahi
Subject: Re: defvar affecting captured closure variables ?
Date: 
Message-ID: <4708bcb6$0$90272$14726298@news.sunsite.dk>
(message (Hello 'Kent)
(you :wrote  :on '(06 Oct 2007 21:50:57 -0400))
(

 ??>> clearly that's a global lexical, right?

 KMP> Of course not.  A global lexical is where you get this behavior:

 KMP>  (define-lexical x 3)
 KMP>  (defun foo () x)
 KMP>  (let ((x 4)) (declare (special x)) (foo))
 KMP>  => 3

aha, i see, but for many purposes (e.g. for problems that OP had) it would 
be OK when:

1. variable has some well-defined value if it's not bound lexically. (that 
can be symbol-value, why not?)
2. it can be bound lexically, it's not globally special.

and actually all implementations implement this behaviour.

is it somehow inferior to global lexical behaviour? i can't imagine in which 
cases global lexicals will be better.
it only makes difference with (declare (special x)) as above, which i 
believe is not what people are doing frequently.

 KMP> You're confusing yourself because you have previously in this
 KMP> implementation done either (proclaim '(special foo))
 KMP> ... or else (defvar foo) or (defparameter foo), which do that
 KMP> implicitly.

no, i'm not :)

 ??>> i believe that this means that all Common Lisp implementations are not
 ??>> conformant to the Common Lisp standard.

 KMP> I try to avoid telling people they are wrong, but I certainly don't
 KMP> know of any passage that would make you right.  You'll need to cite
 KMP> chapter and verse to make me think you're right.

having following definitions:

If a form is a symbol that is not a symbol macro, then it is the name of a 
variable, and the value of that variable is returned.
There are three kinds of variables: lexical variables, dynamic variables, 
and constant variables.
An error of type unbound-variable should be signaled if an unbound variable 
is referenced.
unbound variable n. a name that is syntactically plausible as the name of a 
variable but which is not bound in the variable namespace.

i'm assuming that lexical, dynamic, constant and unbound are _only_ states 
in which variable can be.

what will happen if we see a symbol foo, which is not defined as a variable? 
it's clearly not a lexical or constant variable. it doesn't satisfy 
definition of dynamic variable as given in section 3.1.2.1.1.2. so the only 
possibility is that it's an unbound variable, so according to 3.1.2.1.1 
error of type unbound-variable should be signaled.

but as i've being pointed to, in such situations implementations treat 
variable as being being dynamic.

so i believe they either violate section 3.1.2.1.1.2, which quite strictly 
defines what dynamic variable is
 (A variable is a dynamic variable if one of the following conditions hold: 
It is locally declared or globally proclaimed special. ;
It occurs textually within a form that creates a dynamic binding for a 
variable of the same name, and the binding is not shadowed[2] by a form that 
creates a lexical binding of the same variable name. ).

or they violate 3.1.2.1.1 -- when being given an unbound variable, they 
return symbol-value, instead of signalling an error.

can we somehow interpret the standard in a way that implementations would 
still be conforming? i see few ways -- 
1. assume that definition of dynamic variables is not strict and 
implementations can treat variables as dynamic is other cases too.
2. assume that other states of variable are also possible. i.e. in this 
situation implementations think that this is "undefined-but-bound variable", 
so it's not an unbound variable, and can return a value instead of 
signalling an error.

it seems people prefer first form of interpretation, assuming 
implementations can extend dynamic variable definitions. there is also some 
vague quirk: "It is locally declared or globally proclaimed special. ". so, 
maybe, if it was once locally declared special, we can still assume it's 
dynamic even it's outside that declaration?

and it's very easy to make it more clear: either explicitly allow 
implementations to extend definition of dynamic, allow other reaction on 
unbound variable, or allow other states of variables..

)
(With-best-regards '(Alex Mizrahi) :aka 'killer_storm)
"Hanging In The Balance Of Deceit And Blasphemy") 
From: Kent M Pitman
Subject: Re: defvar affecting captured closure variables ?
Date: 
Message-ID: <usl4m26wx.fsf@nhplace.com>
"Alex Mizrahi" <········@users.sourceforge.net> writes:

> If a form is a symbol that is not a symbol macro, then it is the name of a 
> variable, and the value of that variable is returned.
> There are three kinds of variables: lexical variables, dynamic variables, 
> and constant variables.
> [...]
> i'm assuming that lexical, dynamic, constant and unbound are _only_ states 
> in which variable can be.

Sort of.

First, there are only three states it can be, but all implementations
are not required to agree on the same state.  So the fuzzy part in
this case is whether the implementation is required to conform to that
meaning or the user is required to conform to such uses.

Second, there is no way to establish a global lexical in CL, so if you
say the state is lexical, then that means implementations have
considerable latitude for interpreting the meaning of that term in
that context, since it is otherwise plainly undefined what would occur
and the meaning must be supplied by the implementation.

Before you can claim you have a conforming program, you have to do
more than just pick an isolated code fragment that is syntactically
well-formed.  You have to show that all the variables have been given
values, etc., using some conforming method.  I don't think you can say
that, so you are therefore outside the realm of the spec.  At that
point, quoting passages on what is required to occur won't help you.

If you insist on saying magic pixie dust is used here, you can do that,
but it won't get you closer to the truth.  Magic is invoked when people
don't understand things, but the simpler interpretation is that when the
spec says something is not defined, it's not defined.

Your questions seem to me to be taking the form "I can show by my own
hand that everyone else is wrong and I am right and the world must
suddenly and incompatibly change to suit me".  You may not mean this,
but it's how it sounds to me.  I'm trying my best to explain why the
world might work as it does and why the large number of people who
have used this spec have converged on a meaning different than what
you see and expect.  Your options at this point are either to come to
grips with that or to mount a campaign for change.

Note well that I am not discounting the possibility that you may have
seen a bug in some implementation that causes it to behave in a way
that is confusing you further by making you feel the world is even 
weirder than you might perceive it is under my explanation.  Bugs do 
happen and should be reported.

And certainly the spec, being written in English and not formalese,
does have the issue that not all of it is presented in a way that is
uniformly unambiguous or concise.  It was not written from scratch
according to some pre-articulated principles nor was its ultimate
meaning within the choice of me nor any single person to just change
and allege was valid.  So I take responsibility for, and apologize
for, some suboptimal wording choices and outright typographical errors
in various places, but at the same time I think largely (and specifically
in this case) it has been able to withstand a considerable amount of
careful reading with reasonably convergent results, and so I don't think
you're likely to stand the world on its head at this point and I think
an attempt to adopt such a posture will not serve anyone.

> what will happen if we see a symbol foo, which is not defined as a variable? 
> it's clearly not a lexical or constant variable.

How do you know the implementation does not define that all free toplevel
variables are lexical?

We have left this to implementations to resolve in part because
toplevel is often used for two incompatible reasons that only the
implementation can really resolve.  As a matter of style, I suggest
that using unassigned free variables at all is something you should
outright avoid in code in all cases.

But the two incompatible reasons are this:

 * Some toplevel code is bootstrapping code, gaining a foothold for
   a program.  In this code, the code knows it is at toplevel and
   won't usually be much bothered by concerns about specials vs lexicals
   because it usually goes very carefully until it gets things set up,
   and it is (or should be) written by people who know that the least 
   little global assignment will potentially clobber other global 
   assignments done elsewhere (at least in that package) and that care
   in doing such pollution is warranted.  Furthermore, because of 
   portability issues, one must assume a least common denominator of
   possible interpretations, and this leads again to conservative reading
   of semantics.

 * Some toplevel code is just random code people have stripped out of the
   middle of other code for testing.  e.g., I've written
    (defun foo (x)  #'(lambda () x))
   and I want to test that #'(lambda () x) works correctly.  If I type that
   at toplevel and the Lisp interactive system is overzealous in the
   manner you are insisting on, it will see X as a SPECIAL and so if I do
   (setq foo #'(lambda () x)) I won't be testing what foo will be doing.
   If the implementation just treats this silently and doesn't tell me 
   that _I_, not the implementation, have done something weird, then I am
   likely to confuse myself debugging.  If, on the other hand, it implements
   global lexicals, no problem will result.  But the claim was made, at 
   language design time by the vendors, that implementations needed the 
   freedom to choose, and the language gave them that right.

> can we somehow interpret the standard in a way that implementations would 
> still be conforming? i see few ways -- 
> 1. assume that definition of dynamic variables is not strict and 
> implementations can treat variables as dynamic is other cases too.
> 2. assume that other states of variable are also possible. i.e. in this 
> situation implementations think that this is "undefined-but-bound variable", 
> so it's not an unbound variable, and can return a value instead of 
> signalling an error.

Your program is in error so the standard does not apply here.

 You have not created a constant.

 You have not created a dynamic variable.

 You have not created a lexical variable.

That doesn't entitle you to claim there is another kind of variable,
it entitles you to understand what these implementations are telling
you, which is that your program is in error.

You're assuming you have a right to have your program interpreted as
if you've set things up right when you haven't done the foundational
work to do that.

The spec does not require the signaling of a special variable error here
because to do so would be to preclude implementations from having global
lexicals.

But there is no evidence that it intends or requires (nor even
permits, though I haven't thought hard about that case) some new kind
of variable, which is what you seem to be steadfastly suggesting is
the only option.  What it generally intends in the case of gray areas
like this is for implementations to do something reasonable and tell
users what is being done, either interactively or in the documentation.

> it seems people prefer first form of interpretation, assuming 
> implementations can extend dynamic variable definitions. there is also some 
> vague quirk: "It is locally declared or globally proclaimed special. ". so, 
> maybe, if it was once locally declared special, we can still assume it's 
> dynamic even it's outside that declaration?
 
One thing the spec doesn't say is that you can invent mystic new rules
and then expect them to apply.  If an implementation defined that this is
what happened, and if it violated no other passages in the spec, you could
expect such a rule to apply in that implementation.  But your intented rule
is not canonically dictated by the spec, so who's to say someone couldn't
invent a different fanciful interpretation.

The point is that you cannot rely on this not to be the rule (unless there
is a place that says so) but you cannot rely on it to be the rule either.

The spec is, in the end, a text that says little more than "if you
write your code to certain specs the implementation will try to
interpret it according to certain specs".  In my personal individual
opinion, in the case you've cited, you haven't done your part of the
bargain, since you have not assured that you have used a conforming
way to give an initial value to the variable, and so you're outside
the realm of any agreement between you and any implementation that
hasn't given you special dispensation to infer the meaning you're
trying to infer.

> and it's very easy to make it more clear: either explicitly allow 
> implementations to extend definition of dynamic, allow other reaction on 
> unbound variable, or allow other states of variables..

It's not very easy to make it more clear.  To make it more clear could
and probably would cost:

 * years of work
 * hundreds of thousands of dollars of spec-writing time
 * destabilization of the entire language
 * hundreds of thousands, if not millions, of dollars in the community
   fixing code that was gratuitiously broken by change, retooling the
   various vendors, etc.

This is probably the part you're failing to understand.  The standard
is presently in a stable form that is usable as is.  There are only
two options:

 * scrap it and start over
 * live with it

You probably assume these options, but I claim they do not exist:

 * make an intermediate attempt to fix only things that are a problem
 * make only a targeted small set of fixes

ANSI doesn't have a process whereby you can re-open a process on the 
condition of only fixing one bug.  You have to just re-open the process
as a free-for-all, and then you take whatever comes, with no guarantee
of it being better or worse, and certainly with no guarnatee.  You assume
this is all that would be changed because it's all you personally care
about.  But each person has a list of things they'd like changed.  And
there is no guarantee that anyone would like anyone else's changes.

It periodically comes up that we should have a second constitutional
convention in the US, to revise the problems in the constitution.
Many seem small and fixable.  But I have heard people opine, and I
totally agree, that it's too dangerous.  The work to be done is small,
but the point is that now that the system is deployed, people are no
longer starry-eyed people who want the best for all (if indeed it was
ever that), but rather they are people who are microengineering
specific results they want without regard to the public good.  Going
by individual amendments is slow and tedious, but it has mostly kept
out stupid stuff like flag burning amendments because when given due
time to consider such changes, people are good about it.  But if the
wrong people had in their hands the US constitution, they would be
"fixing" those "pesky" problems of free speech, religion, copyright
and free software, etc. in ways that were utterly untested.  The
original one was untested too, and we're basically just lucky it
worked, to the extent it even did.  There's no reason to believe that
we're entitled to have it work, rather than fail instantly.  

Lisp has the same issues.  ANSI CL is not what everyone wants, but the
worlds out there waiting for you if you open its details to debate are
not near neighbors of what you have now.  The right way to start a new
language, IMO, is just to make your own and see if anyone comes.  I
don't think you think you're trying to do that, but I say as a matter
of rigor that as soon as you say "and it's very easy to make it more clear"
that this is what you have suggested.  Because the standard right now is
what it is and cannot be made more or less clear without a formal process
involving new committees, new voting, new editing, new reviewing, etc.

And there is no metric of "small" or "large" in that process that can
be enforced.  During final voting on the standard, someone objected to
the standard on the basis that the credits contained the name of
someone who they didn't think had participated in the way that was
cited.  This was not even a technical objection, I'd allege, at a time
when only technical objections were allowed.  ANSI or NCITS or whoever
it was that we went to to get this resolved said they had no proper
process for figuring out what was and was not a technical objection,
and that we therefore could not dismiss the comment as non-technical,
and that it would be faster to treat it as technical and respond to it
as such than to dismiss it and risk that the standard would be called
into question for failing to address all community feedback.  We
therefore took an extra six months or a year, I can't recall the time,
to just let the public review play out and respond to that matter.
My point here is just to underscore the long timelines, great expense,
and lack of fine-grained control involved in opening the standard to any
kind of public reconsideration.

Also, because standards serve an antitrust service, you cannot open
reconsideration to only addressing the issues that suit one part of
the community but not others.  The treatment needs to be evenhanded, so
again it's not about this one issues, it's about all pending issues.
And back again to your remark "it's very easy to make it more clear",
in my personal opinion, nothing could be farther from the truth.
From: Don Geddis
Subject: Re: defvar affecting captured closure variables ?
Date: 
Message-ID: <877ilybi4h.fsf@geddis.org>
Kent M Pitman <······@nhplace.com> wrote on 07 Oct 2007 13:0:
> "Alex Mizrahi" <········@users.sourceforge.net> writes:
>> and it's very easy to make it more clear
>
> It's not very easy to make it more clear.  To make it more clear could
> and probably would cost:
>  * years of work
>  * hundreds of thousands of dollars of spec-writing time
>  * destabilization of the entire language
>  * hundreds of thousands, if not millions, of dollars in the community
>    fixing code that was gratuitiously broken by change, retooling the
>    various vendors, etc.
> This is probably the part you're failing to understand.  The standard
> is presently in a stable form that is usable as is.  There are only
> two options:
>  * scrap it and start over
>  * live with it
> You probably assume these options, but I claim they do not exist:
>  * make an intermediate attempt to fix only things that are a problem
>  * make only a targeted small set of fixes

You know, I wonder about your pessimism.

Of course I disagree not at all about your description of how ANSI works.
But it isn't clear that Alex was talking about an ANSI revision.

Many times I've dreamed of a "Common Lisp 1.1", which would be defined more
by a SRFI-like process than a revisiting of the ANSI process.  (Ironically,
it was your own suggestions in the past that got me thinking of this.)

I love the phrase in ANSI CL that an implementation "purports to conform"
to the spec.  That captures the nuances of the situation beautifully.

It seems to me that -- completely separate from ANSI -- someone could define
a community-wide slightly-enhanced version of Common Lisp.  Perhaps starting
with the hyperspec, and then making "unobjectionable" changes and/or
clarifications.  And then they could publish this new thing as
        The Common Lisp 1.1 specification

And then, once such a thing exists in formal form, any given implemention
could choose to announce that they now "purport to conform" to CL 1.1, instead
of to the older ANSI CL.

It seems to me that such a thing would be possible, and almost any process
could be used to generate the CL 1.1 spec -- not just a re-opening of the
ANSI process.  Of course it matters what the process is, simply to get
community buy-in.  If one guy does it by himself without input from anyone
else, it seems unlikely that many CL vendors would choose to transfer their
alligience from the ANSI spec.  But a cleverly-designed process -- yet still
not ANSI -- might well be far cheaper and more contained that you've
described, while at the same time resulting in long-term commitment from the
majority of current (and future?) CL vendors.

Anyway, I don't interpret a comment as "this part of the spec could be more
clear" as an automatic naive call to reopen ANSI.

        -- Don
_______________________________________________________________________________
Don Geddis                  http://don.geddis.org/               ···@geddis.org
Vegetarian:     "I object to KILLING animals for FOOD!"
Captain Ribman: "And I object to eating LIVE animals!"
From: Kent M Pitman
Subject: Re: defvar affecting captured closure variables ?
Date: 
Message-ID: <uhcl2l2eu.fsf@nhplace.com>
Don Geddis <···@geddis.org> writes:

> Many times I've dreamed of a "Common Lisp 1.1", which would be defined more
> by a SRFI-like process than a revisiting of the ANSI process.  (Ironically,
> it was your own suggestions in the past that got me thinking of this.)

Extensions are the things I've advocated managing in a way like that,
but not adherence to the language per se.  I think the language itself
needs to stay the same or change radically, but not be nitpicked by
small changes, for reasons I've outlined in my prior message and here 
below...

My understanding is that if you try to say what implementation does
and doesn't conform to the ANSI standard, there are ways you can open
yourself to lawsuits (I think both under defamation statutes and under
antitrust law).  At least, the case was made to us during our
standards work that this was the case.  We were strongly advised that
we should not be making definitive statements that could be construed
as us in our committee role saying that an implementation did not
conform, since that would effectively put us in the position of
disparaging someone who WAS purporting to conform.

I'm not sure of the details, but I suspect the antitrust issue
probably relates to the fact that it's easy for a group that seems
superficially like a majority to say things should be a certain way,
only to find that what seemed like one outlier was really perceived by
the courts as a cartel of large players crowding out a small player.
ANSI processes may seem cumbersome, but they are specifically designed
to dodge this notion of crowding someone out, by allowing a great many
opportunities for review of even the smallest technical issue, with
special care given to those issues raised by small players who feel
they would be infringed in the market.  Some of those processes are 
public review, but some are more subtle like membership/voting/notification
requirements which help to ensure that all necessary parties have 
heard about the issues.  Any process you make to simulate this that is
less overhead has to be sure to have those same properties, I suspect.

This is part of why I have generally advocated pluralistic processes
that allow multiple points of view as the outcome, not just as the
input.  So having multiple "substandards" for stream protocols, for
example, is better to me than one single Right Way because then people
can voluntarily conform to one or more and there is no dictated one
way of doing things that can be pointed at to say that people are
forced to do it that way.  I don't know that this would dodge the
antitrust issues either, of course, but it has always seemed to me
that it should because it doesn't involve the nasty step that consensus
processes have of forcing agreement and publishing a result in the name
of parties some of whom may have even voted against the result.

The ANSI standard is what it is.  You can make ANOTHER thing which is
not ANSI, and you can get vendors to purport to conform to that, and then
you can modify that.  But you can't make them decide to do that.

And at that point, the vendors have to trust that the other thing is not
subject to new change in a way that will upset their business.  They know
ANSI will only change in fixed and pre-determined ways.  Since the purpose
of another process would not be that, getting vendors to switch over might
be harder than you suspect.  Or maybe not.

> It seems to me that -- completely separate from ANSI -- someone
> could define a community-wide slightly-enhanced version of Common
> Lisp.  Perhaps starting with the hyperspec,

The HyperSpec is copyrighted and not, to my knowledge, available for such use.

You probably mean starting with the TeX sources for dpANS CL, which are
largely assumed to be unencumbered (although there's no legal process for
certifying such matters under American law, and, as such, I'm not legally
competent to advise you that this is a 100% safe avenue, even though I
know of no problems that have ever resulted from people using these).

> and then making "unobjectionable" changes and/or clarifications.

It's no sure indicator, but CL history has suggested there is no such
concept as an unobjectionable change.  Steele came to the 1986
Monterey meeting of CLTL designers with a list of changes he expected
to be non-controversial and a large number were in fact seen to be
unexpectedly controversial.  This is how we got into ANSI--we had no
procedure for otherwise resolving the fact that the community was all
over the map.

> And then they could publish this new thing as The Common Lisp 1.1
> specification

I would personally advise you that it's FAR better to get funding to
make a new, coherent dialect that had these changes already made and
then sell the result, not the promise.  Then people would know what
they were buying in transitioning.  Otherwise, you're selling
destabilization for the promise of convergence.  I don't see the
upside UNLESS you were going to do more than the ultra-light editing
you're suggesting, and yet if you're going to do anything more you
introduce risk.

> And then, once such a thing exists in formal form, any given
> implemention could choose to announce that they now "purport to
> conform" to CL 1.1, instead of to the older ANSI CL.

At some risk of dividing the community and/or spawning many splintered
groups.

If you're not going to make something substantially new, I'm not sure
this is worth the effort.

I'm not even sure that a namable set of extensions that everyone must do
in order to get The Good Name is worth it--I think that defies pluralism.
I think
 (a) there is not substantial disagreement on what is needed to make
     a conforming CL now
 (b) it's easy enough to make voluntary extensions (even redundant ones
     that disagree by doing the same thing three different ways)
 (c) insisting things should be done only one way is inappropriate

But all of that is just my personal opinion.

> It seems to me that such a thing would be possible, and almost any process
> could be used to generate the CL 1.1 spec -- not just a re-opening of the
> ANSI process.

What would this accomplish?  You haven't described it in any way
whatsoever other than that it has a name that seems greaterp than the
original name.  What is going to assure that it's going in the good
direction? What's going to assure that there is a canonical good
direction?  What will happen to those who disagree?  Qualitatively, 
what will distinguish CL 1.1 from Arc? (In other words, something is
either compatible or it isn't--is there some middle ground I'm missing?)

Incidentally, while I have not rallied around Arc per se, my remarks
should not be taken to disparage the notion of what Graham has done
with that.  I think he's following the right process and I think it's
fine for people to experiment with dialects.  I just don't think they
should gamble the CL dialect as part of that experimentation--CL is
where people should park who want stability. I recall the days when
Lispers used to wake every Monday or so to email saying "your programs
no longer mean what they used to; please update them".  CL was an
answer to that instability.  It was not meant to say there couldn't be
dialects evolving, it was just meant to say that some people have real
businesses that depend on stability and need not to be perturbed while
change is in progress.

There are many Lisp dialects out there now and perhaps should be more.
But why should the one stable dialect be perturbed on the promise 
of something new--let's see the new thing, then decide if it's worthy
of the name.

> Of course it matters what the process is, simply to get
> community buy-in.

Is there just one community and how would you define/describe it such
that you could know if you had buy-in?

> If one guy does it by himself without input from anyone
> else, it seems unlikely that many CL vendors would choose to transfer their
> alligience from the ANSI spec.

Honestly, I think it depends on who that one person is and whether they're
he or she is well enough funded to do a good job.  But I suppose it depends
on what your goodness metric is.  Do you suppose The Community has a canonical
sense of what is good in that regard?

> But a cleverly-designed process -- yet still not ANSI -- might well
> be far cheaper and more contained that you've described, while at
> the same time resulting in long-term commitment from the majority of
> current (and future?) CL vendors.

If I were a vendor and I had money to burn on something, it wouldn't be
perturbing the ANSI base.  It would either be finding a new dialect that
was sufficiently better at addressing my market needs that an incompatible
break was in order, or it would be making compatible extensions to the 
investment I'd already made.

But I'd be interested to hear vendor points of view on this.

> Anyway, I don't interpret a comment as "this part of the spec could be more
> clear" as an automatic naive call to reopen ANSI.

Perhaps we see the word "could" differently.  I wouldn't have blinked at
"could have".  But "could" is very different--it suggests future possibility
and that begs an understanding of mechanism.  I think without mechanism
there's no meaning to this observation.

All just my personal opinion, AND I reserve the right to change my mind
dynamically as discussion progresses. :)
From: Don Geddis
Subject: Re: defvar affecting captured closure variables ?
Date: 
Message-ID: <87wstyupus.fsf@geddis.org>
Kent M Pitman <······@nhplace.com> wrote on 07 Oct 2007 23:1:
> Don Geddis <···@geddis.org> writes:
>> Many times I've dreamed of a "Common Lisp 1.1", which would be defined more
>> by a SRFI-like process than a revisiting of the ANSI process.  (Ironically,
>> it was your own suggestions in the past that got me thinking of this.)
>
> Extensions are the things I've advocated managing in a way like that,
> but not adherence to the language per se.

I guess I agree.  I was thinking more of extensions.  Plus perhaps a few
clarifications.  Not a radical new dialect.

I don't think I mentioned some attempt to assert whether an implementation
conformed to a label or not.

> Some of those processes are public review, but some are more subtle like
> membership/voting/notification requirements which help to ensure that all
> necessary parties have heard about the issues.  Any process you make to
> simulate this that is less overhead has to be sure to have those same
> properties, I suspect.

You could be right.  I think a Sufficiently Wise Philosopher King could
handle this with far less cost than ANSI, but perhaps I'm underestimating the
difficulty.

> This is part of why I have generally advocated pluralistic processes
> that allow multiple points of view as the outcome, not just as the
> input.  So having multiple "substandards" for stream protocols, for
> example, is better to me than one single Right Way because then people
> can voluntarily conform to one or more and there is no dictated one
> way of doing things that can be pointed at to say that people are
> forced to do it that way.

Ah, but that's of far less value to users.  That's basically the same as:
"all you can rely on for portable code is the ANSI spec; for anything else,
read the documentation of your implementation."  It seems to me that we could
at least imagine for striving for more that portable code could count on.

> it doesn't involve the nasty step that consensus processes have of forcing
> agreement and publishing a result in the name of parties some of whom may
> have even voted against the result.

I didn't imagine publishing a result in anyone's name.  I imagined a process
that would result in some document ("Common Lisp 1.1"), to which any vendor
could choose to ("purport to") conform, or not.

> The ANSI standard is what it is.  You can make ANOTHER thing which is
> not ANSI, and you can get vendors to purport to conform to that, and then
> you can modify that.  But you can't make them decide to do that.

Agreed.  But that was exactly my thought.  Build the spec first (hopefully
with the input of vendors), and then suggest that different implementations
consider beginning to conform to it.

Not: somehow agree ahead of time that they WILL conform to WHATEVER the
outcome of the process is.

>> It seems to me that -- completely separate from ANSI -- someone could
>> define a community-wide slightly-enhanced version of Common Lisp.  Perhaps
>> starting with the hyperspec,
>
> The HyperSpec is copyrighted and not, to my knowledge, available for such
> use.  You probably mean starting with the TeX sources for dpANS CL, which
> are largely assumed to be unencumbered [although this is not a legal opinion]

Ah.  I was confused, because the Hyperspec is so convenient, and I had
thought I had seen it available on the web in multiple locations.  So I just
assumed it was public domain.

But yes.  You're almost certainly right that the TeX sources are what would
need to be the starting point.

>> and then making "unobjectionable" changes and/or clarifications.
>
> It's no sure indicator, but CL history has suggested there is no such
> concept as an unobjectionable change.  Steele came to the 1986 Monterey
> meeting of CLTL designers with a list of changes he expected to be
> non-controversial and a large number were in fact seen to be unexpectedly
> controversial.  This is how we got into ANSI--we had no procedure for
> otherwise resolving the fact that the community was all over the map.

Yes, I've heard you tell this story before, and I fully accept the
problematic nature of my wording.  (I put the scare quotes there, because
even I recalled your story as I was writing the phrase.)

Here's really the sequence I'm thinking of:

1. ANSI Common Lisp (aka CL 1.0)        The spec as we know it
2. CL 1.1                               Typos, errors from ANSI CL
3. CL 1.2                               Additional "minor" extensions

For CL 1.1, it would be driven NOT by the wishes of the participants, but
instead by observation of the actual behavior of the vendor community in the
last decade or so.  For example, there are a (very few!) sentences in the
official ANSI CL spec that are clearly just in error.  Like a similar
paragraph copied from a different location, that wasn't properly modified in
the new location.  Everybody knows this stuff, and all vendors just "do the
right (obvious) thing".  Even though it officially deviates from the spec.
I'm thinking of things like Franz's list of non-compliance:
        http://franz.com/support/documentation/current/doc/implementation.htm#compliance-1

While some of these things are just errors, others result from the spec being
ambiguous or even inconsistent on some rare corner case.  It would seem to be
of at least minor value just to spell these things out.  Something like:

        The following 10 different CL implementations all looked at section
        3.1.x.y.z in the spec, and all concluded that the real intention is
        B, despite the words there apparently requiring A.  If you're
        building a new CL implemention, this note is merely to let you know
        that everyone to date has found section 3.1.x.y.z problematic, and
        everyone else has concluded that B is the reasonable implementation.

One could also imagine, in the CL 1.1 spec, rewriting something like the
pathname section, which even you admit was on your list for clarification.
NOT to change semantics, but just to make more explicit how all vendors
ALREADY interpret the words that are in existing ANSI spec.  Figuring out all
the consequences may be non trivial, and the community has done that work.
So why not write it down in one obvious place.

As for the more ambitious CL 1.2, perhaps that's a different project.  But
there I was thinking about optional libraries, which might be nice to package
into a single large bag, and make it easy for a vendor to say "I (purport to
conform to CL 1.2) and/or agree to supply these libraries by default."

My own short list might include SPLIT-SEQUENCE, some regular expression
library, probably ITERATE, maybe ASDF, perhaps even Franz's IF* (although I
doubt I'd use that).  One could even propose a requirement that any
functionality added to CL 1.2 would need to include a reference
implementation for the functionality, written in ANSI CL (1.0 or 1.1).

What's the point?  I guess for user convenience, that a user could publish
the code of some application, and say: "this runs on any CL 1.2
implementation".  Without needing to include the source for all the obvious
libraries as well.  Even if you only have a CL 1.0/1.1 implementation
yourself, the process of upgrading it to CL 1.2 would be obvious: all the
functionality is defined in spec form (rather than only as code), and
reference implementations are always available in a well-known place.

Assuming that SOMEHOW both of these efforts (CL 1.1 and CL 1.2) went well, I
can imagine that the success of the process might lead some bold people to
attempt a more ambitious effort for CL 1.3, trying to include things like FFI
or multiprocessing, that have multiple tradeoffs and for which the different
existing implementations have currently made distinct choices.

My intention was never to suggest that the process would somehow force
everyone to submit to some standard in these topics.

But I still think there might be value in formalizing existing community
consensus, and perhaps in slightly expanding the set of built-in libraries.

>> And then they could publish this new thing as The Common Lisp 1.1
>> specification
>
> I would personally advise you that it's FAR better to get funding to
> make a new, coherent dialect that had these changes already made and
> then sell the result, not the promise.

Well, yes, I agree.  That was kind of my proposal.  Why not just start
writing a new "slightly enhanced" spec?  Get comments and feedback from the
community (including vendors), and try to formalize existing practice.
Perhaps even bite off one or two things that are merely gratuitously
incompatible -- but offering the identical functionality -- if any such
examples exist.

Of course this depends on someone with the time and effort and expertise to
do it.  So, a matter of funding, I guess.  But you seemed to have a larger
objection than that.  You seemed to be suggesting that EVEN IF such a person
and/or money could be found, that it would likely be a bad idea.

>> And then, once such a thing exists in formal form, any given implemention
>> could choose to announce that they now "purport to conform" to CL 1.1,
>> instead of to the older ANSI CL.
>
> At some risk of dividing the community and/or spawning many splintered
> groups.  If you're not going to make something substantially new, I'm not
> sure this is worth the effort.

You could be right.  I'd suggest that the differences from CL 1.0 to CL 1.1
are generally in corner cases that most users never notice.  And the
differences between CL 1.1 and CL 1.2 could be fixed by an end user, even
without the cooperation of the vendor.

So I think the risk of splintering is small.  But perhaps the value is small
too, so not worth the cost.

> I'm not even sure that a namable set of extensions that everyone must do
> in order to get The Good Name is worth it--I think that defies pluralism.
> I think
>  (a) there is not substantial disagreement on what is needed to make
>      a conforming CL now
>  (b) it's easy enough to make voluntary extensions (even redundant ones
>      that disagree by doing the same thing three different ways)
>  (c) insisting things should be done only one way is inappropriate

I guess I slightly disagree with (c).  I think Common Lisp is mildly
vulnerable to the criticism from outside, that "CL doesn't have X" (like
regular expressions).  And the response that "an expert could find and
learn about and install and use a useful library to do X -- but outside the
process of their vendor and the spec" is an unnecessarily weak response.

>> It seems to me that such a thing would be possible, and almost any process
>> could be used to generate the CL 1.1 spec -- not just a re-opening of the
>> ANSI process.
>
> What would this accomplish?  You haven't described it in any way
> whatsoever other than that it has a name that seems greaterp than the
> original name.

True.  :-)  So I tried to give more detail this time (above).

> What is going to assure that it's going in the good direction?

That it begin with a charter much like ANSI's charter: codify existing
practice; no new design without existing multi-vendor implementation;
no gratuitous backward incompatible changes; etc.

> What's going to assure that there is a canonical good direction?

That's a legitimate danger.

> What will happen to those who disagree?

They merely continue to offer ANSI CL (1.0), instead of CL 1.2.
But users can probably turn that into 1.2 on their own...

> Qualitatively, what will distinguish CL 1.1 from Arc? (In other words,
> something is either compatible or it isn't--is there some middle ground I'm
> missing?)

It's compatible.  Merely "enhanced".  Much more extensions, then a new
language.

But at the very least, by focusing on relatively uncontroversial subjects,
it may create a process for accommodating future evolution in Common Lisp.

Perhaps you see that as a bad thing.

> I just don't think they should gamble the CL dialect as part of that
> experimentation--CL is where people should park who want stability.

To keep the Common Lisp name, I think any successor language in the style I'm
suggesting would have to continue to run ANSI CL conforming code correctly.

I guess there's a reason I used the name "CL 1.1" and "CL 1.2", instead of
"CL 2.0".  This is not an attempt to reinvent the language.  Merely (?) an
attempt to update it.

>> Of course it matters what the process is, simply to get community buy-in.
>
> Is there just one community and how would you define/describe it such
> that you could know if you had buy-in?

I guess I was referring to the implementor community.  Surely there's just a
handful (a dozen?) software systems, where the authors "purport to conform"
to ANSI CL.  They're the main ones to convince that the effort is worthwhile.
I suppose an implementor might be dragged there by a flood of their own users
demanding the new thing.  And perhaps a few would be "late adopters" in that
sense.  But I can't believe you'd get large user demand without most of the
vendors giving it a try first.

> If I were a vendor and I had money to burn on something, it wouldn't be
> perturbing the ANSI base.

Well, I wasn't necessarily proposing that the vendors fund this process.

> It would either be finding a new dialect that was sufficiently better at
> addressing my market needs that an incompatible break was in order, or it
> would be making compatible extensions to the investment I'd already made.

Yes, compatible extensions was my idea.

But vendors generally are motivated mostly by their own customer community.
Cross-vendor standards are of more value to users (and sometimes of negative
value to vendors).

It's users who want to write portable code, and are best served by have the
most capable language beneath them that they can count on.

        -- Don
_______________________________________________________________________________
Don Geddis                  http://don.geddis.org/               ···@geddis.org
[Warning on knife]:  Caution.  Blade is sharp.  Keep out of children.
From: Rob Warnock
Subject: Re: defvar affecting captured closure variables ?
Date: 
Message-ID: <yqednZFi857ycZTanZ2dnUVZ_oSnnZ2d@speakeasy.net>
Kent M Pitman <······@nhplace.com> wrote:
+---------------
| CL is where people should park who want stability.
+---------------

Hallelujah, brother! Amen!

+---------------
| I recall the days when Lispers used to wake every Monday or so
| to email saying "your programs no longer mean what they used to;
| please update them".  CL was an answer to that instability.
+---------------

Replace "Lispers" with "Schemers" and you have a large part of why
I switched from Scheme to Common Lisp. [And never looked back...]


-Rob

-----
Rob Warnock			<····@rpw3.org>
627 26th Avenue			<URL:http://rpw3.org/>
San Mateo, CA 94403		(650)572-2607
From: Rainer Joswig
Subject: Re: defvar affecting captured closure variables ?
Date: 
Message-ID: <joswig-C27B73.11483008102007@news-europe.giganews.com>
In article <································@speakeasy.net>,
 ····@rpw3.org (Rob Warnock) wrote:

> Kent M Pitman <······@nhplace.com> wrote:
> +---------------
> | CL is where people should park who want stability.
> +---------------
> 
> Hallelujah, brother! Amen!
> 
> +---------------
> | I recall the days when Lispers used to wake every Monday or so
> | to email saying "your programs no longer mean what they used to;
> | please update them".  CL was an answer to that instability.
> +---------------
> 
> Replace "Lispers" with "Schemers" and you have a large part of why
> I switched from Scheme to Common Lisp. [And never looked back...]

It is still kind of, hmm, how should I describe it,
strangely fascinating, hmm, to watch the fate of the Scheme
language. I saw the word 'self-mutilating' when
I read about R6RS.

> 
> 
> -Rob
> 
> -----
> Rob Warnock			<····@rpw3.org>
> 627 26th Avenue			<URL:http://rpw3.org/>
> San Mateo, CA 94403		(650)572-2607

-- 
http://lispm.dyndns.org
From: George Neuner
Subject: Re: defvar affecting captured closure variables ?
Date: 
Message-ID: <u2mkg3hpsc7727faqofsb5f688olafn6c6@4ax.com>
On 07 Oct 2007 23:18:33 -0400, Kent M Pitman <······@nhplace.com>
wrote:

>Don Geddis <···@geddis.org> writes:
>
>> Many times I've dreamed of a "Common Lisp 1.1", which would be defined more
>> by a SRFI-like process than a revisiting of the ANSI process.  (Ironically,
>> it was your own suggestions in the past that got me thinking of this.)
>
>Extensions are the things I've advocated managing in a way like that,
>but not adherence to the language per se.  I think the language itself
>needs to stay the same or change radically, but not be nitpicked by
>small changes, for reasons I've outlined in my prior message and here 
>below...
>
>My understanding is that if you try to say what implementation does
>and doesn't conform to the ANSI standard, there are ways you can open
>yourself to lawsuits (I think both under defamation statutes and under
>antitrust law).  At least, the case was made to us during our
>standards work that this was the case.  We were strongly advised that
>we should not be making definitive statements that could be construed
>as us in our committee role saying that an implementation did not
>conform, since that would effectively put us in the position of
>disparaging someone who WAS purporting to conform.
>
>I'm not sure of the details, but I suspect the antitrust issue
>probably relates to the fact that it's easy for a group that seems
>superficially like a majority to say things should be a certain way,
>only to find that what seemed like one outlier was really perceived by
>the courts as a cartel of large players crowding out a small player.
>ANSI processes may seem cumbersome, but they are specifically designed
>to dodge this notion of crowding someone out, by allowing a great many
>opportunities for review of even the smallest technical issue, with
>special care given to those issues raised by small players who feel
>they would be infringed in the market.  Some of those processes are 
>public review, but some are more subtle like membership/voting/notification
>requirements which help to ensure that all necessary parties have 
>heard about the issues.  Any process you make to simulate this that is
>less overhead has to be sure to have those same properties, I suspect.

Kent, 

I'm not a lawyer, but I do have a several lawyers (IP, RE and general)
in my family and during college I worked as a paralegal researching
and reviewing patent applications for my father's firm.  As a software
developer of twenty-plus years, I have had exposure to IP, business
and FDA law.
[I'm certain you also have had a good amount of legal exposure.  My
only advantage over the next guy is that I read legalese, I have law
books to consult at will and I can get a free lecture when I don't
understand something.]
When in doubt consult your own lawyer.


For a member of a standards body committee, there are strong antitrust
considerations because an evolving standard defines a potential
marketplace.  It is easy to accidentally do something that could place
undue hardship on an existing vendor.  It is harder to accidentally
prevent new vendors from entering the market, but that also might be
possible under certain circumstances.  And, as you noted, the
individual opinion of a committee member might be misconstrued by
outsiders as representative of the whole committee.

However, the duty you had as a committee member has expired - you
can't act on behalf of a committee that no longer exists.  And
antitrust does not apply to individuals.  Only representing a group
could you potentially violate the law.  So long as you make clear when
you are speaking as an individual and when not, and when speaking for
a group you are careful about making anti-competitive statements, I
believe the danger of antitrust violation is minimal.  You seem very
aware of these issues already.

The defamation issue is trickier.  Defamation is knowingly making
false statements about another person or group which results in damage
to that person's or group's reputation.  Defamation is usually
collectively referred to as "libel" although it is sometimes broken
down further into libel (defaming in print) and "slander" (defaming
verbally).  Libel is a category of civil tort law.

Mistaken statements are not libel[*], but even unintentional false
statements may have caused tort injuries which may require
restitution.  True statements are not libel.  Defendants usually are
not held responsible for tort injuries caused by true statements (but
there are exceptions).  Truth is a defense against a libel accusation,
but it won't necessarily prevent a lawsuit - it only serves to protect
the defendant from a malicious damage award.

[*] Libel is hard to prove because the defendant can always claim to
have made a mistake.  The plaintiff must prove that the defendant knew
the statements were false before making them.


There is an additional consideration caused by your status as an
"expert" in certain subjects.  Expert opinions are granted certain
protection from libel accusations in contexts such as private
communications with clients or with other experts, press interviews,
testimony in court, etc.  However, in general, an expert opinion is
presumed to carry significant weight with a lay audience, so when
addressing the subject of her expertise, the expert must be careful
not to make potentially libelous statements in communication with
non-experts who may accept the statements as factual without
qualification.

[I could say something about a certain troll at this point, but it
might consider my statements to be libelous.  I'm not sure about the
legal status of trolls, but illegal aliens seem to have more rights
than citizens, so I'm betting trolls do too.]


>> It seems to me that -- completely separate from ANSI -- someone
>> could define a community-wide slightly-enhanced version of Common
>> Lisp.  Perhaps starting with the hyperspec,
>
>The HyperSpec is copyrighted and not, to my knowledge, available for such use.
>
>You probably mean starting with the TeX sources for dpANS CL, which are
>largely assumed to be unencumbered (although there's no legal process for
>certifying such matters under American law, and, as such, I'm not legally
>competent to advise you that this is a 100% safe avenue, even though I
>know of no problems that have ever resulted from people using these).

Don, 

Reiterating that I'm not a lawyer ...

The safest thing to do is to create a new document with details of
your extensions/modifications, and cite the reference document where
required.  If you have, or can obtain, permission to redistribute the
reference document, that makes it easier to correlate it with yours
... but *don't* make any changes to the reference document itself.  

I believe the Hyperspec's redistribution conditions permit such a use.
http://www.lispworks.com/documentation/HyperSpec/Front/Help.htm#Legal


>> And then they could publish this new thing as The Common Lisp 1.1
>> specification

>> It seems to me that such a thing would be possible, and almost any process
>> could be used to generate the CL 1.1 spec -- not just a re-opening of the
>> ANSI process.

I'm not current with copyright law as it pertains to standards bodies.
Under normal circumstances, you would require explicit permission from
the author(s) of the reference document(s) if you intend to distribute
a derivative work that integrates reference text.  But because the
document is a standard, fair use might apply ... you'd have to ask an
IP lawyer to be sure.

It may also be that, as a standards governing body, ANSI has a duty to
grant permission for non-sanctioned derivatives - you'd have to check
the organization charter.

It is definitely more convenient to present an integrated document to
an audience, but it isn't necessary and it frequently is not done.
Many industry standard documents simply reference others without
integrating the reference text.  So long as you keep your extensions
separate from the reference document(s), you can do with them as you
please.

George
--
for email reply remove "/" from address
From: Alex Mizrahi
Subject: Re: defvar affecting captured closure variables ?
Date: 
Message-ID: <470cf18f$0$90267$14726298@news.sunsite.dk>
(message (Hello 'Kent)
(you :wrote  :on '(07 Oct 2007 13:03:58 -0400))
(

 KMP> Your questions seem to me to be taking the form "I can show by my own
 KMP> hand that everyone else is wrong and I am right and the world must
 KMP> suddenly and incompatibly change to suit me".  You may not mean this,

certainly i don't mean that :)
i've just wanted to clarify an area that looked quite controversial and 
vague for me.
i now see that my understanding of how standard works was quite different 
from how standard works, and omission of a specification for questioned 
behaviour is an intentional desire to allow experiments in this area for 
implementors, as you've said.
thank you for detailed comments on these issues, that were puzzling me for 
quite a lot time..

 KMP> Your program is in error so the standard does not apply here.
 KMP>  You have not created a constant.
 KMP>  You have not created a dynamic variable.
 KMP>  You have not created a lexical variable.

i thought that in this case following rules should apply:

unbound variable n. a name that is syntactically plausible as the name of a 
variable but which is not bound in the variable namespace.
An error of type unbound-variable should be signaled if an unbound variable 
is referenced.

but it looks like if i didn't explicitly define the variable, implementation 
is free to apply it's own rules..

 ??>> and it's very easy to make it more clear: either explicitly allow
 ??>> implementations to extend definition of dynamic, allow other reaction
 ??>> on unbound variable, or allow other states of variables..

 KMP> It's not very easy to make it more clear.  To make it more clear could
 KMP> and probably would cost:

uh.. but that doesn't forbid us to do a thought experiment :)

)
(With-best-regards '(Alex Mizrahi) :aka 'killer_storm)
"Hanging In The Balance Of Deceit And Blasphemy") 
From: Kent M Pitman
Subject: Re: defvar affecting captured closure variables ?
Date: 
Message-ID: <uy7ea29vk.fsf@nhplace.com>
"Alex Mizrahi" <········@users.sourceforge.net> writes:

> i now see that my understanding of how standard works was quite
> different from how standard works, and omission of a specification
> for questioned behaviour is an intentional desire to allow
> experiments in this area for implementors, as you've said.

Yes, if you were around in the time of CLTL, you'd have seen the issue
was MUCH worse then, since really CLTL was more of an effort to allow
vendors to change really not at all and yet to say they were all
conforming.  This was a HUGE irritation to users, and it's a lot better
now, but I'm not trying to deny that it's still sometimes an irritation.

But, on the other hand, users used to spend a lot of time waiting for
the vendors to change things before they could go on with their lives.
Vendors were the bringers of good, kind of like Vaal in the Star Trek
episode "The Apple" [*], and users spent a lot of time feeling not
enabled.  I often regard discussions of this kind as drifting back
toward that time, and I think my remarks about how you shouldn't
expect more can be easily misinterpreted as me attempting to hold
things back and say there should be no progress from those times.
It's not that at all.  Rather, I'm trying to say that the modern world
proceeds by not waiting on Vaal, but by feeding yourself.  And in the
case that that's not enough, IMO, the right solution is to make new
things, not fuss over changing what's there, because it's minimally
invasive to others in the disconnected/unsynchronized world that is
today.  Making everyone stop and agree is expensive.  It's like trying
to get people to agree on coke or pepsi... better to just have both,
have them compete, have a divided market, but have everyone happy...
except those who insist that there should be only one soda for reasons
of proper standards.

[*] http://en.wikipedia.org/wiki/The_Apple_(TOS_episode)

> i thought that in this case following rules should apply:
>  KMP> It's not very easy to make it more clear.  To make it more clear could
>  KMP> and probably would cost:
> 
> uh.. but that doesn't forbid us to do a thought experiment :)

Absolutely not.  My remarks should be taken as trying to frame
expectations from pre-existing processes a little, but certainly not
as any attempt to hold back thought, even critical thought.  I think
one just wants to put such critiques into a framework that is going to
be maximally constructive and minimally disruptive.  And, I suppose,
people differ on even those terms... so you should take all of my
remarks as just one person's opinion, certainly no source of authority
as to how the world must be.
From: Thomas Bakketun
Subject: Re: defvar affecting captured closure variables ?
Date: 
Message-ID: <5mqv4sFes907U1@mid.individual.net>
Alex Mizrahi wrote:

> seems pretty clear, but.. after you do perfectly legal (setf (symbol-value
> 'foo) 5), all implementations i know define FOO as some magical variable
> -- it's not lexical or special, nevertheless it's definitely a variable,
> since it's bound,

It's a dynamic variable, but it's neither special nor magical. According to
the CLHS[1] setting the symbol-value of a symbol will never signal an
error. So this should behave the same in any Common Lisp:

> (setf (symbol-value 'foo) 42)
42
> (boundp 'foo)
T
> (symbol-value 'foo)
42

However, using the symbol as form is not well defined, but it still works:

> (+ foo 32)
74

The relevant part of CLHS here is 3.1.2.1.1 Symbols as Forms[2]. The symbol
is clearly not a symbol-macro, lexical variable or a constant. But is not a
dynamic variable either, since it's not declared special. Anyway it is
still threated as dynamic variable in most (or all?) implementations.

> and indeed there are no complains from implementations 
> when such variables are used.

All the implementations I tested (CMUCL, SBCL, CLISP and ECL) will complain
if such code is compiled. Here is an example in ECL:

> (defun bar () foo)
BAR
> (bar)
42
> (compile 'bar)
;;; End of Pass 1.  
;;; Compiling (LET* (# #) ...).
;;; The variable FOO is undefined.
;;; The compiler will assume this variable is a global.
...

Note that it does not matter if foo is bound or not when the function is
compiled. What matters is if it's declared special:

> (defun bar () (declare (special foo)) foo)
BAR
> (compile 'bar)
;;; End of Pass 1.  

> so, all implementations implement some "magical" kind of variables that
> are not described by standard, 

No, the variables are just completly normal dynamic variables. The "magic"
does not happen during (setf symbol-value), but when a symbol not declared
as special or a lexical variable is used as variable.

> i believe that this means that all Common Lisp implementations are not
> conformant to the Common Lisp standard.

I doubt it's a direct violation of the standard. At least I could not find a
requirement to signal an error if an undefined variable is used. Chapter
3.1.2.1.1 does not mention the possibilty of undefined variables at all.

> so, i wonder is there some rationale for introducing non-trivial
> symbol-macro rules instead of legalizing behaviour that is already present
> in all (most) implementations, and resolving many controversies at same
> time?

Because that behaviour is not what you really want. Silently threating 
undefined variables as dynamic variables, will easly lead to hard to find
bugs. And as mentions above, compilers will warn you about this.


[1] http://www.lispworks.com/documentation/HyperSpec/Body/f_symb_5.htm
[2] http://www.lispworks.com/documentation/HyperSpec/Body/03_abaa.htm
From: Rob Warnock
Subject: Re: defvar affecting captured closure variables ?
Date: 
Message-ID: <vq6dnQgg7OkawZXanZ2dnUVZ_tqtnZ2d@speakeasy.net>
Kent M Pitman  <······@nhplace.com> wrote:
+---------------
| Incidentally, ignore the advice about global lexicals.  For better or
| worse, I'm pretty sure all you want is this:
| 
|  (define-symbol-macro foo (symbol-value 'foo))
| 
| Don't forget to use explicit special bindings when you really mean to 
| bind these globals, since otherwise you'll get a lexical foo when you
| bind foo.
+---------------

For this reason, and also to avoid unbound-variable mistakes, e.g.:

    > (define-symbol-macro foo (symbol-value 'foo))

    FOO
    > foo

    Error in KERNEL::UNBOUND-SYMBOL-ERROR-HANDLER:  the variable FOO is unbound.
       [Condition of type UNBOUND-VARIABLE]
       ...

I would suggest providing/using some binding macro for such shadowable
specials to make it clear what's going on, something like this [feel
free to suggest a shorter yet still perspicuous name]:

    > (defmacro defvar/shadowable (var &optional value (doc nil docp))
        `(progn
           (setf (symbol-value ',var) ,value)
           ,@(when docp
               (list `(setf (documentation ',var 'variable) ,doc)))
           (define-symbol-macro ,var (symbol-value ',var))))
            
    DEFVAR/SHADOWABLE
    > (defvar/shadowable foo 13 "An example shadowable special")
    
    FOO
    > (defun foo ()
	"A function which gives the global value of FOO."
	foo)
    
    FOO
    > (let ((foo 27))
        (list foo (incf foo) foo
	      (foo) (locally (declare (special foo)) (incf foo)) (foo)))
               
    (27 28 28 13 14 14)
    > foo
    
    14
    > 

The "DEFVAR/..." part of the name will suggest that this variable
*is* still intended to be used as a special at least in some contexts
while the ".../SHADOWABLE" [or some shorter, less-awkward phrase?]
will reassure one that its binding is lexically shadowable.


-Rob

p.s. I happen to think that having documentation strings
"just work" is worth a little extra effort in one's macros:

    > (describe 'foo)

    FOO is an internal symbol in the COMMON-LISP-USER package.
    It is a symbol macro with expansion: (SYMBOL-VALUE 'FOO).
    Macro documentation:
      An example shadowable special
    Function: #<Interpreted Function FOO {48A64D41}>
    Function arguments:
      There are no arguments.
    Function documentation:
      A function which gives the global value of FOO.
    Its defined argument types are:
      NIL
    Its result type is:
      *
    Its definition is:
      (LAMBDA () (BLOCK FOO FOO))
    > 

-----
Rob Warnock			<····@rpw3.org>
627 26th Avenue			<URL:http://rpw3.org/>
San Mateo, CA 94403		(650)572-2607
From: Kent M Pitman
Subject: Re: defvar affecting captured closure variables ?
Date: 
Message-ID: <u1wc63osc.fsf@nhplace.com>
····@rpw3.org (Rob Warnock) writes:

> Kent M Pitman  <······@nhplace.com> wrote:
> +---------------
> | Incidentally, ignore the advice about global lexicals.  For better or
> | worse, I'm pretty sure all you want is this:
> | 
> |  (define-symbol-macro foo (symbol-value 'foo))
> | 
> | Don't forget to use explicit special bindings when you really mean to 
> | bind these globals, since otherwise you'll get a lexical foo when you
> | bind foo.
> +---------------
> 
> For this reason, and also to avoid unbound-variable mistakes, e.g.:
> 
>     > (define-symbol-macro foo (symbol-value 'foo))
> 
>     FOO
>     > foo
> 
>     Error in KERNEL::UNBOUND-SYMBOL-ERROR-HANDLER:  the variable FOO is unbound.
>        [Condition of type UNBOUND-VARIABLE]
>        ...

Actually, I didn't include this level of detail because there were too many
options for how to treat the value.  I thought it was self-apparent that by
doing this, you were not assigning FOO and would have to do

 (SETQ FOO 3)

afterward if you wanted to globally assign it to 3.

Everyone should, of coruse, make it a point to know that
DEFINE-SYMBOL-MACRO sets up the variable so that (SETQ var value)
turns into (SETF expansion value), so (SETQ FOO 3) will become (SETF
(SYMBOL-VALUE 'FOO) 3) in this example.

I am not against having other abstractions for declaring/assigning,
it's just that there DEFVAR vs DEFPARAMETER behavioral split (which is
subtle and distracting while trying to teach about symbol macros) and
documentation issue (which is useful but irrelevant to this topic and
again distracting as ar esult).  It just all together makes it hard to
see the simplicity of what's really being done, so I ripped all of
that out of my example before posting and went for the simplest thing
that would illustrate the basic behavior.