Hi,
I'd like to restart a thread from June 2000 (with a similar question
being asked in 2002), found by searching Google groups for
special-operator-p implementation macro group:comp.lang.lisp
The problem I'm currently facing is that the Iterate package, which
relies on a code walker, may stumble on some forms which are or expand to
implementation-defined special operators.
E.g. IGNORE-ERRORS in CLISP expands to HANDLER-BIND which expands to
some internal special form.
CLHS says:
"special operator n. one of a fixed set of symbols, enumerated in
Figure 3-2, that may appear in the car of a form in order to identify
the form as a special form."
Back in 2000, Barry Margolin said:
"[the standard] doesn't say that the implementor can't build in more
[special operators]" ...
"The SPECIAL-OPERATOR-P function is provided so that code walkers can
notice when they've stumbled onto implementation-dependent special forms;
if they don't know how to handle that special form, they can punt and
report the failure to the user."
To which Steven M. Haflich replied
"But that would mean that no code could depend upon the success of a code
walker. That seems a poor design, similar to C++. The reason Lisp has
code walkers is so tools that were not designed to interoperate can still
do so. The Java VM tries to accomplish this with load/run time mechanisms.
Lisp accomplishes this at compile/load/run time. The purposes and tradeoffs
are different."
The thread died there.
So what's left to code walkers? Arbitrary inability to walk some
portable code (e.g. HANDLER-BIND and thus IGNORE-ERRORS in CLISP)?
I thought being able to walk code was one of CL's great pluses??
Currently, I'd resume my experience as follows: "the Iterate package
depends on a code walker. Since code walking may stumble on arbitrary
implementation-defined special operators as the result of expanding
portable code, it cannot be expected to work portably. Therefore it
can be viewed as poor design. Similarly, any purportedly portable
package which depends on a code-walker can be viewed as poor design,
if not otherwise restricted to particular usages or situations. YMMV"
What do you think?
http://groups-beta.google.com/group/comp.lang.lisp/browse_thread/thread/71fdc1dba28f7ca4/e2c1c77d6dc338ea?q=special-operator-p+implementation+macro+group:comp.lang.lisp&_done=%2Fgroups%3Fas_q%3Dspecial-operator-p+implementation+macro%26num%3D30%26as_scoring%3Dr%26hl%3Den%26ie%3DISO-8859-1%26btnG%3DGoogle+Search%26as_epq%3D%26as_oq%3D%26as_eq%3D%26as_ugroup%3Dcomp.lang.lisp%26as_usubject%3D%26as_uauthors%3D%26as_umsgid%3D%26lr%3D%26as_drrb%3Dq%26as_qdr%3D%26as_mind%3D12%26as_minm%3D5%26as_miny%3D1981%26as_maxd%3D13%26as_maxm%3D1%26as_maxy%3D2005%26safe%3Dimages%26&_doneTitle=Back+to+Search&&d#e2c1c77d6dc338ea
Thanks for your help,
Jorg Hohle
Telekom/T-Systems Technology Center
From: Matthew Danish
Subject: Re: what are allowed special forms? (CLHS clarification)
Date:
Message-ID: <87wtuhnnyj.fsf@mapcar.org>
SPECIAL-OPERATOR-P symbol
Returns true if symbol is a /special operator/; otherwise, returns false.
Glossary
/special operator/: n. one of a fixed set of symbols, enumerated in
Figure 3.1.2.1.2.1 Special Forms, that may appear in the car of a
form in order to identify the form as a special form.
3.1.2.1.2.1 Special Forms
The set of special operator names is fixed in Common Lisp; no way is
provided for the user to define a special operator. The next figure
lists all of the Common Lisp symbols that have definitions as
special operators.
block let* return-from
catch load-time-value setq
eval-when locally symbol-macrolet
flet macrolet tagbody
function multiple-value-call the
go multiple-value-prog1 throw
if progn unwind-protect
labels progv
let quote
But!
MACRO-FUNCTION
It is possible for both macro-function and special-operator-p to
return true of symbol. The macro definition must be available for
use by programs that understand only the standard Common Lisp
special forms.
Which seems to imply that it is possible to have non-CL-standard
special operators. This conflicts with the earlier definition of
special operators.
1.5.1.4.1 Resolution of Apparent Conflicts in Exceptional Situations
If more than one passage in this specification appears to apply to
the same situation but in conflicting ways, the passage that appears
to describe the situation in the most specific way (not necessarily
the passage that provides the most constrained kind of error
detection) takes precedence.
I'd say that the definition of special operator is the most specific.
But I think the issue is fairly unimportant, since the CL standard has
code-walkers in mind already:
3.1.2.1.2.2 Macro Forms
An implementation is free to implement a Common Lisp special
operator as a macro. An implementation is free to implement any
macro operator as a special operator, but only if an equivalent
definition of the macro is also provided.
(lisp-implementation-type) => "CLISP"
(special-operator-p 'system::%handler-bind) => NIL
(macro-function 'system::%handler-bind) => NIL
(symbol-function 'system::%handler-bind) => NIL
Which is it? I think most code-walkers will understand HANDLER-BIND
without expanding it, but it shouldn't be necessary.
--
;; Matthew Danish -- user: mrd domain: cmu.edu
;; OpenPGP public key: C24B6010 on keyring.debian.org
From: Duane Rettig
Subject: Re: what are allowed special forms? (CLHS clarification)
Date:
Message-ID: <4brbtxg80.fsf@franz.com>
[Matthew, you are theorizing a conflict in the spec, but there is
no conflict:]
Matthew Danish <··········@cmu.edu> writes:
[...]
> 3.1.2.1.2.1 Special Forms
>
> The set of special operator names is fixed in Common Lisp; no way is
> provided for the user to define a special operator. The next figure
=====================^^^^
> lists all of the Common Lisp symbols that have definitions as
> special operators.
[ ... ]
> 3.1.2.1.2.2 Macro Forms
>
> An implementation is free to implement a Common Lisp special
=======^^^^^^^^^^^^^^
> operator as a macro. An implementation is free to implement any
> macro operator as a special operator, but only if an equivalent
> definition of the macro is also provided.
So the spec is clear; users cannot add special operators, but the
implemetor can.
> (lisp-implementation-type) => "CLISP"
> (special-operator-p 'system::%handler-bind) => NIL
> (macro-function 'system::%handler-bind) => NIL
> (symbol-function 'system::%handler-bind) => NIL
>
> Which is it? I think most code-walkers will understand HANDLER-BIND
> without expanding it, but it shouldn't be necessary.
If it has no macro-function and it is not a special-operator,
then it is a function, pure and simple. Note that you are looking
at internals, not at 'cl:handler-bind.
--
Duane Rettig ·····@franz.com Franz Inc. http://www.franz.com/
555 12th St., Suite 1450 http://www.555citycenter.com/
Oakland, Ca. 94607 Phone: (510) 452-2000; Fax: (510) 452-0182
From: Matthew Danish
Subject: Re: what are allowed special forms? (CLHS clarification)
Date:
Message-ID: <87k6qhnhsb.fsf@mapcar.org>
Duane Rettig <·····@franz.com> writes:
> [Matthew, you are theorizing a conflict in the spec, but there is
> no conflict:]
>
> Matthew Danish <··········@cmu.edu> writes:
>
> [...]
>
> > 3.1.2.1.2.1 Special Forms
> >
> > The set of special operator names is fixed in Common Lisp; no way is
> > provided for the user to define a special operator. The next figure
> =====================^^^^
> > lists all of the Common Lisp symbols that have definitions as
> > special operators.
>
> [ ... ]
The definition of special operator contains strong language that is
incredibly specific. It states that there exists an enumeration of
all valid special operator names, and references the section that it
may be found.
I don't see any other cause to explicitly state "a fixed set" unless
it was meant that section 3.1.2.1.2.1 exhaustively enumerates all
special operator names, in Common Lisp.
> So the spec is clear; users cannot add special operators, but the
> implemetor can.
Of course the implementor can implement operators "like" special
operators: but they must provide equivalent macroexpansions because
code-walkers should not have to know about implementation-specific
optimizations. However, this either has nothing to do with the
definition of SPECIAL-OPERATOR-P, or is in direct conflict with its
CLHS entry.
I cannot find any possible other way to read the definition of
SPECIAL-OPERATOR-P other than:
(defun special-operator-p (symbol)
(check-type symbol symbol) ; in safe code
(member symbol
'(block let* return-from
catch load-time-value setq
eval-when locally symbol-macrolet
flet macrolet tagbody
function multiple-value-call the
go multiple-value-prog1 throw
if progn unwind-protect
labels progv
let quote)))
Any language which indicates that SPECIAL-OPERATOR-P returns non-NIL
for symbols other than those in the table is contradicting the
definition of SPECIAL-OPERATOR-P. I say this as an outsider to the
standardization process who does not know what the personal intentions
of the authors of this portion of the standard were. The language
they use is, even for English, very unambiguous here.
-----
3.8.29 special-operator-p Function
Syntax:
special-operator-p symbol generalized-boolean
Arguments and Values:
symbol - a symbol.
generalized-boolean - a generalized boolean.
Description:
Returns true if symbol is a special operator; otherwise, returns false.
Glossary
special operator: n. one of a fixed set of symbols, enumerated in
Figure 3.1.2.1.2.1 Special Forms, that may appear in the car of a
form in order to identify the form as a special form.
-----
Elsewhere,
"It is possible for both macro-function and special-operator-p to
return true of symbol." -- 3.8.11 macro-function
This follows from "An implementation is free to implement a Common
Lisp special operator as a macro." -- 3.1.2.1.2.2 Macro Forms.
"The macro definition must be available for use by programs that
understand only the standard Common Lisp special forms."
-- 3.8.11 macro-function
This vaguely hints that there are special forms other than "standard
Common Lisp" special forms.
"An implementation is free to implement any macro operator as a
special operator, but only if an equivalent definition of the macro
is also provided." -- 3.1.2.2
This is the basis for the argument that special operators can be other
symbols than those listed in the table. But it hinges on the use of
the word "as" which denotes "likeness" or "similarity", not
necessarily "equivalence" (and that's a sticky concept too). Also it
says "implement as" not "define as". Implementation is not the same
as definition, when it comes to language standards.
If special operators were meant to be extended, then it would make
much more sense to me for the language to define special operators as
merely "a set of symbols" which may be extended by the implementor who
defines, through some implementation-specific method, additional
symbols to be special operators. Then SPECIAL-OPERATOR-P could return
true for all such symbols.
And even if "extensions to Common Lisp" can reasonably claim
additional symbols as special operators (something which I am not
entirely certain on, as it really hurts portable code-walkers), there
is still no call for:
(special-operator-p 'cl:cond) => T
in CLISP.
> > (lisp-implementation-type) => "CLISP"
> > (special-operator-p 'system::%handler-bind) => NIL
> > (macro-function 'system::%handler-bind) => NIL
> > (symbol-function 'system::%handler-bind) => NIL
> >
> > Which is it? I think most code-walkers will understand HANDLER-BIND
> > without expanding it, but it shouldn't be necessary.
>
> If it has no macro-function and it is not a special-operator,
> then it is a function, pure and simple. Note that you are looking
> at internals, not at 'cl:handler-bind.
No way can this be a function. At least not according to normal CL
evaluation rules:
[1]> (macroexpand-1 '(handler-bind ((error #'(lambda (c) nil))) t))
(LET
((#:G8594 #'(LAMBDA NIL (PROGN #'(LAMBDA (C) NIL))))
(#:G8595 #'(LAMBDA NIL (PROGN T))))
(LOCALLY (DECLARE (COMPILE))
(SYSTEM::%HANDLER-BIND
((ERROR #'(LAMBDA (CONDITION) (FUNCALL (FUNCALL #:G8594) CONDITION))))
(FUNCALL #:G8595)))) ;
--
;; Matthew Danish -- user: mrd domain: cmu.edu
;; OpenPGP public key: C24B6010 on keyring.debian.org
From: Duane Rettig
Subject: Re: what are allowed special forms? (CLHS clarification)
Date:
Message-ID: <4oefsx5m4.fsf@franz.com>
Matthew Danish <··········@cmu.edu> writes:
> Duane Rettig <·····@franz.com> writes:
>
> > [Matthew, you are theorizing a conflict in the spec, but there is
> > no conflict:]
I believe I was overly strong in my wording here; I now see in
my mailbox a conversation I had with my colleague Steve Haflich,
way back in 1999, who summarized your point of view nicely (though he
disagrees with your point about specificity):
! In reviewing the ANS I find it inconsistent about whether an
! implementation may implement certain macros as special operators. One
! place (the glossary) in the standard specifies explicitly that the set
! of special operators is not extensible, and 3.1.2.1.2.1 supports this
! interpretation for symbols in the CL package.
!
! However, 3.1.2.1.2.2 says
!
! An implementation is free to implement a Common Lisp special operator
! as a macro. An implementation is free to implement any macro operator
! as a special operator, but only if an equivalent definition of the
! macro is also provided.
!
! and is it not evident which of these passages is more specific, and
! therefore which controls.
!
! I believe that 3.1.2.1.2.2 is the traditional understanding of the
! language and that 3.1.2.1.2.1 may be an incautious artifact of he
! drafting work. It might have been intended to say merely that there
! is no way for user code to define new special forms. But that isn't
! what it actually says.
> > Matthew Danish <··········@cmu.edu> writes:
> >
> The definition of special operator contains strong language that is
> incredibly specific. It states that there exists an enumeration of
> all valid special operator names, and references the section that it
> may be found.
>
> I don't see any other cause to explicitly state "a fixed set" unless
> it was meant that section 3.1.2.1.2.1 exhaustively enumerates all
> special operator names, in Common Lisp.
I had resolved in my mind many years ago that this was just a glitch
in the spec, and had not noticed your reference to 3.1.2.1.2.1 -
Sorry about that. However, see below; perhaps a little history
will put it into perspective.
> > So the spec is clear; users cannot add special operators, but the
> > implemetor can.
>
> Of course the implementor can implement operators "like" special
> operators: but they must provide equivalent macroexpansions because
> code-walkers should not have to know about implementation-specific
> optimizations. However, this either has nothing to do with the
> definition of SPECIAL-OPERATOR-P, or is in direct conflict with its
> CLHS entry.
Given the internal conflict in the spec, and my knowledge as to how
special operators are traditionally implemented, I consider this to
be a weak argument. A special operator is a special operator.
> I cannot find any possible other way to read the definition of
> SPECIAL-OPERATOR-P other than:
[strong argument for special-operator-p description
being definitive ... ]
I looked a little at the history of special-operators (aka
"special-forms" in CLtL1). The SPECIAL-FORM-P-MISNOMER
writeup:
http://www.lisp.org/HyperSpec/Issues/iss321-writeup.html
is rather less than wordy - it certainly doesn't contain the
text that eventually made it into the spec. Also, it seems
to have been a late change; special-operator-p is not described
in CLTL2.
However, conmpare the text in 3.1.2.1.2.2:
"An implementation is free to implement a Common Lisp special
operator as a macro. An implementation is free to implement
any macro operator as a special operator, but only if an
equivalent definition of the macro is also provided."
With the text in CLtL2, pp 194 and 195 (unchanged from CLTL,
p 91):
"It is possible for both macro-function and
special-form-p [sic] to be true of a symbol. This is
possible because an implementation is permitted to implement
any macro also as a special-form for speed. On the other hand,
the macro-definition must be available for use by programs that
understand only the standard special forms in table 5-1".
The table 5-1, on p 73 of CLtL2 (as enhanced from p 57 of CLTL)
is effectively the table on 3.1.2.1.2.2. Furthermore, on p
72 of CLtL2 (unchanged from CLTL, p 57) it says
"The set of special forms is fixed in Common Lisp; no way
is provided for the user to define more."
(recognize this one? :-)
It goes on to recommend that the user use macros to create new
syntactic constructs.
However, if you continue reading in CLTL2, p 73, and CLTL, p 57
(note that this section is still section 5.1.3, Special Forms,
in both editions):
"An implementation is free to implement as a macro any
construct described herein as a special form. Conversely,
an implementation is free to implement as a special form any
construct herein described as a macro if an equivalent macro
definition is also provided."
It then goes on to recommmend how a program-analyzing program
(i.e. a code walker) would want to treat these various forms.
From this historical context, I make these observations:
1. The word "fixed" has always been in the text of Common Lisp
descriptions, and has historically been limited describing how
the user sees it.
2. The ability for implementations to cross-implement
special-operators and macros is historical. I do concede
a point made by someone else that the permission only extends
to macros already defined by Common Lisp.
3. The x3J13 writeup gave no discussion about tightening up
what a special-form/special-operator would be; its only purpose
was to effect a name change for clarity.
[ argument as to what the meaning of the word "as" is elided ...]
> > > (lisp-implementation-type) => "CLISP"
> > > (special-operator-p 'system::%handler-bind) => NIL
> > > (macro-function 'system::%handler-bind) => NIL
> > > (symbol-function 'system::%handler-bind) => NIL
> > >
> > > Which is it? I think most code-walkers will understand HANDLER-BIND
> > > without expanding it, but it shouldn't be necessary.
> >
> > If it has no macro-function and it is not a special-operator,
> > then it is a function, pure and simple. Note that you are looking
> > at internals, not at 'cl:handler-bind.
>
> No way can this be a function. At least not according to normal CL
> evaluation rules:
>
> [1]> (macroexpand-1 '(handler-bind ((error #'(lambda (c) nil))) t))
> (LET
> ((#:G8594 #'(LAMBDA NIL (PROGN #'(LAMBDA (C) NIL))))
> (#:G8595 #'(LAMBDA NIL (PROGN T))))
> (LOCALLY (DECLARE (COMPILE))
> (SYSTEM::%HANDLER-BIND
> ((ERROR #'(LAMBDA (CONDITION) (FUNCALL (FUNCALL #:G8594) CONDITION))))
> (FUNCALL #:G8595)))) ;
But if you call macro-function on handler-bind, you get a definition;
i.e. it is a macro. My comment still stands; the spec is clear
on how a code walker must treat a form whose car
is 'system::%handler-bind - it must walk it as if it is a function.
--
Duane Rettig ·····@franz.com Franz Inc. http://www.franz.com/
555 12th St., Suite 1450 http://www.555citycenter.com/
Oakland, Ca. 94607 Phone: (510) 452-2000; Fax: (510) 452-0182
From: Matthew Danish
Subject: Re: what are allowed special forms? (CLHS clarification)
Date:
Message-ID: <87fz13oomx.fsf@mapcar.org>
Duane Rettig <·····@franz.com> writes:
> > No way can this be a function. At least not according to normal CL
> > evaluation rules:
> >
> > [1]> (macroexpand-1 '(handler-bind ((error #'(lambda (c) nil))) t))
> > (LET
> > ((#:G8594 #'(LAMBDA NIL (PROGN #'(LAMBDA (C) NIL))))
> > (#:G8595 #'(LAMBDA NIL (PROGN T))))
> > (LOCALLY (DECLARE (COMPILE))
> > (SYSTEM::%HANDLER-BIND
> > ((ERROR #'(LAMBDA (CONDITION) (FUNCALL (FUNCALL #:G8594) CONDITION))))
> > (FUNCALL #:G8595)))) ;
>
> But if you call macro-function on handler-bind, you get a definition;
> i.e. it is a macro. My comment still stands; the spec is clear
> on how a code walker must treat a form whose car
> is 'system::%handler-bind - it must walk it as if it is a function.
Yes, and it will find itself in an erroneous situation where it
attempts to use the name (ERROR #'(LAMBDA (CONDITION (FUNCALL (FUNCALL
#:G8594) CONDITION)))) as an operator. This is not a problem with the
spec but a bug in CLISP.
--
;; Matthew Danish -- user: mrd domain: cmu.edu
;; OpenPGP public key: C24B6010 on keyring.debian.org
From: Juan Jose Garcia Ripoll
Subject: Re: what are allowed special forms? (CLHS clarification)
Date:
Message-ID: <34o0k4F4ci04uU1@individual.net>
Matthew Danish wrote:
> MACRO-FUNCTION
>
> It is possible for both macro-function and special-operator-p to
> return true of symbol. The macro definition must be available for
> use by programs that understand only the standard Common Lisp
> special forms.
>
> Which seems to imply that it is possible to have non-CL-standard
> special operators. This conflicts with the earlier definition of
> special operators.
I do not think it matters whether an implementation defines special
symbols or not, but the important point is that for any special operator
that is not in the standard, applying MACROEXPAND on this form should
produce an equivalent form that does not rely on some "magic" from the
implementation. This is indeed all what a code walker needs.
This is for instance how it works in ECL with EXT::LAMBDA-BLOCK (a
lambda-form with a block name), or with the expansion of DO/DO* into
EXT::WHILE. Both "primitives" are recognized by the bytecodes compiler,
which then produces slightly more compact code. However, for the benefit
of code walkers, there are macros that produce equivalent code.
Juanjo
From: Matthew Danish
Subject: Re: what are allowed special forms? (CLHS clarification)
Date:
Message-ID: <87oeftnmf8.fsf@mapcar.org>
Juan Jose Garcia Ripoll <·········@yahoo.de> writes:
> I do not think it matters whether an implementation defines special
> symbols or not, but the important point is that for any special
> operator that is not in the standard, applying MACROEXPAND on this
> form should produce an equivalent form that does not rely on some
> "magic" from the implementation. This is indeed all what a code walker
> needs.
That's what the second part of my article talks about. But the
problem here is an operator in CLISP which is neither a special, a
macro, nor a function.
--
;; Matthew Danish -- user: mrd domain: cmu.edu
;; OpenPGP public key: C24B6010 on keyring.debian.org
From: Duane Rettig
Subject: Re: what are allowed special forms? (CLHS clarification)
Date:
Message-ID: <43bx5xfxk.fsf@franz.com>
Matthew Danish <··········@cmu.edu> writes:
> Juan Jose Garcia Ripoll <·········@yahoo.de> writes:
> > I do not think it matters whether an implementation defines special
> > symbols or not, but the important point is that for any special
> > operator that is not in the standard, applying MACROEXPAND on this
> > form should produce an equivalent form that does not rely on some
> > "magic" from the implementation. This is indeed all what a code walker
> > needs.
>
> That's what the second part of my article talks about. But the
> problem here is an operator in CLISP which is neither a special, a
> macro, nor a function.
See 3.1.2.1.2, second paragraph, last sentence.
--
Duane Rettig ·····@franz.com Franz Inc. http://www.franz.com/
555 12th St., Suite 1450 http://www.555citycenter.com/
Oakland, Ca. 94607 Phone: (510) 452-2000; Fax: (510) 452-0182
From: Sam Steingold
Subject: Re: what are allowed special forms? (CLHS clarification)
Date:
Message-ID: <uekgpxdkk.fsf@gnu.org>
> * Joerg Hoehle <······@hfref.fbheprsbetr.arg> [2005-01-13 15:50:49 +0100]:
>
> The problem I'm currently facing is that the Iterate package, which
> relies on a code walker, may stumble on some forms which are or expand
> to implementation-defined special operators.
why does Iterate need a code walker?!
LOOP and SERIES work without one.
--
Sam Steingold (http://www.podval.org/~sds) running w2k
<http://www.camera.org> <http://www.iris.org.il> <http://www.memri.org/>
<http://www.mideasttruth.com/> <http://www.honestreporting.com>
You can have it good, soon or cheap. Pick two...
From: Raymond Toy
Subject: Re: what are allowed special forms? (CLHS clarification)
Date:
Message-ID: <sxdacrdq98d.fsf@rtp.ericsson.se>
>>>>> "Sam" == Sam Steingold <···@gnu.org> writes:
>> * Joerg Hoehle <······@hfref.fbheprsbetr.arg> [2005-01-13 15:50:49 +0100]:
>>
>> The problem I'm currently facing is that the Iterate package, which
>> relies on a code walker, may stumble on some forms which are or expand
>> to implementation-defined special operators.
Sam> why does Iterate need a code walker?!
Sam> LOOP and SERIES work without one.
SERIES has its own code walker.
Ray
In article <·············@gnu.org>, Sam Steingold <···@gnu.org> wrote:
> > * Joerg Hoehle <······@hfref.fbheprsbetr.arg> [2005-01-13 15:50:49 +0100]:
> >
> > The problem I'm currently facing is that the Iterate package, which
> > relies on a code walker, may stumble on some forms which are or expand
> > to implementation-defined special operators.
>
> why does Iterate need a code walker?!
> LOOP and SERIES work without one.
Typically it's necessary for optimization. The alternative is to make
extensive use of closures, and these are likely to have performance
impact. Since the last place you want a performance hit is in a loop,
optimizing the expansion of iteration constructs is important.
--
Barry Margolin, ······@alum.mit.edu
Arlington, MA
*** PLEASE post questions in newsgroups, not directly to me ***
Sam Steingold <···@gnu.org> writes:
> why does Iterate need a code walker?!
> LOOP and SERIES work without one.
LOOP has been designed specially so as to not depend on code
expansion. It comes with its own restricted grammar, and even advanced
LOOP users sometimes write gramatically incorrect loops. Usually,
users get an error during macroexpansion and rewrite.
A typical restriction of LOOP is unability to COLLECT across nested
loops (don't talk me about using NCONC in the outer loop(s)).
Iterate can do that.
Expansion of loop only requires "superficial" analysis of the loop
body (look up clauses etc.) but never go inside user code. That
doesn't mean that implementing the loop macro correctly isn't hard.
As far as SERIES is concerned, I don't know the sources but I wouldn't
believe that redefining LET and/or DEFUN like SERIES does is enough
for the amount of code-rewriting that SERIES does. It does walk code,
and that's e.g. why Raymond Toy was pleased a couple of years ago to
find a form in CLISP (expand-form) to help him expand MACROLET
correctly (cf. clisp-list archives of December 2002).
I've tried to think of a variant of Iterate which would require more
to be declared in its top-level (e.g. require WITH clauses), not
depend on a code-walker, while still allowing deeply nested COLLECT
expressions etc., posted a little to cll these months, but my thoughts
have made like no progress in this area.
Regards,
Jorg Hohle
Telekom/T-Systems Technology Center
From: Sam Steingold
Subject: Re: what are allowed special forms? (CLHS clarification)
Date:
Message-ID: <uacrcvzac.fsf@gnu.org>
> * Joerg Hoehle <······@hfref.fbheprsbetr.arg> [2005-01-14 13:00:33 +0100]:
>
> Sam Steingold <···@gnu.org> writes:
>> why does Iterate need a code walker?!
>> LOOP and SERIES work without one.
<series does not!>
> A typical restriction of LOOP is unability to COLLECT across nested
> loops (don't talk me about using NCONC in the outer loop(s)).
> Iterate can do that.
WITH-COLLECT in CLOCC/CLLIB/simple.lisp can do that too
- with a simple macrolet.
I can understand a code analysis tool requiring a code walker,
but an iteration macro should not depend on one, IMHO.
As a user, I would not want to rely on such a tool
(unless it replies on the implementation's built-in code walker).
--
Sam Steingold (http://www.podval.org/~sds) running w2k
<http://www.camera.org> <http://www.iris.org.il> <http://www.memri.org/>
<http://www.mideasttruth.com/> <http://www.honestreporting.com>
Time would have been the best Teacher, if it did not kill all its students.
From: Pascal Costanza
Subject: Re: what are allowed special forms? (CLHS clarification)
Date:
Message-ID: <cs6a6q$6hm$1@snic.vub.ac.be>
Joerg Hoehle wrote:
> Currently, I'd resume my experience as follows: "the Iterate package
> depends on a code walker. Since code walking may stumble on arbitrary
> implementation-defined special operators as the result of expanding
> portable code, it cannot be expected to work portably. Therefore it
> can be viewed as poor design. Similarly, any purportedly portable
> package which depends on a code-walker can be viewed as poor design,
> if not otherwise restricted to particular usages or situations. YMMV"
>
> What do you think?
I think that's too harsh. I think this is a vendor's issue, not one of a
library provider. If CL vendors want to make sure that their
respective CL implementations work with such libraries, they should not
include additional special operators. It's good enough for the library
provider to signal appropriate errors.
Pascal
From: Duane Rettig
Subject: Re: what are allowed special forms? (CLHS clarification)
Date:
Message-ID: <4fz15xgtf.fsf@franz.com>
Pascal Costanza <··@p-cos.net> writes:
> Joerg Hoehle wrote:
>
> > Currently, I'd resume my experience as follows: "the Iterate package
> > depends on a code walker. Since code walking may stumble on arbitrary
> > implementation-defined special operators as the result of expanding
> > portable code, it cannot be expected to work portably. Therefore it
> > can be viewed as poor design. Similarly, any purportedly portable
> > package which depends on a code-walker can be viewed as poor design,
> > if not otherwise restricted to particular usages or situations. YMMV"
> > What do you think?
>
>
> I think that's too harsh. I think this is a vendor's issue, not one of
> a library provider. If CL vendors want to make sure that their
> respective CL implementations work with such libraries, they should
> not include additional special operators. It's good enough for the
> library provider to signal appropriate errors.
Neither of these views is correct. They both ignore the provision
that the CL spec makes for implementations to define extra special
operators (each must have macro definitions as well which provide
equivalent functionality - see 3.1.2.1.2.2, last paragraph).
If the implementation correctly provides a macro for the special
operator it defines, then a code walker can easily analyze that
special-form by simply relying on macroexpand to expand through the
special form to something that is functional or which is defined
in terms of the known special operators. And once that point is
reached, the walker knows what to do - it must of course know how
to walk a functional form, and it must also know the pattern of
walking for those special forms that are defined by the spec.
--
Duane Rettig ·····@franz.com Franz Inc. http://www.franz.com/
555 12th St., Suite 1450 http://www.555citycenter.com/
Oakland, Ca. 94607 Phone: (510) 452-2000; Fax: (510) 452-0182
From: David Steuber
Subject: Re: what are allowed special forms? (CLHS clarification)
Date:
Message-ID: <874qhlqbuu.fsf@david-steuber.com>
Duane Rettig <·····@franz.com> writes:
> Neither of these views is correct. They both ignore the provision
> that the CL spec makes for implementations to define extra special
> operators (each must have macro definitions as well which provide
> equivalent functionality - see 3.1.2.1.2.2, last paragraph).
Is a macro allowed to expand into itself? Or must it expand into
something else?
--
An ideal world is left as an excercise to the reader.
--- Paul Graham, On Lisp 8.1
From: Duane Rettig
Subject: Re: what are allowed special forms? (CLHS clarification)
Date:
Message-ID: <4wtuhvx8e.fsf@franz.com>
David Steuber <·····@david-steuber.com> writes:
> Duane Rettig <·····@franz.com> writes:
>
> > Neither of these views is correct. They both ignore the provision
> > that the CL spec makes for implementations to define extra special
> > operators (each must have macro definitions as well which provide
> > equivalent functionality - see 3.1.2.1.2.2, last paragraph).
>
> Is a macro allowed to expand into itself? Or must it expand into
> something else?
3.4.4 describes the Macro Lambda List, which includes the &whole
variable. This variable could probably be used as the return
value, but I believe that it was intended for other things, such as
compiler-macros, which can decline to expand by returning the &whole
variable.
Now let's examine macroexpand/macroexpand-1: The descriptions of
these functions are tied together so much that they are described
on the same page. The description says:
- If form is a macro form, then macroexpand-1 expands the macro
form call once.
- macroexpand repeatedly expands form until it is no longer a macro
form.
So it seems that if you were to define a macro that returns its &whole,
macroexpand would get caught into an infinite loop. I don't think
that this is disallowed, but it certainly is unfriendly.
--
Duane Rettig ·····@franz.com Franz Inc. http://www.franz.com/
555 12th St., Suite 1450 http://www.555citycenter.com/
Oakland, Ca. 94607 Phone: (510) 452-2000; Fax: (510) 452-0182
From: Peter Seibel
Subject: Re: what are allowed special forms? (CLHS clarification)
Date:
Message-ID: <m3fz157zp9.fsf@javamonkey.com>
Duane Rettig <·····@franz.com> writes:
> So it seems that if you were to define a macro that returns its
> &whole, macroexpand would get caught into an infinite loop. I don't
> think that this is disallowed, but it certainly is unfriendly.
From the DEFMACRO dictionary entry:
"Recursive expansion of the form returned must terminate, including
the expansion of other macros which are subforms of other forms
returned."
-Peter
--
Peter Seibel ·····@javamonkey.com
Lisp is the red pill. -- John Fraser, comp.lang.lisp
From: Duane Rettig
Subject: Re: what are allowed special forms? (CLHS clarification)
Date:
Message-ID: <4sm55vv3a.fsf@franz.com>
Peter Seibel <·····@javamonkey.com> writes:
> Duane Rettig <·····@franz.com> writes:
>
> > So it seems that if you were to define a macro that returns its
> > &whole, macroexpand would get caught into an infinite loop. I don't
> > think that this is disallowed, but it certainly is unfriendly.
>
> From the DEFMACRO dictionary entry:
>
> "Recursive expansion of the form returned must terminate, including
> the expansion of other macros which are subforms of other forms
> returned."
Thanks, I stand corrected. And I'm glad that's there :-)
--
Duane Rettig ·····@franz.com Franz Inc. http://www.franz.com/
555 12th St., Suite 1450 http://www.555citycenter.com/
Oakland, Ca. 94607 Phone: (510) 452-2000; Fax: (510) 452-0182
Duane Rettig <·····@franz.com> writes:
> Peter Seibel <·····@javamonkey.com> writes:
> > Duane Rettig <·····@franz.com> writes:
> > > So it seems that if you were to define a macro that returns its
> > > &whole, macroexpand would get caught into an infinite loop. I don't
> > > think that this is disallowed, but it certainly is unfriendly.
> >
> > From the DEFMACRO dictionary entry:
> > "Recursive expansion of the form returned must terminate, including
> > the expansion of other macros which are subforms of other forms
> > returned."
> Thanks, I stand corrected. And I'm glad that's there :-)
Duane, since you're from Allegro, would you take statement to change
Allegro 7.0 beta (and possibly other versions?) to macroexpand the
COND macro into some other form?
There was a note about this in the iterate-devel mailing list in
September 2004, causing Iterate to enter an inifinite loop (or stack overflow):
http://common-lisp.net/pipermail/iterate-devel/2004-September/000002.html
:> (macroexpand '(cond ((>= value 10) nil (collect (cons key (next i))))) nil)
:(COND ((>= VALUE 10) NIL (COLLECT (CONS KEY (NEXT I))))) NIL
I believe this to be clearly a bug in ACL 7.0beta in the light of this
thread, but I have no idea whether this also affects other versions of
ACL. So I write this to remind you to please check for that.
Regards,
Jorg Hohle
Telekom/T-Systems Technology Center
From: Duane Rettig
Subject: Re: what are allowed special forms? (CLHS clarification)
Date:
Message-ID: <48y6warpn.fsf@franz.com>
Joerg Hoehle <······@users.sourceforge.net> writes:
> Duane Rettig <·····@franz.com> writes:
> > Peter Seibel <·····@javamonkey.com> writes:
> > > Duane Rettig <·····@franz.com> writes:
> > > > So it seems that if you were to define a macro that returns its
> > > > &whole, macroexpand would get caught into an infinite loop. I don't
> > > > think that this is disallowed, but it certainly is unfriendly.
> > >
> > > From the DEFMACRO dictionary entry:
> > > "Recursive expansion of the form returned must terminate, including
> > > the expansion of other macros which are subforms of other forms
> > > returned."
> > Thanks, I stand corrected. And I'm glad that's there :-)
>
> Duane, since you're from Allegro, would you take statement to change
> Allegro 7.0 beta (and possibly other versions?) to macroexpand the
> COND macro into some other form?
>
> There was a note about this in the iterate-devel mailing list in
> September 2004, causing Iterate to enter an inifinite loop (or stack overflow):
>
> http://common-lisp.net/pipermail/iterate-devel/2004-September/000002.html
> :> (macroexpand '(cond ((>= value 10) nil (collect (cons key (next i))))) nil)
> :(COND ((>= VALUE 10) NIL (COLLECT (CONS KEY (NEXT I))))) NIL
>
> I believe this to be clearly a bug in ACL 7.0beta in the light of this
> thread, but I have no idea whether this also affects other versions of
> ACL. So I write this to remind you to please check for that.
Yes, 7.0.beta broke several macroexpansion situations, and that is why
I'm so close to these issues. But then, that's what a beta is for :-)
In 7.0 final, there is no such infinite loop:
CL-USER(1): (macroexpand '(cond ((>= value 10) nil (collect (cons key (next i))))) nil)
(IF (>= VALUE 10) (PROGN NIL (COLLECT (CONS KEY (NEXT I)))) NIL)
T
CL-USER(2): (macroexpand *)
(IF (>= VALUE 10) (PROGN NIL (COLLECT (CONS KEY (NEXT I)))) NIL)
NIL
CL-USER(3):
--
Duane Rettig ·····@franz.com Franz Inc. http://www.franz.com/
555 12th St., Suite 1450 http://www.555citycenter.com/
Oakland, Ca. 94607 Phone: (510) 452-2000; Fax: (510) 452-0182
From: David Steuber
Subject: Re: what are allowed special forms? (CLHS clarification)
Date:
Message-ID: <87k6qflnex.fsf@david-steuber.com>
Duane Rettig <·····@franz.com> writes:
> Peter Seibel <·····@javamonkey.com> writes:
>
> > Duane Rettig <·····@franz.com> writes:
> >
> > > So it seems that if you were to define a macro that returns its
> > > &whole, macroexpand would get caught into an infinite loop. I don't
> > > think that this is disallowed, but it certainly is unfriendly.
> >
> > From the DEFMACRO dictionary entry:
> >
> > "Recursive expansion of the form returned must terminate, including
> > the expansion of other macros which are subforms of other forms
> > returned."
>
> Thanks, I stand corrected. And I'm glad that's there :-)
Infinite recursion in a macro would certainly slow down compilation.
So how is LAMBDA handled?
CL-USER> (macroexpand-1 '(lambda (x) x))
#'(LAMBDA (X) X)
T
Does the #' magicly inhibit further expansion? LAMBDA is not in
figure 3-2. It would appear that it does:
CL-USER> (macroexpand-1 '(function (lambda (x) x)))
#'(LAMBDA (X) X)
NIL
Of course, FUNCTION _is_ in figure 3-2. I suppose that would answer
my question.
--
An ideal world is left as an excercise to the reader.
--- Paul Graham, On Lisp 8.1
From: Sam Steingold
Subject: Re: what are allowed special forms? (CLHS clarification)
Date:
Message-ID: <uwtuhvwr8.fsf@gnu.org>
> * Duane Rettig <·····@senam.pbz> [2005-01-13 11:35:56 -0800]:
>
> CL spec makes for implementations to define extra special operators
> (each must have macro definitions as well which provide equivalent
> functionality - see 3.1.2.1.2.2, last paragraph).
You appear to refer to this:
<http://www.lisp.org/HyperSpec/Body/sec_3-1-2-1-2-2.html>:
An implementation is free to implement a Common Lisp special
operator as a macro. An implementation is free to implement any
macro operator as a special operator, but only if an equivalent
definition of the macro is also provided.
I parse the last sentence to refer only to CL standard macro operators
because implementation-specific special operators are not "macro
operators" and thus the term "any macro operators" is not applicable to
them.
Therefore an implementation is not required to provide a macro
equivalent to its internal special operator
(in fact, I would think that such a requirement is quite onerous).
BTW, why is HANDLER-BIND a macro and not a special operator?
--
Sam Steingold (http://www.podval.org/~sds) running w2k
<http://www.camera.org> <http://www.iris.org.il> <http://www.memri.org/>
<http://www.mideasttruth.com/> <http://www.honestreporting.com>
Cannot handle the fatal error due to a fatal error in the fatal error handler.
From: Duane Rettig
Subject: Re: what are allowed special forms? (CLHS clarification)
Date:
Message-ID: <4k6qgx4wh.fsf@franz.com>
Sam Steingold <···@gnu.org> writes:
> > * Duane Rettig <·····@senam.pbz> [2005-01-13 11:35:56 -0800]:
> >
> > CL spec makes for implementations to define extra special operators
> > (each must have macro definitions as well which provide equivalent
> > functionality - see 3.1.2.1.2.2, last paragraph).
>
> You appear to refer to this:
> <http://www.lisp.org/HyperSpec/Body/sec_3-1-2-1-2-2.html>:
> An implementation is free to implement a Common Lisp special
> operator as a macro. An implementation is free to implement any
> macro operator as a special operator, but only if an equivalent
> definition of the macro is also provided.
>
> I parse the last sentence to refer only to CL standard macro operators
> because implementation-specific special operators are not "macro
> operators" and thus the term "any macro operators" is not applicable to
> them.
I agree - the spec only addresses those operators defined as macros
by Common Lisp.
> Therefore an implementation is not required to provide a macro
> equivalent to its internal special operator
> (in fact, I would think that such a requirement is quite onerous).
I would argue exactly the opposite way. If we as implementors
define our own special-operators, and if there is any chance that
a form with that special operator will be seen by a code walker,
what is the code walker supposed to do with it? How does it
know what the syntax is?
I believe that in order to ensure that code-walkers can be written
portably, the implementor _must_ provide a macro that allows the
walker to expand that form and get past it.
> BTW, why is HANDLER-BIND a macro and not a special operator?
Since it is a macro, it could be a special-form. From my reading
in order to answer Matthew's question, I noticed that CLtL and
CLtL2 both explicitly mention the desire to keep the set of
special-operators small - from CLTL2, p 72:
"The set of special forms [sic] in Common Lisp is purposely kept
very small because any program-analyzing program must have special
knowledge about every type of special form"
It goes on to mention the desirability of some such analyzers,
e.g. compilers, to have such special knowledge, and explicitly
mentions typecase and multiple-value-bind as candidates for more
efficient compilation. It does not mention handler-bind, but of
course it does allow for it anyway later.
--
Duane Rettig ·····@franz.com Franz Inc. http://www.franz.com/
555 12th St., Suite 1450 http://www.555citycenter.com/
Oakland, Ca. 94607 Phone: (510) 452-2000; Fax: (510) 452-0182
Duane Rettig <·····@franz.com> writes:
> Sam Steingold <···@gnu.org> writes:
>
> > > * Duane Rettig <·····@senam.pbz> [2005-01-13 11:35:56 -0800]:
> > >
> > > CL spec makes for implementations to define extra special operators
> > > (each must have macro definitions as well which provide equivalent
> > > functionality - see 3.1.2.1.2.2, last paragraph).
> >
> > You appear to refer to this:
> > <http://www.lisp.org/HyperSpec/Body/sec_3-1-2-1-2-2.html>:
> > An implementation is free to implement a Common Lisp special
> > operator as a macro. An implementation is free to implement any
> > macro operator as a special operator, but only if an equivalent
> > definition of the macro is also provided.
> >
> > I parse the last sentence to refer only to CL standard macro operators
> > because implementation-specific special operators are not "macro
> > operators" and thus the term "any macro operators" is not applicable to
> > them.
>
> I agree - the spec only addresses those operators defined as macros
> by Common Lisp.
>
> > Therefore an implementation is not required to provide a macro
> > equivalent to its internal special operator
> > (in fact, I would think that such a requirement is quite onerous).
>
> I would argue exactly the opposite way. If we as implementors
> define our own special-operators, and if there is any chance that
> a form with that special operator will be seen by a code walker,
> what is the code walker supposed to do with it? How does it
> know what the syntax is?
>
> I believe that in order to ensure that code-walkers can be written
> portably, the implementor _must_ provide a macro that allows the
> walker to expand that form and get past it.
I agree that it would be desirable. But what if the implementation
specific special operator implements a feature that is not
implementable (in less than a virtual machine) with COMMON-LISP
special operators?
eg. the special operators to implement continuations in Common Lisp.
Really, it's hard to ask the implementors to provide a whole virtual
machine implemented in Common-Lisp to explain to the code walkers the
semantics of their special operators.
--
__Pascal Bourguignon__ http://www.informatimago.com/
Until real software engineering is developed, the next best practice
is to develop with a dynamic system that has extreme late binding in
all aspects. The first system to really do this in an important way
is Lisp. -- Alan Kay
From: Duane Rettig
Subject: Re: what are allowed special forms? (CLHS clarification)
Date:
Message-ID: <44qhjc32c.fsf@franz.com>
Pascal Bourguignon <····@mouse-potato.com> writes:
> Duane Rettig <·····@franz.com> writes:
>
> > I believe that in order to ensure that code-walkers can be written
> > portably, the implementor _must_ provide a macro that allows the
> > walker to expand that form and get past it.
>
> I agree that it would be desirable. But what if the implementation
> specific special operator implements a feature that is not
> implementable (in less than a virtual machine) with COMMON-LISP
> special operators?
>
> eg. the special operators to implement continuations in Common Lisp.
>
> Really, it's hard to ask the implementors to provide a whole virtual
> machine implemented in Common-Lisp to explain to the code walkers the
> semantics of their special operators.
I've seen this kind of thing said before on this thread, but I don't
understand why it has been said: why do you limit what a macro can
expand to to be in terms of special-operators? What is wrong with
a macroexpansion for an implementation-specific special-operator
expanding to just a function call? Or perhaps a combination of
function calls and standard special-operators? The function
syntax is far more common in Common Lisp than special operators
are; it is the default for parsing a form if the symbol in the car
of the form has no other definition.
Remember, the whole name of this walking game is understanding the
syntax of the special form; if some element is to be evaluated out
of the normal left-to-right order, it can be easily expanded by a
macro that switches the order. If an element is to be conditionally
evaluated, then it might expand into an IF form or a series of the
same. I've never been interested in analyzing this, but if memory
serves correctly, some have said that call/cc can be implemented in
terms of unwind-protect (and/or catch/throw? whatever...).
And so on...
--
Duane Rettig ·····@franz.com Franz Inc. http://www.franz.com/
555 12th St., Suite 1450 http://www.555citycenter.com/
Oakland, Ca. 94607 Phone: (510) 452-2000; Fax: (510) 452-0182
Duane Rettig <·····@franz.com> writes:
> Pascal Bourguignon <····@mouse-potato.com> writes:
>
> > Duane Rettig <·····@franz.com> writes:
> >
> > > I believe that in order to ensure that code-walkers can be written
> > > portably, the implementor _must_ provide a macro that allows the
> > > walker to expand that form and get past it.
> >
> > I agree that it would be desirable. But what if the implementation
> > specific special operator implements a feature that is not
> > implementable (in less than a virtual machine) with COMMON-LISP
> > special operators?
> >
> > eg. the special operators to implement continuations in Common Lisp.
> >
> > Really, it's hard to ask the implementors to provide a whole virtual
> > machine implemented in Common-Lisp to explain to the code walkers the
> > semantics of their special operators.
>
> I've seen this kind of thing said before on this thread, but I don't
> understand why it has been said: why do you limit what a macro can
> expand to to be in terms of special-operators? What is wrong with
> a macroexpansion for an implementation-specific special-operator
> expanding to just a function call? Or perhaps a combination of
> function calls and standard special-operators? The function
> syntax is far more common in Common Lisp than special operators
> are; it is the default for parsing a form if the symbol in the car
> of the form has no other definition.
>
> Remember, the whole name of this walking game is understanding the
> syntax of the special form; if some element is to be evaluated out
> of the normal left-to-right order, it can be easily expanded by a
> macro that switches the order. If an element is to be conditionally
> evaluated, then it might expand into an IF form or a series of the
> same. I've never been interested in analyzing this, but if memory
> serves correctly, some have said that call/cc can be implemented in
> terms of unwind-protect (and/or catch/throw? whatever...).
>
> And so on...
Yes, in practice it must be simplier than what I imagine in theory.
--
__Pascal Bourguignon__ http://www.informatimago.com/
I need a new toy.
Tail of black dog keeps good time.
Pounce! Good dog! Good dog!
Duane Rettig <·····@franz.com> writes:
> What is wrong with
> a macroexpansion for an implementation-specific special-operator
> expanding to just a function call? [...]
> Remember, the whole name of this walking game is understanding the
> syntax of the special form;
That's is IMHO a too limietd view on it. It's good enough for the XREF
package, but not for Iterate (as currently implemented).
Iterate requires the code walker to expand to working code, because it
will return that expansion as part of its own.
It does is so that because it needs to find out COLLECT INTO and other
forms deep inside its body to know which variables to bind:
(iterate ... (when (foo) (collect bar into x)))
has Iterate bind a variable named x around the body.
Depending on that requirement may be considered poor design. I raised
a thread in Nov 2004 about how to lift that requirement.
Subject: Re: Macroexpansion, bindings and information flow (Iterate package)
I'd very much appreciate hints.
One idea that right now occurs to me could be two passes: one
"throw-away" pass just to find out about necessary bindings (or
whatever else), then produce the expansion (certainly involving
MACROLET). Afterwards the compiler will do the actual expansion for
code generation.
Could that possibly work? It would still be sensitive to misleading cases like
(let (x 123) (iterate .... (use x) (collect foo into x)))
since x would be seen as a refering to LET while expanding Iterate,
whereas it would refer to Iterate's binding in the second pass...
That would become even worse with MACROLET or SYMBOL-MACROLET.
So two passes don't seem to buy anything.
Regards,
Jorg Hohle
Telekom/T-Systems Technology Center
From: Duane Rettig
Subject: Re: what are allowed special forms? (CLHS clarification)
Date:
Message-ID: <4vf9zaiwn.fsf@franz.com>
Joerg Hoehle <······@users.sourceforge.net> writes:
> Duane Rettig <·····@franz.com> writes:
> > What is wrong with
> > a macroexpansion for an implementation-specific special-operator
> > expanding to just a function call? [...]
> > Remember, the whole name of this walking game is understanding the
> > syntax of the special form;
>
> That's is IMHO a too limietd view on it. It's good enough for the XREF
> package, but not for Iterate (as currently implemented).
>
> Iterate requires the code walker to expand to working code, because it
> will return that expansion as part of its own.
OK, I suppose I should have been more careful in the way I stated it.
Replace "the whole name of" with "a major part of".
> It does is so that because it needs to find out COLLECT INTO and other
> forms deep inside its body to know which variables to bind:
> (iterate ... (when (foo) (collect bar into x)))
> has Iterate bind a variable named x around the body.
>
> Depending on that requirement may be considered poor design. I raised
> a thread in Nov 2004 about how to lift that requirement.
> Subject: Re: Macroexpansion, bindings and information flow (Iterate package)
> I'd very much appreciate hints.
>
> One idea that right now occurs to me could be two passes: one
> "throw-away" pass just to find out about necessary bindings (or
> whatever else), then produce the expansion (certainly involving
> MACROLET). Afterwards the compiler will do the actual expansion for
> code generation.
>
> Could that possibly work? It would still be sensitive to misleading cases like
> (let (x 123) (iterate .... (use x) (collect foo into x)))
> since x would be seen as a refering to LET while expanding Iterate,
> whereas it would refer to Iterate's binding in the second pass...
> That would become even worse with MACROLET or SYMBOL-MACROLET.
[ I assume you mean "(let ((x 123)) (iterate ..." here]
I looked over that thread just now. I don't know iterate, but I
see that I probably should get to know it; it looks like a very good
candidate to make good use of my Environments Access module.
If it can know what it is seeing in the environment argument, rather
than an opaque black-box, perhaps then it could perform some of
the optimizations and functional transformations that are being asked
of it. And besides, that would give me a good subject to write an
ILC2005 paper!
What is the best location to get the latest version?
--
Duane Rettig ·····@franz.com Franz Inc. http://www.franz.com/
555 12th St., Suite 1450 http://www.555citycenter.com/
Oakland, Ca. 94607 Phone: (510) 452-2000; Fax: (510) 452-0182
From: Matthew Danish
Subject: Re: what are allowed special forms? (CLHS clarification)
Date:
Message-ID: <87brbroo0d.fsf@mapcar.org>
Pascal Bourguignon <····@mouse-potato.com> writes:
> eg. the special operators to implement continuations in Common Lisp.
Bad example; I think that any such special operators can easily be
converted into a functional form. In fact I can only think of one
special operator that might be used for this, and it is easily a
macro:
(defmacro let/cc (var &body body)
`(call/cc #'(lambda (,var) ,@body)))
Continuations can easily be expressed as functions so they don't
require any special syntax.
--
;; Matthew Danish -- user: mrd domain: cmu.edu
;; OpenPGP public key: C24B6010 on keyring.debian.org
From: Kalle Olavi Niemitalo
Subject: Re: what are allowed special forms? (CLHS clarification)
Date:
Message-ID: <87u0pjkcja.fsf@Astalo.kon.iki.fi>
Sam Steingold <···@gnu.org> writes:
> I parse the last sentence to refer only to CL standard macro operators
> because implementation-specific special operators are not "macro
> operators" and thus the term "any macro operators" is not applicable to
> them.
Yeah, that's how I read it too. However, that means it also
allows an implementation to expand any standard macro CL:FOO to a
special operator EXT:FOO that does not have a macro definition.
That makes it rather pointless to forbid defining CL:FOO directly
as a non-macro special operator.
From: Bruno Haible
Subject: Re: what are allowed special forms? (CLHS clarification)
Date:
Message-ID: <cs8ng0$m5g$1@laposte.ilog.fr>
Duane Rettig wrote:
> If the implementation correctly provides a macro for the special
> operator it defines, then a code walker can easily analyze that
> special-form by simply relying on macroexpand to expand through the
> special form to something that is functional or which is defined
> in terms of the known special operators.
OK, and what do you do for special-operators that cannot be emulated
by other elements in the language? COMPILER-LET is the best known
example of this category.
In clisp:
[1]> (special-operator-p 'ext:compiler-let)
T
[2]> (macro-function 'ext:compiler-let)
NIL
In Allegro CL:
CL-USER(1): (special-operator-p 'excl::compiler-let)
#<special operator COMPILER-LET @ #x71042872>
CL-USER(2): (macro-function 'excl::compiler-let)
NIL
Code-walkers do need special knowledge about these, otherwise they
cannot walk forms like
(symbol-macrolet ((foo bar))
(macrolet ((foo (x) `(baz, x)))
(compiler-let ((foo foo))
(foo foo))))
Bruno
From: Duane Rettig
Subject: Re: what are allowed special forms? (CLHS clarification)
Date:
Message-ID: <4zmzbamik.fsf@franz.com>
Bruno Haible <·····@clisp.org> writes:
> Duane Rettig wrote:
> > If the implementation correctly provides a macro for the special
> > operator it defines, then a code walker can easily analyze that
> > special-form by simply relying on macroexpand to expand through the
> > special form to something that is functional or which is defined
> > in terms of the known special operators.
>
> OK, and what do you do for special-operators that cannot be emulated
> by other elements in the language? COMPILER-LET is the best known
> example of this category.
>
> In clisp:
> [1]> (special-operator-p 'ext:compiler-let)
> T
> [2]> (macro-function 'ext:compiler-let)
> NIL
>
> In Allegro CL:
> CL-USER(1): (special-operator-p 'excl::compiler-let)
> #<special operator COMPILER-LET @ #x71042872>
> CL-USER(2): (macro-function 'excl::compiler-let)
> NIL
Heh; it's interesting that you should pick this one. I consider
compiler-let to be in a very special category, it is only one
of two special-operators to be removed as a special-operator
(having originally been a special-form in CLtL). Furthermore,
it is the _only_ special operator to be removed entirely from the
language - cl:compiler-let does not even exist anymore.
That it does not have a macro-function is not a bug, per the
ansi spec, since the spec effectively only addresses those
special-operators that are defined as macros in the spec. But
I do consider it to be unfriendly, with respect to code walkers.
Please understand my point of view here: I've been arguing
in this thread specifically from the spec - both its literal
meaning and its spirit/history, but my main motivation is
oriented around something beyond the spec: the Environments
Access module and what it can do to make code-walkers more
accessible and programmable. As such, all of the _extra_
things I've been talking about, such as variable-information,
function-information, augment-environment, are done with a
major goal of tracking the spec as closely as possible, but
with a larger goal of taking us beyond the spec (without
alienating ourselves from it).
It is for this reason that I argue that special operators should
always have macro definitions, so as to allow code walkers to
portably walk all reasonably-generated code.
Consider this: if we believe that Common Lisp can truly write
Common Lisp (and other languages) and if we find that there is
a special operator that can be shown not to be implementable in
Common lisp (i.e. by macroexpansion into either standard
special-forms or function-forms) then we've found a hole in the
language, and it should be plugged, at least intellectually.
I have not tried to write a macro for compiler-let, so I can't
say whether it is one of those items that perhaps should not
have been removed from the language.
> Code-walkers do need special knowledge about these, otherwise they
> cannot walk forms like
>
> (symbol-macrolet ((foo bar))
> (macrolet ((foo (x) `(baz, x)))
> (compiler-let ((foo foo))
> (foo foo))))
Yes, that's true. Why doesn't it come up? I have three theories:
1. It does come up, and people just punt or end up with implementation-
specific applications because their code walker doesn't know about
compiler-let.
2. Many of the code walkers that have been well-tested were started
in the CLTL days, before compiler-let had been removed. As a result,
each has had to deal with the loss in whatever way they saw fit, but
it is likely, if they are likely to generate compiler-let forms at
all, that they _do_ have special-knowledge of compiler-let. If it
walks like a duck, talks like a duck, looks like a duck, but is
mandated not to be a duck, then ... ?
3. Code walkers simply don't tend to see compiler-let forms. In
your example case, the code-walker might see a bad syntax for an
unknown function form (i.e. '(foo foo) is a bad function for the
first argument to the function compiler-let) or it might just give
up and return the form unchanged and unwalked. There may be other
more clever things that could be done, but I suspect that none of
them involve successful (and portable) walking of the form.
--
Duane Rettig ·····@franz.com Franz Inc. http://www.franz.com/
555 12th St., Suite 1450 http://www.555citycenter.com/
Oakland, Ca. 94607 Phone: (510) 452-2000; Fax: (510) 452-0182
From: Sam Steingold
Subject: Re: what are allowed special forms? (CLHS clarification)
Date:
Message-ID: <uzmzbvmbj.fsf@gnu.org>
> * Duane Rettig <·····@senam.pbz> [2005-01-14 10:33:39 -0800]:
>
> Consider this: if we believe that Common Lisp can truly write Common
>Lisp (and other languages) and if we find that there is a special
>operator that can be shown not to be implementable in Common lisp
>(i.e. by macroexpansion into either standard special-forms or
>function-forms) then we've found a hole in the language, and it should
>be plugged, at least intellectually.
The issue is _not_ whether HANDLER-BIND can be implemented as a macro in
CLISP (it can be and there is already commented-out code for that from
times immemorial).
The issue is that this macro implementation is suboptimal, so it is
suitable for code analysis tools (like XREF) but _NOT_ code generation
tools (lie ITERATE and SERIES).
--
Sam Steingold (http://www.podval.org/~sds) running w2k
<http://www.camera.org> <http://www.iris.org.il> <http://www.memri.org/>
<http://www.mideasttruth.com/> <http://www.honestreporting.com>
My other CAR is a CDR.
From: Duane Rettig
Subject: Re: what are allowed special forms? (CLHS clarification)
Date:
Message-ID: <4mzvbag2h.fsf@franz.com>
Sam Steingold <···@gnu.org> writes:
> > * Duane Rettig <·····@senam.pbz> [2005-01-14 10:33:39 -0800]:
> >
> > Consider this: if we believe that Common Lisp can truly write Common
> >Lisp (and other languages) and if we find that there is a special
> >operator that can be shown not to be implementable in Common lisp
> >(i.e. by macroexpansion into either standard special-forms or
> >function-forms) then we've found a hole in the language, and it should
> >be plugged, at least intellectually.
>
> The issue is _not_ whether HANDLER-BIND can be implemented as a macro in
> CLISP (it can be and there is already commented-out code for that from
> times immemorial).
Yeah; we've got quite a few commented-out versions sitting around
as well. Who says computing isn't an art? :-)
> The issue is that this macro implementation is suboptimal, so it is
> suitable for code analysis tools (like XREF) but _NOT_ code generation
> tools (lie ITERATE and SERIES).
It would be nice if someday we can provide a code walker and a form
to a CL compiler and have it both analyze _and_ compile the form
using the walker.
I don't know of any way to do this with the currently opaque
environments we have now...
--
Duane Rettig ·····@franz.com Franz Inc. http://www.franz.com/
555 12th St., Suite 1450 http://www.555citycenter.com/
Oakland, Ca. 94607 Phone: (510) 452-2000; Fax: (510) 452-0182
Duane Rettig wrote:
>
> It would be nice if someday we can provide a code walker and a form
> to a CL compiler and have it both analyze _and_ compile the form
> using the walker.
But code walkers and compilers do a very different job: While code walkers
only need to be able to understand which parts of the code are forms
vs. function names vs. data, a compiler needs to care about all details.
It is for this reason also that the environment object passed to a
macroexpander contains only information about macros (and function
bindings because they can shadow macros) and symbol-macros (and
variable bindings and SPECIAL declarations of variables, because they
can shadow symbol-macros).
Whereas the notion of environment in a compiler also contains
- information about BLOCK tags,
- information about TAGBODY tags,
- information about all kinds of declarations (TYPE, OPTIMIZE, ...),
- details about variable bindings,
- details about function bindings.
Any attempt to unify the two kinds of environments is like confusing
apples with pears.
Bruno
Bruno Haible <·····@clisp.org> writes:
> Duane Rettig wrote:
> >
> > It would be nice if someday we can provide a code walker and a form
> > to a CL compiler and have it both analyze _and_ compile the form
> > using the walker.
>
> But code walkers and compilers do a very different job: While code walkers
> only need to be able to understand which parts of the code are forms
> vs. function names vs. data, a compiler needs to care about all details.
Code walkers do more than that; they can also rewrite code, either
to be thrown away at the end of the analysis or as a new piece of code
to be supplied to the compiler and/or interpreter.
> It is for this reason also that the environment object passed to a
> macroexpander contains only information about macros (and function
> bindings because they can shadow macros) and symbol-macros (and
> variable bindings and SPECIAL declarations of variables, because they
> can shadow symbol-macros).
The definition of macroexpand/macroexpand-1 give these as minimal
compilation requirements (the paragraph agrees with 3.2.2.2, the formal
specification of Minimal Compilation) But there are no maximal
compilation requirements. Compilers are free to ignore most
declarations, or they are free to make maximal use of them for their own
or their users' benefits.
> Whereas the notion of environment in a compiler also contains
> - information about BLOCK tags,
> - information about TAGBODY tags,
> - information about all kinds of declarations (TYPE, OPTIMIZE, ...),
> - details about variable bindings,
> - details about function bindings.
>
> Any attempt to unify the two kinds of environments is like confusing
> apples with pears.
Why do you believe this? Do you find anything in the spec that
prohibits it? Or is it just the case that you find my attempt to
cross-breed apples and pears to discover a delicious new fruit
to be an abomination?
Regardless of what a compiler might or might not do with declarations,
etc.: Given an implementation that provides some kind of CLtL2
Environments Access functionality, and given a file foo.cl, with
these contents (with variable-information provided in whatever
package is required by the implementation):
(defmacro foo (hyg &environment env)
(multiple-value-bind (kind ign decl)
(sys::variable-information hyg env)
(declare (ignore ign))
(format t "hyg is ~s ·@[with decl ~s~]~%"
kind decl))
(multiple-value-bind (kind ign decl)
(sys::variable-information 'non-hyg env)
(declare (ignore ign))
(format t "non-hyg is ~s ·@[with decl ~s~]~%"
kind decl))
`(bar ,hyg non-hyg))
(defun bas (x y)
(let ((hyg x)
(non-hyg y))
(declare (type (single-float 1.0 10.0) hyg)
(fixnum non-hyg))
(foo hyg)))
What would you expect to be shown in the output when the file
is compiled?
I ran this on Allegro CL, both in 6.2 (where variable-information
had not yet been exported) and in 7.0:
6.2:
CL-USER(1): :cf foo
;;; Compiling file foo.cl
hyg is NIL
non-hyg is NIL
;;; Writing fasl file foo.fasl
[...]
7.0:
CL-USER(1): :cf foo
;;; Compiling file foo.cl
hyg is :LEXICAL with decl ((TYPE (SINGLE-FLOAT 1.0 10.0)))
non-hyg is :LEXICAL with decl ((TYPE (INTEGER -536870912 536870911)))
;;; Writing fasl file foo.fasl
[ ... ]
Both give perfectly valid outputs. However, I don't know about
you, but as a user, I definitely prefer the information
available in 7.0 over that available in 6.2...
--
Duane Rettig ·····@franz.com Franz Inc. http://www.franz.com/
555 12th St., Suite 1450 http://www.555citycenter.com/
Oakland, Ca. 94607 Phone: (510) 452-2000; Fax: (510) 452-0182
Bruno Haible <·····@clisp.org> writes:
> In clisp:
> [1]> (special-operator-p 'ext:compiler-let)
> T
> In Allegro CL:
> CL-USER(1): (special-operator-p 'excl::compiler-let)
> #<special operator COMPILER-LET @ #x71042872>
That's IMHO TRT, and why in clisp
(special-operator-p 'sys::%handler-bind)
must also yield true.
Regards,
Jorg Hohle
Telekom/T-Systems Technology Center
From: Christopher C. Stacy
Subject: Re: what are allowed special forms? (CLHS clarification)
Date:
Message-ID: <u3bx5421o.fsf@news.dtpq.com>
Pascal Costanza <··@p-cos.net> writes:
> Joerg Hoehle wrote:
>
> > Currently, I'd resume my experience as follows: "the Iterate package
> > depends on a code walker. Since code walking may stumble on arbitrary
> > implementation-defined special operators as the result of expanding
> > portable code, it cannot be expected to work portably. Therefore it
> > can be viewed as poor design. Similarly, any purportedly portable
> > package which depends on a code-walker can be viewed as poor design,
> > if not otherwise restricted to particular usages or situations. YMMV"
> > What do you think?
>
> I think that's too harsh. I think this is a vendor's issue, not one of
> a library provider. If CL vendors want to make sure that their
> respective CL implementations work with such libraries, they should
> not include additional special operators.
Or else they should provide a generalized code walker,
based on a (eg. community) standard.
Joerg Hoehle <······@users.sourceforge.net> writes:
> [...]
> To which Steven M. Haflich replied
> "But that would mean that no code could depend upon the success of a code
> walker. That seems a poor design, similar to C++. The reason Lisp has
> code walkers is so tools that were not designed to interoperate can still
> do so. The Java VM tries to accomplish this with load/run time mechanisms.
> Lisp accomplishes this at compile/load/run time. The purposes and tradeoffs
> are different."
> [...]
> What do you think?
I think that a code walker could know about COMMON-LISP macros, in
addition to COMMON-LISP special operators.
Of course, it would be simplier if everything coalesced to the
specified special operators, but this is not realistic to expect
it. (^)Therefore the portable code walkers must know more about Common
Lisp and walk correctly the COMMON-LISP macros. They can be adaptive,
expanding all macros and only if they find an unknown special
operator, they would backtrack and interpret the COMMON-LISP macro
following its specification.
Now, there remains the problem of walking macros and special operators
defined in implementation specific packages. Either:
- let's write a CLRFI to specify a portable implementation
specific code walker, or
- let's write a CLRFI to specify the needed introspective
functions for special operators to be able to write a portable
code walker whatever the set of special operators. We'd need to
know for any special operator what arguments are expected, of
what kind (evaluated data, non-evaluated data, code (always
evaluated), conditionally evaluated code, etc).
- we could drop the requirement of a portable code walker for
implementation specific code, meaning that the code walker would
have to know about the implementation too, as above(^).
--
__Pascal Bourguignon__ http://www.informatimago.com/
There is no worse tyranny than to force a man to pay for what he does not
want merely because you think it would be good for him. -- Robert Heinlein
From: Duane Rettig
Subject: Re: what are allowed special forms? (CLHS clarification)
Date:
Message-ID: <47jmhxg1t.fsf@franz.com>
Pascal Bourguignon <····@mouse-potato.com> writes:
> Joerg Hoehle <······@users.sourceforge.net> writes:
> > [...]
> > To which Steven M. Haflich replied
> > "But that would mean that no code could depend upon the success of a code
> > walker. That seems a poor design, similar to C++. The reason Lisp has
> > code walkers is so tools that were not designed to interoperate can still
> > do so. The Java VM tries to accomplish this with load/run time mechanisms.
> > Lisp accomplishes this at compile/load/run time. The purposes and tradeoffs
> > are different."
> > [...]
> > What do you think?
>
> I think that a code walker could know about COMMON-LISP macros, in
> addition to COMMON-LISP special operators.
It's not necessary. A walker is going to call macroexpand on a
macro form; it need not parse the form.
> Of course, it would be simplier if everything coalesced to the
> specified special operators, but this is not realistic to expect
> it. (^)Therefore the portable code walkers must know more about Common
> Lisp and walk correctly the COMMON-LISP macros. They can be adaptive,
> expanding all macros and only if they find an unknown special
> operator, they would backtrack and interpret the COMMON-LISP macro
> following its specification.
How to walk a macro:
(macroexpand macro-form environment)
> Now, there remains the problem of walking macros and special operators
> defined in implementation specific packages. Either:
>
> - let's write a CLRFI to specify a portable implementation
> specific code walker, or
>
> - let's write a CLRFI to specify the needed introspective
> functions for special operators to be able to write a portable
> code walker whatever the set of special operators. We'd need to
> know for any special operator what arguments are expected, of
> what kind (evaluated data, non-evaluated data, code (always
> evaluated), conditionally evaluated code, etc).
>
> - we could drop the requirement of a portable code walker for
> implementation specific code, meaning that the code walker would
> have to know about the implementation too, as above(^).
Really, you are making this much more complex than it need be.
--
Duane Rettig ·····@franz.com Franz Inc. http://www.franz.com/
555 12th St., Suite 1450 http://www.555citycenter.com/
Oakland, Ca. 94607 Phone: (510) 452-2000; Fax: (510) 452-0182