Trying to understand the (absence of) specification given on the book,
I came across a very strange feature that is shared by several
implementations of Common Lisp:
(defparameter z 12)
(defstruct (foo (:constructor mkfoo (&optional (z 666))))
(y z)
z)
(mkfoo) -> #S(FOO Y 666 Z 666)
This, of course, is not intuitive... unless you know the
implementation, as is often the case with Common Lisp. This shows up
more clearly in the following example (which doesn't work at all in
several implementations):
(defstruct (bar (:constructor mkbar (&optional (z 666) w)))
x
(y (+ x z)) ; <-
z
(w y)) ; <- according to the lambda-list, w should be
; bound before y (if you know that y will be
; rejected in the &aux parameter list)
Putting aside implementation issues, it would be
desirable (IMO) to preserve as much as possible the fact that the
second slot's default init is evaluated in an environment where the
first slot is bound, and so on, giving to MKBAR the following
lambda-list:
(&dumb (x nil) &dumb (y z) &optional (z 666) (w y))
where &dumb binds a parameter in the manner of &aux, without consuming
an argument.
If such a semantics is not desirable [it has problems, too], then
another semantics ought to be specified. It is funny to see [as a
clarification] at the end of page 472, that the default slot init
forms should be considered in their lexical environment (in BOA lambda
lambda-lists), when -- given the let*-style binding in lambda-lists --
this environment is never specified !
Now, coming back to implementation issues, we must remark that &dumb
parameters (or anything that would achieve a similar effect) are not
so difficult to implement, and in particular are suitable for static
binding (which is legal when optimizations are turned on, and keyword
bindings are not involved).
Further remarks: CLTL2 states explicitly that extra (non-slot)
parameters are allowed in BOA lambda-lists, and (although every
example seem to indicate slot ordering), doesn't seem to forbid the
reordering of slots names within the lambda-list. The disturbing
effect on _how_ and _when_ the default values are evaluated is left
unspecified.
It seems that complex lambda-lists and macros form a convenient set of
tools for the _user_ to build extensions, or syntactic convenient
features, etc. But, from a language point of view, they are no
substitute for clean semantics, and implementations often do not
resist beyond simplistic examples. Although sufficient for (I believe)
100% of the CL programmers, the current specifications for structures
rely to much 'the' implicit implementation, and lack the definition of
proper namespace for slots.
This is very disappointing for such a basic feature of the language:
what should we discover in CLOS on top of that (or, as some say, as
part of the language) ???
Vincent
In article <·····················@marcellus.parc.xerox.com> ········@parc.xerox.com (Vincent Delacour) writes:
>Trying to understand the (absence of) specification given on the book,
>I came across a very strange feature that is shared by several
>implementations of Common Lisp:
>
> (defparameter z 12)
>
> (defstruct (foo (:constructor mkfoo (&optional (z 666))))
> (y z)
> z)
>
> (mkfoo) -> #S(FOO Y 666 Z 666)
>
>This, of course, is not intuitive... unless you know the
>implementation, as is often the case with Common Lisp.
Unfortunately, CLTL's description of BOA constructors neglects to say
anything explicit about slots that aren't mentioned in the constructor
lambda list. The intent was that they be initialized in the normal way
(since providing an &AUX parameter with the name of a slot and no default
is described as the way to override this normal initialization).
The behavior you showed above is due to the the simplest implementation of
BOA constructors, which is that it effectively defines the following
function:
(defun mkfoo (&optional (z 666))
(make-foo :z z))
However, this equivalence is not officially specified anywhere. X3J13 has
addressed a related issue regarding the default keyword constructor; we
decided that these *shouldn't* actually bind variables with the names of
the slots, i.e. the effective definition would be something like:
(defun make-foo (&key ((:y #:g1) z) ((:z #:g2)))
(internal-structure-creator 'foo #:g1 #:g2))
In fact, it takes some pretty contorted coding to generate a BOA
constructor that doesn't evaluate Y's initform in an environment where Z is
bound; here's what I came up with:
(defun mkfoo (&rest args)
(flet ((get-slots (&optional (z 666))
(list :z z)))
(apply #'make-foo (apply #'get-slots args))))
I actually think this latter interpretation is closer to what was intended.
Initforms are supposed to be evaluated in the lexical environment where the
DEFSTRUCT appears, and the dynamic environment of the constructor *call*,
not the dynamic environment of the body of the constructor.
> This shows up
>more clearly in the following example (which doesn't work at all in
>several implementations):
>
> (defstruct (bar (:constructor mkbar (&optional (z 666) w)))
> x
> (y (+ x z)) ; <-
> z
> (w y)) ; <- according to the lambda-list, w should be
> ; bound before y (if you know that y will be
> ; rejected in the &aux parameter list)
There is nothing in the DEFSTRUCT description that suggests that other
slots may be referenced as variables in initforms. The description of BOA
constructors only says that the *parameter* variables can be referenced in
initializations for later parameters. And assuming an implementation that
defines BOA constructors in either of the ways I described above, and also
conforms to the X3J13 decision regarding the default constructor, the slots
won't be accessible as variables.
>Putting aside implementation issues, it would be
>desirable (IMO) to preserve as much as possible the fact that the
>second slot's default init is evaluated in an environment where the
>first slot is bound, and so on, giving to MKBAR the following
>lambda-list:
>
> (&dumb (x nil) &dumb (y z) &optional (z 666) (w y))
>
>where &dumb binds a parameter in the manner of &aux, without consuming
>an argument.
What in the description of DEFSTRUCT suggests that slots can *ever* be
referenced as if they were variables? I can understand why you might
desire it as a change to the language, but not why you would expect it in
current implementations.
>If such a semantics is not desirable [it has problems, too], then
>another semantics ought to be specified. It is funny to see [as a
>clarification] at the end of page 472, that the default slot init
>forms should be considered in their lexical environment (in BOA lambda
>lambda-lists), when -- given the let*-style binding in lambda-lists --
>this environment is never specified !
I think the clarification on p.472 is perfectly clear. It says that the
slot default-init form is evaluated in the lexical environment where the
DEFSTRUCT appears, and also that the initialization forms in BOA lambda
lists are evaluated in the lexical environment of the DEFSTRUCT (the
detailed description of BOA lambda lists then expands on this to specify
that each parameter augments the environment to produce the environment for
the following parameter initialization forms).
The primary intent of the clarification was to make it clear that the
following is valid:
(let ((y-init-1 1)
(y-init-2 2))
(defstruct (foo (:constructor mkfoo (&optional (y y-init-1))))
(y y-init-2)))
(mkfoo) -> #S(FOO :Y 1)
(make-foo) -> #S(FOO :Y 2)
>Now, coming back to implementation issues, we must remark that &dumb
>parameters (or anything that would achieve a similar effect) are not
>so difficult to implement, and in particular are suitable for static
>binding (which is legal when optimizations are turned on, and keyword
>bindings are not involved).
No argument there. But that doesn't mean that this feature is needed.
>Further remarks: CLTL2 states explicitly that extra (non-slot)
>parameters are allowed in BOA lambda-lists, and (although every
>example seem to indicate slot ordering), doesn't seem to forbid the
>reordering of slots names within the lambda-list. The disturbing
>effect on _how_ and _when_ the default values are evaluated is left
>unspecified.
The second paragraph of 19.6 says that "&OPTIONAL, &AUX, and &REST ... work
in the way you might expect". We all interpreted this to mean that they
work just like in normal lambda lists, except for the differences described
in the following paragraphs. Since the order of parameters is significant
in normal lambda lists, and the rest of the section never says that this is
one of the exceptions, order is clearly significant in BOA lambda lists.
>It seems that complex lambda-lists and macros form a convenient set of
>tools for the _user_ to build extensions, or syntactic convenient
>features, etc. But, from a language point of view, they are no
>substitute for clean semantics, and implementations often do not
>resist beyond simplistic examples. Although sufficient for (I believe)
>100% of the CL programmers, the current specifications for structures
>rely to much 'the' implicit implementation, and lack the definition of
>proper namespace for slots.
It's technically infeasible to define a language as large and complex as
Common Lisp with completely clean semantics. X3J13 has been working since
1986 to clean up as much as we could. Unfortunately, we could only fix the
bugs we actually discovered; had you sent your message appeared a year ago
we could have dealt with it in our draft. As of now we're getting ready to
put the standard out for public review, so we'll probably have to wait
until we're dealing with review comments in order to deal with this.
>This is very disappointing for such a basic feature of the language:
>what should we discover in CLOS on top of that (or, as some say, as
>part of the language) ???
DEFSTRUCT is a basic feature, but it's also very old and the CL designers
basically adopted it, along with much of its description, from previous
dialects. Newer features such as CLOS have had much more careful design.
I'm not saying that there are no ambiguities in the new features, but the
designers had the benefit of the experience with the original CL
specification, so were better prepared to specify things more precisely.
And the X3J13 editors have done quite a bit of work to recast the informal
language of CLTL more precisely in the ANSI draft spec. Again, it's not
perfect, but I think you'll be pleased.
--
Barry Margolin, Thinking Machines Corp.
······@think.com
{uunet,harvard}!think!barmar
One correction to my previous post:
I repeatedly said that CLTL2 says that slot initforms are evaluated in the
dynamic environment of the constructor call, but I actually find that it
never actually says this. What it says is that the initform is evaluated
each time it is needed (i.e. each time the constructor is called and no
value for the slot is specified in the call). This was intended to make it
clear that the initform is *not* evaluated when the DEFSTRUCT is executed.
I think it should be obvious that it should be evaluated in the dynamic
environment of the constructor call, but as I pointed out it's easy to
screw this up in the implementation; Lucid CL 4.0.1 gets it wrong, but
Symbolics Genera 8.0.1 does it right.
--
Barry Margolin, Thinking Machines Corp.
······@think.com
{uunet,harvard}!think!barmar
A clarification to my previous post. X3J13 *has* specified the dynamic
environment in which slot initforms are evaluated. The following cleanup
was approved in November, 1989 (it isn't reflected in CLTL2, because the
book only has cleanups through June, 1989):
Proposal (DEFSTRUCT-CONSTRUCTOR-SLOT-VARIABLES:NOT-BOUND):
Clarify that the symbols which name slots must *not* be bound as
lambda variables by the keyword constructor function. The slot
default init forms are evaluated in the lexical environment
in which the DEFSTRUCT form itself appears and the dynamic environment
in which the call to the constructor function appears.
As I said in my previous message, when a BOA constructor is used, any slots
not specified in the BOA lambda list are initialized in the normal way, in
which case the above correction applies. Thus, these slot initforms are
evaluated in the dynamic environment in which the call to the BOA
constructor appears, not in the dynamic environment of the body of the BOA
constructor.
I intend to report Lucid's behavior as a bug. Symbolics Genera gets it
right.
--
Barry Margolin, Thinking Machines Corp.
······@think.com
{uunet,harvard}!think!barmar
In article <············@early-bird.think.com> ······@think.com (Barry Margolin) writes:
> Thus, these slot initforms are
>evaluated in the dynamic environment in which the call to the BOA
>constructor appears, not in the dynamic environment of the body of the BOA
>constructor.
I don't understand why the dynamic environment of the body
of the BOA constructor would be significant. What would it
add to the dynamic environment of the call? Also, can you
give an example of what Ludid does wrong?
-- jeff
In article <····@skye.ed.ac.uk> ····@aiai.ed.ac.uk (Jeff Dalton) writes:
> In article <············@early-bird.think.com> ······@think.com (Barry Margolin) writes:
> > Thus, these slot initforms are
> >evaluated in the dynamic environment in which the call to the BOA
> >constructor appears, not in the dynamic environment of the body of the BOA
> >constructor.
>
> I don't understand why the dynamic environment of the body
> of the BOA constructor would be significant. What would it
> add to the dynamic environment of the call? Also, can you
The semantics of variables in CL are such that special variables may
be bound as parameters.
> give an example of what Ludid does wrong?
>
> -- jeff
>
Vincent
In article <·····················@marcellus.parc.xerox.com> ········@marcellus.parc.xerox.com (Vincent Delacour) writes:
>In article <····@skye.ed.ac.uk> ····@aiai.ed.ac.uk (Jeff Dalton) writes:
>
>> In article <············@early-bird.think.com> ······@think.com (Barry Margolin) writes:
>> > Thus, these slot initforms are
>> >evaluated in the dynamic environment in which the call to the BOA
>> >constructor appears, not in the dynamic environment of the body of the BOA
>> >constructor.
>>
>> I don't understand why the dynamic environment of the body
>> of the BOA constructor would be significant. What would it
>> add to the dynamic environment of the call? Also, can you
>
>The semantics of variables in CL are such that special variables may
>be bound as parameters.
I know that. But why would it matter? It's important to say exactly
the right thing in the standard, sure, but the cases where a user
would actually notice a difference must be fairly obscure.
Maybe it would all be clear to me if I had an example, but as it
is I'm still not sure exactly what you're talking about.
>> give an example of what Ludid does wrong?
>>
>> -- jeff
>>
>
> Vincent
In article <············@early-bird.think.com> ······@think.com (Barry Margolin) writes:
Thank you very much for your clarification, as alway perfectly clear
and sensible.
> A clarification to my previous post. X3J13 *has* specified the dynamic
> environment in which slot initforms are evaluated. The following cleanup
> was approved in November, 1989 (it isn't reflected in CLTL2, because the
> book only has cleanups through June, 1989):
>
> Proposal (DEFSTRUCT-CONSTRUCTOR-SLOT-VARIABLES:NOT-BOUND):
>
> Clarify that the symbols which name slots must *not* be bound as
> lambda variables by the keyword constructor function. The slot
> default init forms are evaluated in the lexical environment
> in which the DEFSTRUCT form itself appears and the dynamic environment
> in which the call to the constructor function appears.
>
This puts it clear and clean enough. I understand that the book (and
Steele says so) reflects only an intermediate step between CLTL-84 and
ANSI CL. As a clarification, I was not asking for a modification of
the language, but rather asking for such a clarification, which I was
not able to find in the book.
> As I said in my previous message, when a BOA constructor is used, any slots
> not specified in the BOA lambda list are initialized in the normal way, in
> which case the above correction applies. Thus, these slot initforms are
> evaluated in the dynamic environment in which the call to the BOA
> constructor appears, not in the dynamic environment of the body of the BOA
> constructor.
>
> I intend to report Lucid's behavior as a bug. Symbolics Genera gets it
> right.
> --
> Barry Margolin, Thinking Machines Corp.
>
> ······@think.com
> {uunet,harvard}!think!barmar
>
Allegro has the bug too, and KCL too. (how about others, someone?)
However, now that this point is clarified, I persist in my opinion
that macros and lambda-lists are not a panacea for implementing a
language such as Common Lisp. The inconvenient of implementing basic
features such as DEFSTRUCT as macros is that the ground level is not
well defined. DEFSTRUCT is quite a false macro anyway, because CL
provides no way to implement it (does it?). It is in that respect far
different from, say, LOOP.
Although some steps seem to have been taken to help make CL a language
for applications delivery (see for example new things on inlining and
redefinition of structures), it is worth noting that by defining forms
such as DEFUN of DEFSTRUCT to be macros makes the static analysis of
programs unnecessarily harder, because they can expand in arbitrary
executable code. This load-time code must then be either pre-executed
by the analyzing program, and the resulting state inspected [much a
pain, probably impracticable], or the DEFUN forms have to be treated
as special forms. The question is then, would it be possible to devise
a language that would be almost exactly like CL, but in which defining
forms would be special forms, and, say, would accept almost all of the
existing CL programs (with only very small module-related changes) ?
I suppose that the answer would be no, but that if we restrict the set
of programs to be accepted to those that conform to the module-related
guidelines in CLTL, the answer would be yes.
Vincent
In article <·····················@marcellus.parc.xerox.com> ········@marcellus.parc.xerox.com (Vincent Delacour) writes:
>Allegro has the bug too, and KCL too. (how about others, someone?)
Well, we're not a user of either of them, so it wouldn't be appropriate for
me to send a bug report. If you're a Franz customer, I suggest you do so.
I don't know if KCL is officially maintained by anyone.
>However, now that this point is clarified, I persist in my opinion
>that macros and lambda-lists are not a panacea for implementing a
>language such as Common Lisp. The inconvenient of implementing basic
>features such as DEFSTRUCT as macros is that the ground level is not
>well defined. DEFSTRUCT is quite a false macro anyway, because CL
>provides no way to implement it (does it?). It is in that respect far
>different from, say, LOOP.
No one has ever claimed that macros are a panacea. They're just a
reasonable way to allow application developers to extend the language.
The only aspect of CL's DEFSTRUCT that isn't possible to emulate in a
portable implementation is the underlying type of the structure object when
the :TYPE option isn't specified.
The main reason for special operators is that they must interact with the
evaluator/compiler in some deep way, such as by accessing and/or altering
the environment directly. DEFSTRUCT does not need to do this; it can do
everything by expanding into definitions of functions (which may need to
call implementation-dependent functions, but that's OK because an analysis
program should understand function calls even if it doesn't recognize the
function) and declarations. Any interaction it needs to do with the
compiler can be done using (EVAL-WHEN (COMPILE) ...); again, the body of
this might contain calls to implementation-dependent functions.
Compare this with SETQ. It needs to know whether a variable is lexical or
dynamic, and if it's lexical it needs to alter the environment somehow.
Since CL doesn't have first-class environments, SETQ must be implemented
specially.
>Although some steps seem to have been taken to help make CL a language
>for applications delivery (see for example new things on inlining and
>redefinition of structures), it is worth noting that by defining forms
>such as DEFUN of DEFSTRUCT to be macros makes the static analysis of
>programs unnecessarily harder, because they can expand in arbitrary
>executable code. This load-time code must then be either pre-executed
>by the analyzing program, and the resulting state inspected [much a
>pain, probably impracticable], or the DEFUN forms have to be treated
>as special forms. The question is then, would it be possible to devise
>a language that would be almost exactly like CL, but in which defining
>forms would be special forms, and, say, would accept almost all of the
>existing CL programs (with only very small module-related changes) ?
Requiring defining forms to be implemented as special forms would not make
program analysis any easier. Currently, a program-analysis program has its
choice of recognizing DEFUN specially or macroexpanding it, depending on
its needs. Turning these macros into special forms forces *all* such
applications to special-case them. A program that special-cases DEFUN will
be the same in both Lisp dialects, but a CL program that doesn't care about
DEFUN would have to be changed in order to run in the proposed dialect.
In other words, program-analysis programs don't *have* to expand all
macros.
>I suppose that the answer would be no, but that if we restrict the set
>of programs to be accepted to those that conform to the module-related
>guidelines in CLTL, the answer would be yes.
I think the only programs that would not work properly in such a dialect
are program-analysis programs that only special-case the Common Lisp
special forms, and expect everything else to be either a function or macro
invocation.
--
Barry Margolin, Thinking Machines Corp.
······@think.com
{uunet,harvard}!think!barmar
In article <············@early-bird.think.com> ······@think.com (Barry Margolin) writes:
The only aspect of CL's DEFSTRUCT that isn't possible to emulate in a
portable implementation is the underlying type of the structure object when
the :TYPE option isn't specified.
Yes, it is exactly in that respect that defstruct is a false macro.
Agreed 100% that the syntactic sugar belongs to the macro world. But
the basic facts when dealing with types are 1- the identity of those
said types, type-checking and objects creation, and 2- access paths
through data. For keeping track of these activities in a portable way
we have no other solution than to emulate defstruct as a whole, rather
than accept the expansion product. The point is that the rest of the
language doesn't make this state of affairs necessary.
[To agree with your following argument, I would say that the
'ground level' that I'm asking for may well be implemented as macros,
and that anyone should feel free to either treat it as such , or as a
special case.]
Anyway, all this is no really big deal -- only a matter of a
few hundred lines, and I'm now satisfied with the clarifications I
got.
Sorry if I have confused some by raising points on both what is
written and what is not written in the book, and on what is and what
could be in the language.
Vincent
In article <·····················@marcellus.parc.xerox.com> ········@marcellus.parc.xerox.com (Vincent Delacour) writes:
>However, now that this point is clarified, I persist in my opinion
>that macros and lambda-lists are not a panacea for implementing a
>language such as Common Lisp. The inconvenient of implementing basic
>features such as DEFSTRUCT as macros is that the ground level is not
>well defined. DEFSTRUCT is quite a false macro anyway, because CL
>provides no way to implement it (does it?). It is in that respect far
>different from, say, LOOP.
I do not agree that DEFSTRUCT is a false macro. All that's needed
to implement it are a small number of implementation-dependent functions
(potentially only one). I think any accusations of falseness should
be reserved for macros that require implementation-dependent special
forms. But this is, presumably, a minor point.
Of course, even if DEFSTRUCT were not specified to be a macro, it
could still be implemented by a macro-like transformation. I don't
see why that kind of implementation would be "inconvenient".
On the other hand, I would agree that it would be better to have a
well-defined, visible "ground level", if by that we mean the functions
needed by DEFSTRUCT rather than, say, all of DEFSTRUCT. Indeed,
CLOS (or its MOP) could provide such primitives for structure classes
as it does for standard classes.
>Although some steps seem to have been taken to help make CL a language
>for applications delivery (see for example new things on inlining and
>redefinition of structures),
[Can you tell me what's changed about redefinition? I haven't
kept track of this.]
> it is worth noting that by defining forms
>such as DEFUN of DEFSTRUCT to be macros makes the static analysis of
>programs unnecessarily harder, because they can expand in arbitrary
>executable code. This load-time code must then be either pre-executed
>by the analyzing program, and the resulting state inspected [much a
>pain, probably impracticable], or the DEFUN forms have to be treated
>as special forms. The question is then, would it be possible to devise
>a language that would be almost exactly like CL, but in which defining
>forms would be special forms, and, say, would accept almost all of the
>existing CL programs (with only very small module-related changes) ?
But why should that _actually be_ special forms, if the analysis
programs can treat them as special forms even if they're macros?
Moreover, if it turns out that some analysis programs can analyse
the expansion of DEFSTRUCT, treating it as a macro and not as a
special form, what's wrong with that?
>I suppose that the answer would be no, but that if we restrict the set
>of programs to be accepted to those that conform to the module-related
>guidelines in CLTL, the answer would be yes.
What are the module-related guidelines? I'm not sure what you're
referring to.
-- jeff
In article <·····················@marcellus.parc.xerox.com> ········@marcellus.parc.xerox.com (Vincent Delacour) writes:
Trying to understand the (absence of) specification given on the book,
I came across a very strange feature that is shared by several
implementations of Common Lisp:
(defparameter z 12)
(defstruct (foo (:constructor mkfoo (&optional (z 666))))
(y z)
z)
(mkfoo) -> #S(FOO Y 666 Z 666)
You are right, it makes no sense based on the specification on page
474 of CLTL II:
"If the default initialization form is used, it is evaluated at
construction time, but in the lexical environment of the DEFSTRUCT
form in which it appeared."
This follows the lexical semantics of the rest of CommonLisp.
There is an unclear point in CLTL II, however. The next paragraph
states,
"It is as if the initialization forms were used as the _init_ forms
for the keyword parameters of the constructor function."
This might imply that the init forms use the lexical environment of
the constructor lambda list. However, I cannot believe that is the
intention of the designers. This should probably be clarified,
perhaps by referring to initfunctions as CLOS does.
If you want default slot initialization forms to depend on the initial
value of other slots, the possibility still (partially) exists by
explicitly specifying the init forms for constructor keyword
arguments rather than in the slot descriptions themselves.
-- Harley Davis
--
------------------------------------------------------------------------------
nom: Harley Davis ILOG S.A.
net: ·····@ilog.fr 2 Avenue Gallie'ni, BP 85
tel: (33 1) 46 63 66 66 94253 Gentilly Cedex, France