From: Jeff Dalton
Subject: A pitfalls list
Date:
Message-ID: <7062@skye.ed.ac.uk>
I've been keeping a list of Common Lisp "pitfalls" -- things that
I sometimes forget, that have caught students and other people
in Edinburgh, or that I think might catch people in unexpected
ways.
There's some overlap with the FAQ pitfalls list, but since I wanted
to include items what were not "frequently asked" and wanted to allow
"checklist" entries without much in the way of explanation, I thought
it made more sense to keep a spearate list.
So, in the hope that it might be useful, here it is.
I would appreciate any comments or suggested additions.
(I'm a bit worried about the claim that names declared by
DEFCONSTANT can't be bound even as lexical variables. That
sure seems to be what CLtL II says, but is it still that way
in the X3J13 draft?)
----------------------------------------------------------------------
This is a list of "pitfalls": somewhat counterintuitive things that
programmers may not expect Common Lisp to do and that tend to be
revealed only by a careful reading of CLtL.
A different, but overlapping, list of pitfalls can be found in the
"Frequently Asked Questions" list of Comp.lang.lisp.
Functions.
* The result of many non-destructive functions such as REMOVE
and UNION can share structure with an argument.
* SORT is (usually) destructive.
So, for instance, (SORT (REMOVE ...) ...) may not be safe.
* Destructive functions that you think would modify CDRs might
modify CARs instead. (Eg, NREVERSE.)
* Array elements might not be initialized to NIL. Eg,
(make-array 10) => #(0 0 0 0 0 0 0 0 0 0)
* READ-FROM-STRING has some optional arguments before the
keyword parameters. If you want to supply some keyword
arguments, you have to give all of the optional ones too.
* (flet ((f ...)) (eq #'f #'f)) can return false.
* The function LIST-LENGTH is not a faster, list-specific version
of the sequence function LENGTH. It is list-specific, but it's
slower than LENGTH because it can handle circular lists.
Iteration vs closures.
* DO and DO* update the iteration variables by assignment; DOLIST and
DOTIMES are allowed to use assignment (rather than a new binding).
(All CLtL says of DOLIST and DOTIMES is that the variable "is
bound" which has been taken as _not_ implying that there will be
separate bindings for each iteration.)
Consequently, if you make closures over an iteration variable
in separate iterations they may nonetheless be closures over
the same variable and hence will all refer to the same value
-- whatever value the variable was given last. Eg,
(let ((fns '()))
(do ((x '(1 2) (cdr x)))
((null x))
(push #'(lambda () x)
fns))
(mapcar #'funcall (reverse fns)))
returns (nil nil), not (1 2), not even (2 2).
Limits.
* The array-total-size-limit may be as small as 1024.
* Such things as (apply #'append list-of-lists) to flatten a list
of lists may run afoul of call-arguments-limit.
Alternatives include:
(reduce #'append list-of-lists :from-end t)
(mapcan #'copy-list list-of-lists)
Definitions and declarations.
* (DEFVAR var init) assigns to the variable only if it does not
already have a value. So if you edit a DEFVAR in a file and
reload the file only to find that the value has not changed,
this is the reason. (Cf DEFPARAMETER.)
* DEFCONSTANT has several potentially unexpected properties:
- Once a name has been declared constant, it cannot be used a
the name of a local variable (lexical or special) or function
parameter. Really. See page 87 of CLtL II.
- A DEFCONSTANT cannot be re-evaluated (eg, by reloading the
file in which it appears) unless the new value is EQL to the
old one. Strictly speaking, even that may not be allowed.
(DEFCONSTANT is "like DEFPARAMETER" and hence does an
assignment, which is not allowed if the name has already
been declared constant by DEFCONSTANT.)
Note that this makes it difficult to use anything other
than numbers, symbols, and characters as constants.
- When compiling (DEFCONSTANT name form) in a file, the form
may be evaluated at compile-time, load-time, or both.
(You might think it would be evaluated at compile-time and
the _value_ used to obtain the object at load-time, but it
doesn't have to work that way.)
* You often have to declare the result type to get the most
efficient arithmetic. Eg,
(the fixnum (+ (the fixnum e1) (the fixnum e2)))
rather than
(+ (the fixnum e1) (the fixnum e2))
* Declaring the iteration variable of a DOTIMES to have type FIXNUM
does not guarantee that fixnum arithmetic will be used. That is,
implementations that use fixnum-specific arithmetic in the presence
of appropriate declaration may not think _this_ declaration is
sufficient. It may help to declare that the limit is also a
fixnum, or you may have to write out the loop as a DO and add
appropriate declarations for each operation involved.
----------------------------------------------------------------------
Jeff Dalton, JANET: ········@uk.ac.ed
AI Applications Institute, ARPA: ········@ed.ac.uk
Edinburgh University. UUCP: ...!ukc!ed.ac.uk!J.Dalton
In article <····@skye.ed.ac.uk> ····@aiai.ed.ac.uk (Jeff Dalton) writes:
>(I'm a bit worried about the claim that names declared by
>DEFCONSTANT can't be bound even as lexical variables. That
>sure seems to be what CLtL II says, but is it still that way
>in the X3J13 draft?)
Yes. I can see where you're coming from -- it's a violation of referential
transparency.
I suspect this was done to generalize on the restriction against binding T
and NIL, which has been in most dialects of Lisp for a long time. In
Common Lisp, T is just a variable that's predefined as a constant, and it
obeys the rules that DEFCONSTANT specifies (NIL is still special, though,
because of its equivalence to the empty list).
--
Barry Margolin
System Manager, Thinking Machines Corp.
······@think.com {uunet,harvard}!think!barmar
From: Jeff Dalton
Subject: Re: DEFCONSTANT (was Re: A pitfalls list)
Date:
Message-ID: <7081@skye.ed.ac.uk>
In article <············@early-bird.think.com> ······@think.com (Barry Margolin) writes:
>In article <····@skye.ed.ac.uk> ····@aiai.ed.ac.uk (Jeff Dalton) writes:
>>(I'm a bit worried about the claim that names declared by
>>DEFCONSTANT can't be bound even as lexical variables. That
>>sure seems to be what CLtL II says, but is it still that way
>>in the X3J13 draft?)
>
>Yes. I can see where you're coming from -- it's a violation of referential
>transparency.
Yes, and another thing I noticed last night (when reading up on
declarations in CLtL II) is that the new rules for type declarations
mean that:
(let ((x 1))
(declare (fixnum x))
(let ((x ...))
;; This X must be a fixnum too!
...))
I must have seen the change before and not really paid attention,
because it certainly looks wrong to me now. Moreover, I'm not sure
what the reasons for it can be. Changing type declarations to
apply to refs rather than bindings seems reasonable. But the
inner X isn't a reference to the same thing.
So I guess this is another pitfall.
Anyway, I can understand the reasoning behind disallowing bindings of
T and NIL. I think a generalization to some other constants might be
reasonable, but don't the rules on the LISP package already prevent
rebinding any "standard" constants? The full generalization to all
constants seems somewhat questionable (though I'm not sure it's
wrong, all things considered), and so I wondered whether X3J13
might have changed its mind.
-- jeff
In article <····@skye.ed.ac.uk> ····@aiai.ed.ac.uk (Jeff Dalton) writes:
>Yes, and another thing I noticed last night (when reading up on
>declarations in CLtL II) is that the new rules for type declarations
>mean that:
>
> (let ((x 1))
> (declare (fixnum x))
> (let ((x ...))
> ;; This X must be a fixnum too!
> ...))
I think this example in CLtL2 is bogus and unsupported by anything in
DECLARE-TYPE-FREE:LEXICAL. That issue covered type declarations that
aren't at the head of the body of a form that binds the variable. A
more correct example would be:
(let ((x 1))
(locally (declare (fixnum x))
(let ((x ...))
;; This X must be a fixnum too!
...)))
Because LOCALLY doesn't bind variables, its type declarations are not
associated with a specific variable binding, but to all bindings of the
specified variable within its scope. Also, the following is still correct:
(let ((x 'foo))
...
(setq x 1)
(locally (declare (fixnum x))
(let ((x ...))
;; This X must be a fixnum too!
...)))
X is initially bound to a symbol, but that is outside the scope of the
LOCALLY so it's OK. You just have to make sure that you assign a fixnum to
it before entering the LOCALLY (even if you never actually reference the
outer X inside the LOCALLY).
>Anyway, I can understand the reasoning behind disallowing bindings of
>T and NIL. I think a generalization to some other constants might be
>reasonable, but don't the rules on the LISP package already prevent
>rebinding any "standard" constants? The full generalization to all
>constants seems somewhat questionable (though I'm not sure it's
>wrong, all things considered), and so I wondered whether X3J13
>might have changed its mind.
I don't recall the issue ever being brought up to the full committee.
Someone who was on the Cleanup committee could possibly tell you whether
they ever discussed it.
--
Barry Margolin
System Manager, Thinking Machines Corp.
······@think.com {uunet,harvard}!think!barmar
From: Jeff Dalton
Subject: Re: DEFCONSTANT (was Re: A pitfalls list)
Date:
Message-ID: <7110@skye.ed.ac.uk>
In article <············@early-bird.think.com> ······@think.com (Barry Margolin) writes:
>In article <····@skye.ed.ac.uk> ····@aiai.ed.ac.uk (Jeff Dalton) writes:
>>Yes, and another thing I noticed last night (when reading up on
>>declarations in CLtL II) is that the new rules for type declarations
>>mean that:
>>
>> (let ((x 1))
>> (declare (fixnum x))
>> (let ((x ...))
>> ;; This X must be a fixnum too!
>> ...))
>
>I think this example in CLtL2 is bogus and unsupported by anything in
>DECLARE-TYPE-FREE:LEXICAL. That issue covered type declarations that
>aren't at the head of the body of a form that binds the variable.
I hope you're right, but CLtL II sys explicitly (page 219) that
type declarations refer to every reference no matter what binding
the reference refers to. And then on page 222 CLtL says that local
type declarations don't the proclaimed type of special variables,
noting that this is consistent with "the treatment of nested local
type declarations".
So, in addition to DECLARE-TYPE-FREE:LEXICAL, SPECIAL-TYPE-SHADOWING:
CLARIFY may be relevant.
(Unfortunately, I can't ftp them at present.)
>A more correct example would be:
>
>(let ((x 1))
> (locally (declare (fixnum x))
> (let ((x ...))
> ;; This X must be a fixnum too!
> ...)))
>
>Because LOCALLY doesn't bind variables, its type declarations are not
>associated with a specific variable binding, but to all bindings of the
>specified variable within its scope.
I think this is a reasonable thing for LOCALLY to do, even though
it does make LOCALLY ++magic. No wonder it's now a special form!
-- jeff
In article <····@skye.ed.ac.uk> ····@aiai.ed.ac.uk (Jeff Dalton) writes:
>I hope you're right, but CLtL II sys explicitly (page 219) that
>type declarations refer to every reference no matter what binding
>the reference refers to. And then on page 222 CLtL says that local
>type declarations don't the proclaimed type of special variables,
>noting that this is consistent with "the treatment of nested local
>type declarations".
>
>So, in addition to DECLARE-TYPE-FREE:LEXICAL, SPECIAL-TYPE-SHADOWING:
>CLARIFY may be relevant.
SPECIAL-TYPE-SHADOWING only concerns special variables, so it's not an
issue here. But after rereading DECLARE-TYPE-FREE:LEXICAL, now I'm not as
sure as I was yesterday. Since you (and perhaps other readers) can't FTP
it, I'll include it here:
Specify that a type declaration does not only "affect variable bindings";
rather, type declarations are legal in all declarations. The interpretation
of a type declaration is that, during the execution of any reference to the
declared variable within the scope of the declaration, it is an error for
the value of the declared variable not to be of the declared type; and
during the execution of any SETQ of the declared variable within the scope
of the declaration, it is an error for the newly assigned value of the
declared variable not to be of the declared type; and at the moment the
scope of the declaration is entered, it is an error for the value of the
declared variable not to be of the declared type.
In this proposal, a type declaration affects only variable references within
its scope, and the meaning of "free" and "variable-binding-associated" type
declarations can be described identically.
This proposal is equivalent to saying that the meaning of a type declaration
is equivalent to changing each reference to <var> within the scope of the
declaration to (THE <type> <var>), changing each expression assigned to the
variable within the scope of the declaration to (THE <type> <new-value>),
and executing (THE <type> <var>) at the moment the scope of the declaration
is entered.
Clarify that if nested type declarations refer to the same variable,
then the value of the variable must be a member of the intersection of
the declared types.
I believe the intent of this proposal was as an extension. Prior to this,
type declarations were only allowed at the beginning of the body of the
form that bound the variables whose types were being declared. The
extension is to allow them in other places, and specify what the scope of
those new declarations is. All the examples in DECLARE-TYPE-FREE are of
declarations that are not at the head of a form that binds the variable
(unlike the example on p.219).
Earlier on p.219, there's a bullet that says, "If the declaration applies
to a name binding, then the scope of the declaration also includes the
scope of the name binding." This is derived from
DECLARATION-SCOPE:NO-HOISTING, whose primary purpose was to remove initial
value forms from the scope of declarations. Formerly, in the form:
(let ((x (foo x)))
(declare (integer x))
...)
the INTEGER declaration applied both to the X being bound and the outer
X being referenced in the initialization form; now it only applies to the
one being bound.
But what of the previous bullet, which says, "always includes the body
forms"? This seems like it could imply the interpretation that Steele
made, which bothered you. What this specifically means isn't really made
clear in the proposal, but the Discussion section says, "Remember also that
new name bindings "shadow" (after a fashion) any higher level binding or
declarations." I think the use of the term "also" may have been a mistake
(by the author of DECLARATION-SCOPE:NO-HOISTING, not GLS); I believe the
intent was that when a declaration applies to a name being bound, that the
declaration's scope is precisely the same as the binding being introduced
by that form.
>>(let ((x 1))
>> (locally (declare (fixnum x))
>> (let ((x ...))
>> ;; This X must be a fixnum too!
>> ...)))
>>
>>Because LOCALLY doesn't bind variables, its type declarations are not
>>associated with a specific variable binding, but to all bindings of the
>>specified variable within its scope.
>
>I think this is a reasonable thing for LOCALLY to do, even though
>it does make LOCALLY ++magic. No wonder it's now a special form!
I take back my example. The intent of DECLARE-TYPE-FREE:LEXICAL was to
allow the type declaration in the LOCALLY, but it still only affects the
binding that is in effect at the time. Thus, the following is OK:
(let ((x 'foo))
...
(setq x 0)
(locally (declare (fixnum x))
;; X must be a fixnum now
(let ((x 'bar))
;; This X need not be a fixnum
...)))
The point of saying that the declaration applies to references and not
bindings is that it doesn't necessarily reach out to the entire scope of
the binding, but just those references to that binding within the scope of
the form introducing the declaration.
I'm not entirely sure of all this, so I'd appreciate it if other X3J13
members would reread these issues and let me know what you think.
--
Barry Margolin
System Manager, Thinking Machines Corp.
······@think.com {uunet,harvard}!think!barmar
From: Jeff Dalton
Subject: Re: DEFCONSTANT (was Re: A pitfalls list)
Date:
Message-ID: <7122@skye.ed.ac.uk>
In article <············@early-bird.think.com> ······@think.com (Barry Margolin) writes:
>In article <····@skye.ed.ac.uk> ····@aiai.ed.ac.uk (Jeff Dalton) writes:
>>I hope you're right, but CLtL II sys explicitly (page 219) that
>>type declarations refer to every reference no matter what binding
>>the reference refers to. And then on page 222 CLtL says that local
>>type declarations don't the proclaimed type of special variables,
>>noting that this is consistent with "the treatment of nested local
>>type declarations".
>>
>>So, in addition to DECLARE-TYPE-FREE:LEXICAL, SPECIAL-TYPE-SHADOWING:
>>CLARIFY may be relevant.
>
>SPECIAL-TYPE-SHADOWING only concerns special variables, so it's not an
>issue here.
I think it's relevant only because of the consistency issue.
> But after rereading DECLARE-TYPE-FREE:LEXICAL, now I'm not as
>sure as I was yesterday. Since you (and perhaps other readers) can't FTP
>it, I'll include it here:
Darn.
> Specify that a type declaration does not only "affect variable bindings";
> rather, type declarations are legal in all declarations. The interpretation
> of a type declaration is that, during the execution of any reference to the
> declared variable within the scope of the declaration, it is an error for
> the value of the declared variable not to be of the declared type; and
> during the execution of any SETQ of the declared variable within the scope
> of the declaration, [...]
So we need to know what is the "scope of the declaration" ...
>type declarations were only allowed at the beginning of the body of the
>form that bound the variables whose types were being declared. The
>extension is to allow them in other places, and specify what the scope of
>those new declarations is. All the examples in DECLARE-TYPE-FREE are of
>declarations that are not at the head of a form that binds the variable
>(unlike the example on p.219).
Ok.
>Earlier on p.219, there's a bullet that says, "If the declaration applies
>to a name binding, then the scope of the declaration also includes the
>scope of the name binding."
> [...]
>But what of the previous bullet, which says, "always includes the body
>forms"? This seems like it could imply the interpretation that Steele
>made, which bothered you. What this specifically means isn't really made
>clear in the proposal, but the Discussion section says, "Remember also that
>new name bindings "shadow" (after a fashion) any higher level binding or
>declarations." I think the use of the term "also" may have been a mistake
>(by the author of DECLARATION-SCOPE:NO-HOISTING, not GLS); I believe the
>intent was that when a declaration applies to a name being bound, that the
>declaration's scope is precisely the same as the binding being introduced
>by that form.
That makes sense.
>>>(let ((x 1))
>>> (locally (declare (fixnum x))
>>> (let ((x ...))
>>> ;; This X must be a fixnum too!
>>> ...)))
>>>
>>>Because LOCALLY doesn't bind variables, its type declarations are not
>>>associated with a specific variable binding, but to all bindings of the
>>>specified variable within its scope.
>>
>>I think this is a reasonable thing for LOCALLY to do, even though
>>it does make LOCALLY ++magic. No wonder it's now a special form!
>
>I take back my example. The intent of DECLARE-TYPE-FREE:LEXICAL was to
>allow the type declaration in the LOCALLY, but it still only affects the
>binding that is in effect at the time.
That also makes sense.
I've just looked at the X3J13 draft 12.24 (is that the right one),
pages 3.32 and 3-33, and it doesn't completely resolve the issue
either.
When "scope of a declaration" is finally defined, what we see is:
The scope of a declaration at the head of a [form]
1. ... always includes the body forms ...
2. ... also includes the scope of the name binding, if any,
to which it applies ...
So the ambiguities you noted above are retained.
-- jd