From: Jeff Dalton
Subject: A pitfalls list
Date: 
Message-ID: <7062@skye.ed.ac.uk>
I've been keeping a list of Common Lisp "pitfalls" -- things that
I sometimes forget, that have caught students and other people
in Edinburgh, or that I think might catch people in unexpected
ways.

There's some overlap with the FAQ pitfalls list, but since I wanted
to include items what were not "frequently asked" and wanted to allow
"checklist" entries without much in the way of explanation, I thought
it made more sense to keep a spearate list.

So, in the hope that it might be useful, here it is.

I would appreciate any comments or suggested additions.

(I'm a bit worried about the claim that names declared by
DEFCONSTANT can't be bound even as lexical variables.  That
sure seems to be what CLtL II says, but is it still that way
in the X3J13 draft?)

----------------------------------------------------------------------
This is a list of "pitfalls": somewhat counterintuitive things that 
programmers may not expect Common Lisp to do and that tend to be
revealed only by a careful reading of CLtL.

A different, but overlapping, list of pitfalls can be found in the 
"Frequently Asked Questions" list of Comp.lang.lisp.

Functions.

  * The result of many non-destructive functions such as REMOVE
    and UNION can share structure with an argument.

  * SORT is (usually) destructive.

    So, for instance, (SORT (REMOVE ...) ...) may not be safe.

  * Destructive functions that you think would modify CDRs might
    modify CARs instead.  (Eg, NREVERSE.)

  * Array elements might not be initialized to NIL.  Eg,

       (make-array 10) => #(0 0 0 0 0 0 0 0 0 0)

  * READ-FROM-STRING has some optional arguments before the
    keyword parameters.  If you want to supply some keyword
    arguments, you have to give all of the optional ones too.

  * (flet ((f ...)) (eq #'f #'f)) can return false.

  * The function LIST-LENGTH is not a faster, list-specific version
    of the sequence function LENGTH.  It is list-specific, but it's
    slower than LENGTH because it can handle circular lists.

Iteration vs closures.

  * DO and DO* update the iteration variables by assignment; DOLIST and
    DOTIMES are allowed to use assignment (rather than a new binding).
    (All CLtL says of DOLIST and DOTIMES is that the variable "is
    bound" which has been taken as _not_ implying that there will be
    separate bindings for each iteration.)

    Consequently, if you make closures over an iteration variable
    in separate iterations they may nonetheless be closures over
    the same variable and hence will all refer to the same value
    -- whatever value the variable was given last.  Eg,

     (let ((fns '()))
       (do ((x '(1 2) (cdr x)))
           ((null x))
         (push #'(lambda () x)
               fns))
       (mapcar #'funcall (reverse fns)))

    returns (nil nil), not (1 2), not even (2 2).

Limits.

  * The array-total-size-limit may be as small as 1024.

  * Such things as (apply #'append list-of-lists) to flatten a list
    of lists may run afoul of call-arguments-limit.

     Alternatives include:

       (reduce #'append list-of-lists :from-end t)

       (mapcan #'copy-list list-of-lists)

Definitions and declarations.

  * (DEFVAR var init) assigns to the variable only if it does not
    already have a value.  So if you edit a DEFVAR in a file and
    reload the file only to find that the value has not changed,
    this is the reason.  (Cf DEFPARAMETER.)

  * DEFCONSTANT has several potentially unexpected properties:

     - Once a name has been declared constant, it cannot be used a
       the name of a local variable (lexical or special) or function
       parameter.  Really.  See page 87 of CLtL II.

     - A DEFCONSTANT cannot be re-evaluated (eg, by reloading the
       file in which it appears) unless the new value is EQL to the
       old one.  Strictly speaking, even that may not be allowed.
       (DEFCONSTANT is "like DEFPARAMETER" and hence does an
       assignment, which is not allowed if the name has already
       been declared constant by DEFCONSTANT.)

       Note that this makes it difficult to use anything other
       than numbers, symbols, and characters as constants.       

     - When compiling (DEFCONSTANT name form) in a file, the form
       may be evaluated at compile-time, load-time, or both.  

       (You might think it would be evaluated at compile-time and
       the _value_ used to obtain the object at load-time, but it
       doesn't have to work that way.)

  * You often have to declare the result type to get the most
    efficient arithmetic.  Eg, 

       (the fixnum (+ (the fixnum e1) (the fixnum e2)))

     rather than

       (+ (the fixnum e1) (the fixnum e2))

  * Declaring the iteration variable of a DOTIMES to have type FIXNUM
    does not guarantee that fixnum arithmetic will be used.  That is,
    implementations that use fixnum-specific arithmetic in the presence
    of appropriate declaration may not think _this_ declaration is
    sufficient.  It may help to declare that the limit is also a
    fixnum, or you may have to write out the loop as a DO and add
    appropriate declarations for each operation involved.

----------------------------------------------------------------------

Jeff Dalton,                      JANET: ········@uk.ac.ed             
AI Applications Institute,        ARPA:  ········@ed.ac.uk
Edinburgh University.             UUCP:  ...!ukc!ed.ac.uk!J.Dalton

From: Barry Margolin
Subject: DEFCONSTANT (was Re: A pitfalls list)
Date: 
Message-ID: <15p53mINNn38@early-bird.think.com>
In article <····@skye.ed.ac.uk> ····@aiai.ed.ac.uk (Jeff Dalton) writes:
>(I'm a bit worried about the claim that names declared by
>DEFCONSTANT can't be bound even as lexical variables.  That
>sure seems to be what CLtL II says, but is it still that way
>in the X3J13 draft?)

Yes.  I can see where you're coming from -- it's a violation of referential
transparency.

I suspect this was done to generalize on the restriction against binding T
and NIL, which has been in most dialects of Lisp for a long time.  In
Common Lisp, T is just a variable that's predefined as a constant, and it
obeys the rules that DEFCONSTANT specifies (NIL is still special, though,
because of its equivalence to the empty list).
-- 
Barry Margolin
System Manager, Thinking Machines Corp.

······@think.com          {uunet,harvard}!think!barmar
From: Jeff Dalton
Subject: Re: DEFCONSTANT (was Re: A pitfalls list)
Date: 
Message-ID: <7081@skye.ed.ac.uk>
In article <············@early-bird.think.com> ······@think.com (Barry Margolin) writes:
>In article <····@skye.ed.ac.uk> ····@aiai.ed.ac.uk (Jeff Dalton) writes:
>>(I'm a bit worried about the claim that names declared by
>>DEFCONSTANT can't be bound even as lexical variables.  That
>>sure seems to be what CLtL II says, but is it still that way
>>in the X3J13 draft?)
>
>Yes.  I can see where you're coming from -- it's a violation of referential
>transparency.

Yes, and another thing I noticed last night (when reading up on
declarations in CLtL II) is that the new rules for type declarations
mean that:

  (let ((x 1))
    (declare (fixnum x))
    (let ((x ...))
      ;; This X must be a fixnum too!
      ...))

I must have seen the change before and not really paid attention,
because it certainly looks wrong to me now.  Moreover, I'm not sure
what the reasons for it can be.  Changing type declarations to
apply to refs rather than bindings seems reasonable.  But the
inner X isn't a reference to the same thing.

So I guess this is another pitfall.

Anyway,	 I can understand the reasoning behind disallowing bindings of
T and NIL.  I think a generalization to some other constants might be
reasonable, but don't the rules on the LISP package already prevent
rebinding any "standard" constants?  The full generalization to all
constants seems somewhat questionable (though I'm not sure it's
wrong, all things considered), and so I wondered whether X3J13
might have changed its mind.

-- jeff
From: Barry Margolin
Subject: Re: DEFCONSTANT (was Re: A pitfalls list)
Date: 
Message-ID: <15rmdnINNag1@early-bird.think.com>
In article <····@skye.ed.ac.uk> ····@aiai.ed.ac.uk (Jeff Dalton) writes:
>Yes, and another thing I noticed last night (when reading up on
>declarations in CLtL II) is that the new rules for type declarations
>mean that:
>
>  (let ((x 1))
>    (declare (fixnum x))
>    (let ((x ...))
>      ;; This X must be a fixnum too!
>      ...))

I think this example in CLtL2 is bogus and unsupported by anything in
DECLARE-TYPE-FREE:LEXICAL.  That issue covered type declarations that
aren't at the head of the body of a form that binds the variable.  A
more correct example would be:

(let ((x 1))
  (locally (declare (fixnum x))
    (let ((x ...))
      ;; This X must be a fixnum too!
      ...)))

Because LOCALLY doesn't bind variables, its type declarations are not
associated with a specific variable binding, but to all bindings of the
specified variable within its scope.  Also, the following is still correct:

(let ((x 'foo))
  ...
  (setq x 1)
  (locally (declare (fixnum x))
    (let ((x ...))
      ;; This X must be a fixnum too!
      ...)))

X is initially bound to a symbol, but that is outside the scope of the
LOCALLY so it's OK.  You just have to make sure that you assign a fixnum to
it before entering the LOCALLY (even if you never actually reference the
outer X inside the LOCALLY).

>Anyway, I can understand the reasoning behind disallowing bindings of
>T and NIL.  I think a generalization to some other constants might be
>reasonable, but don't the rules on the LISP package already prevent
>rebinding any "standard" constants?  The full generalization to all
>constants seems somewhat questionable (though I'm not sure it's
>wrong, all things considered), and so I wondered whether X3J13
>might have changed its mind.

I don't recall the issue ever being brought up to the full committee.
Someone who was on the Cleanup committee could possibly tell you whether
they ever discussed it.
-- 
Barry Margolin
System Manager, Thinking Machines Corp.

······@think.com          {uunet,harvard}!think!barmar
From: Jeff Dalton
Subject: Re: DEFCONSTANT (was Re: A pitfalls list)
Date: 
Message-ID: <7110@skye.ed.ac.uk>
In article <············@early-bird.think.com> ······@think.com (Barry Margolin) writes:
>In article <····@skye.ed.ac.uk> ····@aiai.ed.ac.uk (Jeff Dalton) writes:
>>Yes, and another thing I noticed last night (when reading up on
>>declarations in CLtL II) is that the new rules for type declarations
>>mean that:
>>
>>  (let ((x 1))
>>    (declare (fixnum x))
>>    (let ((x ...))
>>      ;; This X must be a fixnum too!
>>      ...))
>
>I think this example in CLtL2 is bogus and unsupported by anything in
>DECLARE-TYPE-FREE:LEXICAL.  That issue covered type declarations that
>aren't at the head of the body of a form that binds the variable. 

I hope you're right, but CLtL II sys explicitly (page 219) that
type declarations refer to every reference no matter what binding
the reference refers to.  And then on page 222 CLtL says that local
type declarations don't the proclaimed type of special variables,
noting that this is consistent with "the treatment of nested local
type declarations".

So, in addition to DECLARE-TYPE-FREE:LEXICAL, SPECIAL-TYPE-SHADOWING:
CLARIFY may be relevant.

(Unfortunately, I can't ftp them at present.)

>A more correct example would be:
>
>(let ((x 1))
>  (locally (declare (fixnum x))
>    (let ((x ...))
>      ;; This X must be a fixnum too!
>      ...)))
>
>Because LOCALLY doesn't bind variables, its type declarations are not
>associated with a specific variable binding, but to all bindings of the
>specified variable within its scope.

I think this is a reasonable thing for LOCALLY to do, even though
it does make LOCALLY ++magic.  No wonder it's now a special form!

-- jeff
From: Barry Margolin
Subject: Re: DEFCONSTANT (was Re: A pitfalls list)
Date: 
Message-ID: <15ujksINN4f7@early-bird.think.com>
In article <····@skye.ed.ac.uk> ····@aiai.ed.ac.uk (Jeff Dalton) writes:
>I hope you're right, but CLtL II sys explicitly (page 219) that
>type declarations refer to every reference no matter what binding
>the reference refers to.  And then on page 222 CLtL says that local
>type declarations don't the proclaimed type of special variables,
>noting that this is consistent with "the treatment of nested local
>type declarations".
>
>So, in addition to DECLARE-TYPE-FREE:LEXICAL, SPECIAL-TYPE-SHADOWING:
>CLARIFY may be relevant.

SPECIAL-TYPE-SHADOWING only concerns special variables, so it's not an
issue here.  But after rereading DECLARE-TYPE-FREE:LEXICAL, now I'm not as
sure as I was yesterday.  Since you (and perhaps other readers) can't FTP
it, I'll include it here:

  Specify that a type declaration does not only "affect variable bindings";
  rather, type declarations are legal in all declarations. The interpretation
  of a type declaration is that, during the execution of any reference to the
  declared variable within the scope of the declaration, it is an error for
  the value of the declared variable not to be of the declared type; and
  during the execution of any SETQ of the declared variable within the scope
  of the declaration, it is an error for the newly assigned value of the
  declared variable not to be of the declared type; and at the moment the
  scope of the declaration is entered, it is an error for the value of the
  declared variable not to be of the declared type.

  In this proposal, a type declaration affects only variable references within
  its scope, and the meaning of "free" and "variable-binding-associated" type
  declarations can be described identically.

  This proposal is equivalent to saying that the meaning of a type declaration
  is equivalent to changing each reference to <var> within the scope of the
  declaration to (THE <type> <var>), changing each expression assigned to the
  variable within the scope of the declaration to (THE <type> <new-value>),
  and executing (THE <type> <var>) at the moment the scope of the declaration
  is entered.

  Clarify that if nested type declarations refer to the same variable,
  then the value of the variable must be a member of the intersection of
  the declared types.

I believe the intent of this proposal was as an extension.  Prior to this,
type declarations were only allowed at the beginning of the body of the
form that bound the variables whose types were being declared.  The
extension is to allow them in other places, and specify what the scope of
those new declarations is.  All the examples in DECLARE-TYPE-FREE are of
declarations that are not at the head of a form that binds the variable
(unlike the example on p.219).

Earlier on p.219, there's a bullet that says, "If the declaration applies
to a name binding, then the scope of the declaration also includes the
scope of the name binding."  This is derived from
DECLARATION-SCOPE:NO-HOISTING, whose primary purpose was to remove initial
value forms from the scope of declarations.  Formerly, in the form:

(let ((x (foo x)))
  (declare (integer x))
  ...)

the INTEGER declaration applied both to the X being bound and the outer
X being referenced in the initialization form; now it only applies to the
one being bound.

But what of the previous bullet, which says, "always includes the body
forms"?  This seems like it could imply the interpretation that Steele
made, which bothered you.  What this specifically means isn't really made
clear in the proposal, but the Discussion section says, "Remember also that
new name bindings "shadow" (after a fashion) any higher level binding or
declarations."  I think the use of the term "also" may have been a mistake
(by the author of DECLARATION-SCOPE:NO-HOISTING, not GLS); I believe the
intent was that when a declaration applies to a name being bound, that the
declaration's scope is precisely the same as the binding being introduced
by that form.

>>(let ((x 1))
>>  (locally (declare (fixnum x))
>>    (let ((x ...))
>>      ;; This X must be a fixnum too!
>>      ...)))
>>
>>Because LOCALLY doesn't bind variables, its type declarations are not
>>associated with a specific variable binding, but to all bindings of the
>>specified variable within its scope.
>
>I think this is a reasonable thing for LOCALLY to do, even though
>it does make LOCALLY ++magic.  No wonder it's now a special form!

I take back my example.  The intent of DECLARE-TYPE-FREE:LEXICAL was to
allow the type declaration in the LOCALLY, but it still only affects the
binding that is in effect at the time.  Thus, the following is OK:

(let ((x 'foo))
  ...
  (setq x 0)
  (locally (declare (fixnum x))
    ;; X must be a fixnum now
    (let ((x 'bar))
      ;; This X need not be a fixnum
        ...)))

The point of saying that the declaration applies to references and not
bindings is that it doesn't necessarily reach out to the entire scope of
the binding, but just those references to that binding within the scope of
the form introducing the declaration.

I'm not entirely sure of all this, so I'd appreciate it if other X3J13
members would reread these issues and let me know what you think.
-- 
Barry Margolin
System Manager, Thinking Machines Corp.

······@think.com          {uunet,harvard}!think!barmar
From: Jeff Dalton
Subject: Re: DEFCONSTANT (was Re: A pitfalls list)
Date: 
Message-ID: <7122@skye.ed.ac.uk>
In article <············@early-bird.think.com> ······@think.com (Barry Margolin) writes:
>In article <····@skye.ed.ac.uk> ····@aiai.ed.ac.uk (Jeff Dalton) writes:
>>I hope you're right, but CLtL II sys explicitly (page 219) that
>>type declarations refer to every reference no matter what binding
>>the reference refers to.  And then on page 222 CLtL says that local
>>type declarations don't the proclaimed type of special variables,
>>noting that this is consistent with "the treatment of nested local
>>type declarations".
>>
>>So, in addition to DECLARE-TYPE-FREE:LEXICAL, SPECIAL-TYPE-SHADOWING:
>>CLARIFY may be relevant.
>
>SPECIAL-TYPE-SHADOWING only concerns special variables, so it's not an
>issue here. 

I think it's relevant only because of the consistency issue.

> But after rereading DECLARE-TYPE-FREE:LEXICAL, now I'm not as
>sure as I was yesterday.  Since you (and perhaps other readers) can't FTP
>it, I'll include it here:

Darn.

>  Specify that a type declaration does not only "affect variable bindings";
>  rather, type declarations are legal in all declarations. The interpretation
>  of a type declaration is that, during the execution of any reference to the
>  declared variable within the scope of the declaration, it is an error for
>  the value of the declared variable not to be of the declared type; and
>  during the execution of any SETQ of the declared variable within the scope
>  of the declaration,  [...]

So we need to know what is the "scope of the declaration" ...

>type declarations were only allowed at the beginning of the body of the
>form that bound the variables whose types were being declared.  The
>extension is to allow them in other places, and specify what the scope of
>those new declarations is.  All the examples in DECLARE-TYPE-FREE are of
>declarations that are not at the head of a form that binds the variable
>(unlike the example on p.219).

Ok.

>Earlier on p.219, there's a bullet that says, "If the declaration applies
>to a name binding, then the scope of the declaration also includes the
>scope of the name binding."

> [...]

>But what of the previous bullet, which says, "always includes the body
>forms"?  This seems like it could imply the interpretation that Steele
>made, which bothered you.  What this specifically means isn't really made
>clear in the proposal, but the Discussion section says, "Remember also that
>new name bindings "shadow" (after a fashion) any higher level binding or
>declarations."  I think the use of the term "also" may have been a mistake
>(by the author of DECLARATION-SCOPE:NO-HOISTING, not GLS); I believe the
>intent was that when a declaration applies to a name being bound, that the
>declaration's scope is precisely the same as the binding being introduced
>by that form.
 
That makes sense.

>>>(let ((x 1))
>>>  (locally (declare (fixnum x))
>>>    (let ((x ...))
>>>      ;; This X must be a fixnum too!
>>>      ...)))
>>>
>>>Because LOCALLY doesn't bind variables, its type declarations are not
>>>associated with a specific variable binding, but to all bindings of the
>>>specified variable within its scope.
>>
>>I think this is a reasonable thing for LOCALLY to do, even though
>>it does make LOCALLY ++magic.  No wonder it's now a special form!
>
>I take back my example.  The intent of DECLARE-TYPE-FREE:LEXICAL was to
>allow the type declaration in the LOCALLY, but it still only affects the
>binding that is in effect at the time. 

That also makes sense.

I've just looked at the X3J13 draft 12.24 (is that the right one),
pages 3.32 and 3-33, and it doesn't completely resolve the issue
either.

When "scope of a declaration" is finally defined, what we see is:

   The scope of a declaration at the head of a [form] 

     1. ... always includes the body forms ...

     2. ... also includes the scope of the name binding, if any,
	to which it applies ...

So the ambiguities you noted above are retained.

-- jd