From: Greg Bacon
Subject: apparently undefined function called by macro
Date: 
Message-ID: <12a2jt2ta2gk539@corp.supernews.com>
I'm using the following simple system:

    ; foo.asd

    (defpackage :foo-system (:use :asdf :cl))
    (in-package :foo-system)

    (defsystem foo
      :components
      ((:file "packages")
       (:file "foo" :depends-on ("packages"))))

    ; packages.lisp

    (in-package :cl-user)

    (defpackage :foo
      (:use :common-lisp))

    ; foo.lisp

    (in-package :foo)

    ;(eval-when (:load-toplevel :compile-toplevel :execute)
    (defun id (x) x)
    ;)

    (defmacro define-foo-class (name slots)
      `(defclass ,name ()
        ,(mapcar #'id slots)))

    (define-foo-class bar
      ((baz  :initarg :baz  :accessor baz)
       (quux :initarg :quux :accessor quux)))

If I try to load it in a fresh image, clisp complains about undefined id:

    [1]> (asdf:oos 'asdf:load-op :foo)
    ; loading system definition from foo.asd into #<PACKAGE ASDF0>
    ;; Loading file foo.asd ...
    ; registering #<SYSTEM FOO #x102A2839> as FOO
    ;; Loaded file foo.asd
    ;; Compiling file /tmp/asdf/packages.lisp ...
    ;; Wrote file /tmp/asdf/packages.fas
    ;; Loading file /tmp/asdf/packages.fas ...
    ;; Loaded file /tmp/asdf/packages.fas
    ;; Compiling file /tmp/asdf/foo.lisp ...
    *** - FUNCTION: undefined function ID
    [...]

Loading by hand, i.e., (progn (load "packages.lisp") (load "foo.lisp")),
is no problem, and as you can guess from the commented lines, wrapping
ID in EVAL-WHEN is also a workaround.

Well, hmm, my misunderstanding is more basic than ASDF or even the
reader. After removing the initial IN-PACKAGE form from foo.lisp,
COMPILE-FILE still complains about undefined ID.

Why is ID undefined? Please help me correct the error in my mental
model.

Thanks,
Greg

From: Stefan Mandl
Subject: Re: apparently undefined function called by macro
Date: 
Message-ID: <4gd3pbF1ltb63U1@news.dfncis.de>
> Loading by hand, i.e., (progn (load "packages.lisp") (load "foo.lisp")),
> is no problem, and as you can guess from the commented lines, wrapping
> ID in EVAL-WHEN is also a workaround.

> Why is ID undefined? Please help me correct the error in my mental
> model.

 From my limited knowledge:

Remember that macros are functions that run during compilation.

When you call:

(define-foo-class bar
       ((baz  :initarg :baz  :accessor baz)
        (quux :initarg :quux :accessor quux)))

the *compiler* has to call the macro function and therefore has to call 
your id function.

So in order to make sure that the compiler knows id's definition, 
wrapping id into a eval-when form is no mere workaround but standard 
technique.

Regards,
Stefan
From: Greg Bacon
Subject: Re: apparently undefined function called by macro
Date: 
Message-ID: <12a2pjrciudu785@corp.supernews.com>
In article <···············@news.dfncis.de>,
    Stefan Mandl  <············@informatik.uni-erlangen.de> wrote:

: From my limited knowledge:
:
: Remember that macros are functions that run during compilation.
:
: When you call:
:
: (define-foo-class bar
:        ((baz  :initarg :baz  :accessor baz)
:         (quux :initarg :quux :accessor quux)))
:
: the *compiler* has to call the macro function and therefore has to call 
: your id function.
:
: So in order to make sure that the compiler knows id's definition, 
: wrapping id into a eval-when form is no mere workaround but standard 
: technique.

Thanks, Stefan.

I see that FAQ 3-7 also addresses the issue:

http://www.cs.cmu.edu/Groups/AI/html/faqs/lang/lisp/part3/faq-doc-8.html

I was confused because in chapter 24 of PCL, Seibel defines similar
macros that call functions, and all the code is ASDF-loadable.

The difference that I missed is he defines the macros in one package and
expands them in others after the whole supporting cast has been loaded.

Greg
From: Pascal Costanza
Subject: Re: apparently undefined function called by macro
Date: 
Message-ID: <4gdlsnF1n65dqU1@individual.net>
Greg Bacon wrote:

>     ;(eval-when (:load-toplevel :compile-toplevel :execute)
>     (defun id (x) x)
>     ;)
> 
>     (defmacro define-foo-class (name slots)
>       `(defclass ,name ()
>         ,(mapcar #'id slots)))
> 
>     (define-foo-class bar
>       ((baz  :initarg :baz  :accessor baz)
>        (quux :initarg :quux :accessor quux)))
> 
> If I try to load it in a fresh image, clisp complains about undefined id:
> 
>     [1]> (asdf:oos 'asdf:load-op :foo)
>     ; loading system definition from foo.asd into #<PACKAGE ASDF0>
>     ;; Loading file foo.asd ...
>     ; registering #<SYSTEM FOO #x102A2839> as FOO
>     ;; Loaded file foo.asd
>     ;; Compiling file /tmp/asdf/packages.lisp ...
>     ;; Wrote file /tmp/asdf/packages.fas
>     ;; Loading file /tmp/asdf/packages.fas ...
>     ;; Loaded file /tmp/asdf/packages.fas
>     ;; Compiling file /tmp/asdf/foo.lisp ...
>     *** - FUNCTION: undefined function ID
>     [...]
> 
> Loading by hand, i.e., (progn (load "packages.lisp") (load "foo.lisp")),
> is no problem, and as you can guess from the commented lines, wrapping
> ID in EVAL-WHEN is also a workaround.
> 
> Well, hmm, my misunderstanding is more basic than ASDF or even the
> reader. After removing the initial IN-PACKAGE form from foo.lisp,
> COMPILE-FILE still complains about undefined ID.

This isn't related to packages or systems (only in the sense that 
systems can help you to solve this problem - see below).

> Why is ID undefined? Please help me correct the error in my mental
> model.

Common Lisp distinguishes between interpretation and compilation. When 
code is interpreted, each top-level form is interpreted after another, 
which means that each form can rely on all effects being "correctly" 
produced by the previous forms. That is, a defun defines a function, 
defmacro defines a macro, a defvar defines a variable, and so on.

When code is compiled, this is not necessarily the case. It is the case 
that each form is processed by the compiler, and for example it is 
guaranteed that all macros are completely expanded and thus "removed" 
from the code, but it is not necessarily the case that the effects are 
produced. Typically, the effects will only be produced at runtime. So, 
for example, a defun only announces the presence of a function at 
compile-time (so that other parts of the code can be checked, for 
example whether they pass the right number of arguments, or whether you 
accidentally redefine a function somewhere else), but the function 
itself will not be available. Likewise, a defvar only announces the 
presence of a variable, but the binding and its value will not be 
available at compile time.

However, as almost always in Common Lisp ;), there are exceptions to 
this rules: Some top-level forms do indeed have effects at compile time. 
For example, a defmacro definition _will_ be fully available at compile 
time. The reason is that you typically want to base subsequent code on 
your macro definitions, and the compiler must be able to completely 
macro-expand away these macro definitions as well. But this in turn 
means that all the functions that a macro uses to produce the expansion 
must also be available at compile time - but for function definitions, 
this is typically not the case, as I explained above.

The (eval-when ...) form is there to exactly provide you with the a way 
to tell the compiler that, say, a global function definition should be 
available at compile time as well (or even to tell the compiler that it 
is _only_ available at compile time, etc.).

However, using eval-when all over the place is ugly and can be hard to 
deal with. A better way to organize your code is to take advantage of 
systems. A system definition allows you to declare that one file of lisp 
code depends on some other file of lisp code. By default, a system 
definition will process a file by compiling _and_ loading it before 
proceeding to the next file, so if a macro definition depends on the 
presence of some functions at compile time, it is a good to put all the 
support code in another file and declare in the system definition that 
your macros depend on that support code. Then you can safely forget 
about (eval-when ...) in 95% of the time.

And this also aligns well with another rule of thumb when implementing 
macros (at least more complex ones): It is a good idea to provide a 
functional layer of what you want to express at one level, i.e., purely 
by use of defuns, and then to provide macros to ease using the 
functional layer at a higher level. Just put the functional layer in one 
file (or set of files), and the macro / syntactic layer in another file 
(or set of files), and declare the correct dependencies in a system 
definition - done.

I hope this helps.


Pascal

-- 
3rd European Lisp Workshop
July 3 - Nantes, France - co-located with ECOOP 2006
http://lisp-ecoop06.bknr.net/
From: Greg Bacon
Subject: Re: apparently undefined function called by macro
Date: 
Message-ID: <12a3dcogn4b7170@corp.supernews.com>
In article <···············@individual.net>,
    Pascal Costanza  <··@p-cos.net> wrote:

: [...]
:
: I hope this helps.

Yes, it does. Thank you for your thorough followup!

Greg
From: Kaz Kylheku
Subject: Re: apparently undefined function called by macro
Date: 
Message-ID: <1151627038.404695.251430@y41g2000cwy.googlegroups.com>
Greg Bacon wrote:
>     ;(eval-when (:load-toplevel :compile-toplevel :execute)
>     (defun id (x) x)
>     ;)

Note the semicolons here which get rid of the (eval-when ...).

>     (defmacro define-foo-class (name slots)
>       `(defclass ,name ()
>         ,(mapcar #'id slots)))

If this macro has to be expanded as a part of compiling code, then the
function ID has exist.

A compiler does not automatically define the functions created by
DEFUN. Of course, it reads them and translates them to the object code.
But it does not install them in its own image.

A macro is a program that runs in the compiler's image. If it calls any
functions, they have to exist in the compiler's image also. That
commented-out EVAL-WHEN is what tells the compiler to not only compile
the DEFUN but also define the function so that the macro can work.

> If I try to load it in a fresh image, clisp complains about undefined id:
>
>     [1]> (asdf:oos 'asdf:load-op :foo)
>     ; loading system definition from foo.asd into #<PACKAGE ASDF0>
>     ;; Loading file foo.asd ...
>     ; registering #<SYSTEM FOO #x102A2839> as FOO
>     ;; Loaded file foo.asd
>     ;; Compiling file /tmp/asdf/packages.lisp ...
>     ;; Wrote file /tmp/asdf/packages.fas
>     ;; Loading file /tmp/asdf/packages.fas ...
>     ;; Loaded file /tmp/asdf/packages.fas
>     ;; Compiling file /tmp/asdf/foo.lisp ...
>     *** - FUNCTION: undefined function ID

Note that the terror occurs in "Compiling file /tmp/asdf/foo.lisp".
Foo.lisp isn't being loaded, but compiled.

For instance look what processing happened immediately before with
packages.lisp:

  ;; Compiling file /tmp/asdf/packages.lisp ...
  ;; Wrote file /tmp/asdf/packages.fas
  ;; Loading file /tmp/asdf/packages.fas ...
  ;; Loaded file /tmp/asdf/packages.fas

It was compiled to produce a .fas file, which was then loaded.

Your foo.lisp is stuck at the first step: compiling! There was never an
attempt to load it; things didn't get that far.

>     [...]
>
> Loading by hand, i.e., (progn (load "packages.lisp") (load "foo.lisp")),

But (load "foo.lisp") doesn't compile it! You're skipping the step that
doesn't work.

In Lisp we can load things without compiling them, no surprises there.

They run, just a tad more slowly.

> is no problem, and as you can guess from the commented lines, wrapping
> ID in EVAL-WHEN is also a workaround.

That is not a workaround; that's the standard ANSI CL way to make sure
a function is available within the compiler at macro-expansion time.

> Well, hmm, my misunderstanding is more basic than ASDF or even the
> reader. After removing the initial IN-PACKAGE form from foo.lisp,
> COMPILE-FILE still complains about undefined ID.

It has nothing to do with packaging. The compiler has no problem with
the symbol ID itself, the error is that there is no function called ID.
If you change the package in which ID is read, it simply switches its
complaint to a different ID, since there is no function called ID in
this package, nor in that one.
From: Pascal Bourguignon
Subject: Re: apparently undefined function called by macro
Date: 
Message-ID: <87fyhnzdj3.fsf@thalassa.informatimago.com>
"Kaz Kylheku" <········@gmail.com> writes:
> Note that the terror occurs in "Compiling file /tmp/asdf/foo.lisp".
> Foo.lisp isn't being loaded, but compiled.

Shouldn't we all programmers get medals?

-- 
__Pascal Bourguignon__                     http://www.informatimago.com/

HEALTH WARNING: Care should be taken when lifting this product,
since its mass, and thus its weight, is dependent on its velocity
relative to the user.