I'm a schemer, and ran into the following general problem, and I'm
wondering if there is a built-in solution in CL.
I'm using a Scheme system (specifically Gambit) with an
object system on top of it (Meroon). The object system has some
fairly complicated macros that do code-walks of forms and replace
what look like variable references with slot accessors and setters.
If I, as the user, define and use macros that introduce forms that introduce
new bindings, then the Meroon code-walker gets confused, because it doesn't
know about the bindings introduced by my macros.
This seems to be a fairly straightforward problem, in that Lisp
is a language for defining languages for defining languages ...,
so if one defines things in layers, each layer must, in some way,
assume that the code it's handed is defined in terms of the forms
defined by that and by all lower layers. In other words, forms
introduced by all higher layers must have already been macro-expanded.
And this needs to be done recursively.
So, my question is, is there a standard way to do this in CL?
Brad Lucier
···@cs.purdue.edu (Bradley J Lucier) writes:
> I'm a schemer, and ran into the following general problem, and I'm
> wondering if there is a built-in solution in CL.
>
> I'm using a Scheme system (specifically Gambit) with an
> object system on top of it (Meroon). The object system has some
> fairly complicated macros that do code-walks of forms and replace
> what look like variable references with slot accessors and setters.
Learn about symbol macros. You don't have to do a codewalk.
(let ((z (list 'a 'b 'c)))
(symbol-macrolet ((x (car z)))
(setq x 'd)
(list x z)))
=> (D (D B C))
> If I, as the user, define and use macros that introduce forms that introduce
> new bindings, then the Meroon code-walker gets confused, because it doesn't
> know about the bindings introduced by my macros.
Sounds the code-walker isn't up to actually doing its job. But
you should be able to get around using it. CLOS uses SYMBOL-MACROLET
to implement WITH-SLOTS and WITH-ACCESSORS.
See CLHS for more info on SYMBOL-MACROLET and DEFINE-SYMBOL-MACRO.
http://www.xanalys.com/software_tools/reference/HyperSpec/FrontMatter/
> This seems to be a fairly straightforward problem, in that Lisp
> is a language for defining languages for defining languages ...,
> so if one defines things in layers, each layer must, in some way,
> assume that the code it's handed is defined in terms of the forms
> defined by that and by all lower layers. In other words, forms
> introduced by all higher layers must have already been macro-expanded.
> And this needs to be done recursively.
Yes, but not necessarily explicitly. It's not incompatible with the
Lisp philosophy to have a code-walker, but Lisp happens not to. And,
without prejudice as to whether that makes things grossly harder,
mostly you can get by without one.
> So, my question is, is there a standard way to do this in CL?
See above.
···@cs.purdue.edu (Bradley J Lucier) writes:
> I'm using a Scheme system (specifically Gambit) with an
> object system on top of it (Meroon). The object system has some
> fairly complicated macros that do code-walks of forms and replace
> what look like variable references with slot accessors and setters.
We have a standard way to do _that_ in CL, in the form of
symbol-macros, BTW.
> If I, as the user, define and use macros that introduce forms that introduce
> new bindings, then the Meroon code-walker gets confused, because it doesn't
> know about the bindings introduced by my macros.
If I understand your problem correctly, then it seems to me that the
code-walker of Meroon is simply incorrect. A code-walker will have to
macroexpand each form, before it code-walks it. Otherwise, how is it
supposed to know what each subform of a form means?
But maybe I'm misunderstanding your problem, since I can't imagine
someone getting this wrong when writing a code-walker...
> So, my question is, is there a standard way to do this in CL?
You macroexpand as you go (walk) along, using macroexpand (there are
some issues about environments, if the code you walk introduces new
locally scoped macros, that need to be expanded, etc., but that
doesn't seem to be your problem here). Once macroexpand is done, the
car of the current form is either a special-operator, a lambda-form,
or a valid function name. In the later case the form is a function
call, and you simply code-walk the argument forms recursively. In the
former case its a special form, and you know how to identify sub-forms
for each valid special operator. Again walk those forms recursively.
Again, I feel that I'm missing something here...
Regs, Pierre.
--
Pierre R. Mai <····@acm.org> http://www.pmsf.de/pmai/
The most likely way for the world to be destroyed, most experts agree,
is by accident. That's where we come in; we're computer professionals.
We cause accidents. -- Nathaniel Borenstein
Pierre Mai writes:
> ···@cs.purdue.edu (Bradley J Lucier) writes:
>
> > If I, as the user, define and use macros that introduce forms that introduce
> > new bindings, then the Meroon code-walker gets confused, because it doesn't
> > know about the bindings introduced by my macros.
>
> If I understand your problem correctly, then it seems to me that the
> code-walker of Meroon is simply incorrect. A code-walker will have to
> macroexpand each form, before it code-walks it. Otherwise, how is it
> supposed to know what each subform of a form means?
Thank you for your response. I read the hyperspec about evaluation
and macros after reading your next paragraph.
It seems to be implicit to you that a macro that uses a code-walker
must transform its arguments to expand all macros in its arguments before
applying whatever macro expansion function it has. This seems to be the
opposite of usual macro expansion rules, which transform from the outer-most
macro form in. But, as you say, you can do this in CL with macroexpand.
So this implies that a macro that uses a code-walker in your style cannot rely on
recognizing and/or transforming any forms other than the predefined special
forms defined in section 3.1.2.1.2.1 of the hyperspec. And even those
forms might be removed/transformed by macroexpand, since, by section
3.1.2.1.2.2, "an implementation is free to implement a Common Lisp
special operator as a macro." Or is macroexpand guaranteed *not* to expand
special operators that are defined as macro operators.
So in CL one can solve this specific example of the general problem with
symbol-macros (which are cool) or by changing the default order of macro
expansion with explicit calls to macroexpand.
To get back to your question "how is it supposed to know what each subform
of a form means?", in this layered, language-upon-language, model, it needs
only expand the forms that were defined in languages built on top of
the language that contains the macro/code-walker, since the macro/code-walker
knows, in principle, about all forms in languages that it is built on.
So, to be more specific, can one specify with macroexpand which sets of
macro forms to expand (like the ones defined in languages built higher on
the pile) and which not to expand (like special operators that might happen
to be defined as macros in any given implementation)?
Brad
···@cs.purdue.edu (Bradley J Lucier) writes:
> It seems to be implicit to you that a macro that uses a code-walker
> must transform its arguments to expand all macros in its arguments before
> applying whatever macro expansion function it has. This seems to be the
> opposite of usual macro expansion rules, which transform from the outer-most
> macro form in.
I don't understand your remarks here well enough to be sure, but it sounds
like you're still slightly confused.
First, statistically, most macros do not expand their arguments. They merely
divide them into the arguments that are syntactic in structure and the
arguments that are evaluable forms and reassemble a new form without regard
to what those forms mean. For example:
(defmacro twice (x)
`(let ((temp ,x)) (+ temp temp)))
This macro does not expand the macro which is its argument. It merely
puts it somewhere in the new structure and leaves it to the system to
figure out how to continue. So if you do
(twice (twice x))
its expansion will be (let ((temp (twice x))) (+ temp temp)). Then the
system will proceed to process that new form until it encounters the
(twice x)
and it will expand it in place, getting
(let ((temp x)) (+ temp temp))
which it will compile in the lexical position as if
(let ((temp (let ((temp x)) (+ temp temp)))) (+ temp temp))
had been done.
But, second, this effect is an outward-in processing just as in most
systems. There is no rational way that you can proceed inward-out in
a system with macros because you can't know what is even a form (and
what is just syntax). For example, given the above:
(let ((twice 3))
(cond (twice 4) (t 5)))
will return 4 but it will not expand the macro TWICE at all because there
are no subforms of the expression which are references to the macro TWICE.
You can't just go find the two inner places that are (twice 3) and (twice 4)
and assume they are macro references because the outer LET and COND have
not been processed.
So macro expansion does proceed from the outside in, but macro definitions
get involved midway in that process and so they receive some arguments that
are expanded and some that are not. The system can't expand the arguments
to your macro before you see them because your macro might be like LET or
COND and might be intending to use the mere shape of the macro in some
way not as forms.
> But, as you say, you can do this in CL with macroexpand.
You can involve macroexpand, but it will expand only the current form and
not its subforms. To expand the subforms, you must write a code-walker
that works more correctly than it sounds like the one you're using does.
> So this implies that a macro that uses a code-walker in your style cannot rely on
> recognizing and/or transforming any forms other than the predefined special
> forms defined in section 3.1.2.1.2.1 of the hyperspec.
No, it just must call the MACROEXPAND function when it encounters them.
What it cannot easily do, because no operation is provided to do this, is
accomodate MACROLET, FLET, and LABELS forms, because these bind the set
of lexically apparent forms in a way that MACROEXPAND needs to know about.
If you are given a form like:
(setf (car x) x)
you can call macroexpand on it and it will probably give you more
primitive forms involving special forms and functions that you can then
codewalk. But if you are given a form like
(macrolet ((zap (x) `(setf (car ,x) nil)))
(zap x))
then it is fairly difficult to code-walk because you first need to augment
the lexical environment with the definition of ZAP, and then you need to
call MACROEXPAND with that augmented lexical environment, and no operators
are provided in the language to let you do this.
> And even those
> forms might be removed/transformed by macroexpand, since, by section
> 3.1.2.1.2.2, "an implementation is free to implement a Common Lisp
> special operator as a macro." Or is macroexpand guaranteed *not* to expand
> special operators that are defined as macro operators.
You're not understanding that this part you're quoting is helping you,
not making it harder. You can still recognize the special operator
by using SPECIAL-OPERATOR-P. You don't have to macroexpand the special form.
You can understand it directly. But you have the option of calling the
expander if you want. You have choices here. Perhaps you didn't see
the operator called SPECIAL-OPERATOR-P so that's why you worried.
> So in CL one can solve this specific example of the general problem with
> symbol-macros (which are cool) or by changing the default order of macro
> expansion with explicit calls to macroexpand.
The latter option is a little tricky.
If you get into this really heavily and you need a code-walker to work and
you're really conversant in these issues pretty strongly, there is some
advice from me about this in the deja.com archives. Search for
posts that contain the words MACROLET and FLET and LABELS and it may narrow
your search.
> To get back to your question "how is it supposed to know what each subform
> of a form means?", in this layered, language-upon-language, model, it needs
> only expand the forms that were defined in languages built on top of
> the language that contains the macro/code-walker, since the macro/code-walker
> knows, in principle, about all forms in languages that it is built on.
> So, to be more specific, can one specify with macroexpand which sets of
> macro forms to expand (like the ones defined in languages built higher on
> the pile) and which not to expand (like special operators that might happen
> to be defined as macros in any given implementation)?
You have a conceptual problem here. MACROEXPAND does not work
recursively. It iterates expanding an individual form until it's done
expanding and then it returns. MACROEXPAND-1 doesn't even work
iteratively. It is the job of the active language processor (either a
code walker or the interpreter or a compiler) to then continue its own
recursive descent into the form, continuing to look for macro
subforms. See my example with TWICE above.
(Kent Pitman explains in detail how CL macros are expanded.)
Thank you, I understand now. In defense of Meroon, I see now that
Scheme does not give one the tools available in CL to write a correct
codewalker (which is something that Queinnec says at one point, but
something I didn't understand until now), at least not with defmacro
style macros (which were never a part of the Scheme standard, but which
are available in many Scheme implementations). I don't know what the
situation is with the hygienic macros in R5RS.
Brad Lucier