Suppose I write a compiler-macro such that (add constant constant) ==>
constant. When will this macro get executed? What happens if the
optimizer determines the constants later on, will it call my macro
again?
[forgive my Lisp syntax, I'm really a schemer]
(let* ((x (+ 1 2)) (y (* x x)))
(add x y))
Given some suitably complicated program, the optimizer may run for
quite a while before it discovers that x and y are constants. Will it
then run my compiler-macro again? Or does it only run the macros once
before the main optimizer runs?
Thanks,
Pinku
"Pinku Surana" <·······@gmail.com> writes:
> Given some suitably complicated program, the optimizer may run for
> quite a while before it discovers that x and y are constants. Will it
> then run my compiler-macro again? Or does it only run the macros once
> before the main optimizer runs?
I suggest sticking a breakpoint in the middle of your compiler macro
and checking out the backtrace.
-russ
Russell McManus <···············@yahoo.com> writes:
> "Pinku Surana" <·······@gmail.com> writes:
>
> > Given some suitably complicated program, the optimizer may run for
> > quite a while before it discovers that x and y are constants. Will it
> > then run my compiler-macro again? Or does it only run the macros once
> > before the main optimizer runs?
>
> I suggest sticking a breakpoint in the middle of your compiler macro
> and checking out the backtrace.
I suggest not.
We had a HUGE debate in the design of CL about whether to allow
compiler macros at all. The fear of questions like the one above is
actually the key reason given by most vendors as to why there should
not be any compiler macros in CL--that users would want all kinds of
access to implementation details that could simply not be portably
agreed upon.
If you do as Russel McManus suggests, you can probably detect where the
compiler macro is done in some implementation on some circumstances, but
there is no guarantee that this will be true in another implementation.
I pleaded with the committee to just let people write these things at all,
so that they had at least SOME way to get at least SOME subset of what the
implementations often provide with considerably more control. My argument
was that anything was better than nothing. And this eventually prevailed.
There are certainly portable, useful things you can do.
But if you want to make the argument that a facility of this kind is not
powerful enough, nor well enough described, to allow all possible kinds of
optimizations, then certainly that's true, too.
The question in my mind isn't "are there times that the compiler won't
detect a constant?", etc. But "are there times when the compiler
macro makes things worse than if you didn't have it?" If you want to
raise such a case, I'd be a bit more interested just by way of
learning why my argument based on "incremental improvement" wasn't right.
Of course, if you're using only one implementation and plan to stick with
it to the bitter end, you could also just contact your vendor and ask them.
CL is a standard that allows portability, but every implementation of CL
is generally also a non-portable product with additional features that CL
doesn't have.
Kent M Pitman <······@nhplace.com> writes:
> Russell McManus <···············@yahoo.com> writes:
>
>> "Pinku Surana" <·······@gmail.com> writes:
>>
>> > Given some suitably complicated program, the optimizer may run
>> > for quite a while before it discovers that x and y are
>> > constants. Will it then run my compiler-macro again? Or does it
>> > only run the macros once before the main optimizer runs?
>>
>> I suggest sticking a breakpoint in the middle of your compiler macro
>> and checking out the backtrace.
>
> I suggest not.
I don't understand from your reply why you think it's a bad idea. You
will see what the arguments are, and what the stack trace is, etc.
This is the sort of thing that the poster asked for. I think it's
perfectly good practice to poke your implementation and listen to see
where it squeaks.
> If you do as Russel McManus suggests, you can probably detect where
> the compiler macro is done in some implementation on some
> circumstances, but there is no guarantee that this will be true in
> another implementation.
Yeah. And. Programmers need to keep in mind what the spec guarantees
and what it does not.
Perhaps an analogy is in order:
OP: How does a toilet work?
RM: Open a toilet and look! You'll learn something by just
examining it in action!
KP: Not all toilets are the same. Some use a floater, and some use
suction, blah blah blah...
In short, I don't think your comment helps the poster learn how
compiler macros work.
-russ
On Thu, 05 May 2005 15:01:42 -0400, <···············@yahoo.com> wrote:
>
> In short, I don't think your comment helps the poster learn how
> compiler macros work.
But it does pre-empt the followup, and I enjoyed the history lesson.
--
Everyman has three hearts;
one to show the world, one to show friends, and one only he knows.
Russell McManus <···············@yahoo.com> writes:
> Kent M Pitman <······@nhplace.com> writes:
>
> > Russell McManus <···············@yahoo.com> writes:
...
> > If you do as Russel McManus suggests, you can probably detect where
> > the compiler macro is done in some implementation on some
> > circumstances, but there is no guarantee that this will be true in
> > another implementation.
>
> Yeah. And. Programmers need to keep in mind what the spec guarantees
> and what it does not.
>
> Perhaps an analogy is in order:
>
> OP: How does a toilet work?
>
> RM: Open a toilet and look! You'll learn something by just
> examining it in action!
>
> KP: Not all toilets are the same. Some use a floater, and some use
> suction, blah blah blah...
>
> In short, I don't think your comment helps the poster learn how
> compiler macros work.
The difference is that when you learn about a toilet and you later
apply that knowledge, you do so with your full human ability to use
common sense about whether the knowledge applies.
When you look inside an implementation to see how it works, you
immediately write code that runs unsupervised in other circumstances
and might not do the same. If you're in an area that you don't
understand, you probably don't know how to make it not apply in cases
that it shouldn't. You certainly aren't likely to make it stop and
ask questions or call a plumber if the implementation looks different.
You also answered unconditionally without asking whether the person
was a novice or an expert.
Novices need guiderails to keep them on track. Often they are not
really asking about the particular feature and are asking about proper
ways to do things. Often they also make an assumption from other
languages that have only one implementation that implementations are
not permitted to vary as widely as Lisp implementations are permitted
to vary. Lisp implementations can be quite different, which is why
the standard is so important.
There is a responsibility that comes with teaching. Simply dumping
knowledge unfettered, without guidelines, is not something I do. I
recognize that others do it. And I like to sometimes at least
highlight why that might not be the approach the reader wants to take.
Nothing I said forces the OP to actually take my advice over yours.
Kent M Pitman <······@nhplace.com> writes:
> You also answered unconditionally without asking whether the person
> was a novice or an expert.
Fair nuff.
> There is a responsibility that comes with teaching. Simply dumping
> knowledge unfettered, without guidelines, is not something I do. I
> recognize that others do it.
Guilty as charged.
> And I like to sometimes at least highlight why that might not be the
> approach the reader wants to take. Nothing I said forces the OP to
> actually take my advice over yours.
Yeah, but you made my suggestion look dumb :-( But if you only know a
bit of Lisp (but not too much) it is actually very handy. If you want
to know when a function runs (even during compilation), then put in a
break point! Very nifty for debugging macros.
True I should have provided more context.
-russ
Russell McManus <···············@yahoo.com> writes:
> Yeah, but you made my suggestion look dumb :-(
I didn't mean to do that. Just dangerous (in the wrong hands). Dangerous
can be dumb or not, but it's definitely dumb if it's uninformed. People
do informed but dangerous things for good reasons all the time.
> But if you only know a
> bit of Lisp (but not too much) it is actually very handy. If you want
> to know when a function runs (even during compilation), then put in a
> break point! Very nifty for debugging macros.
Actually, I _do_ think this is a useful way to learn a great many things.
It just has to be done with extreme care. Having a lesson on the difference
between definition and implementation, understanding the difference between
learning "how things might be done", "what you should do", "what you can do",
is critical here. But given that you do constantly partition things into
the right bin, I completely agree that there are very fun and informative
things that one can do to explore Lisp using its introspective facilities.
Pinku Surana wrote:
> Suppose I write a compiler-macro such that (add constant constant) ==>
> constant. When will this macro get executed? What happens if the
> optimizer determines the constants later on, will it call my macro
> again?
>
> [forgive my Lisp syntax, I'm really a schemer]
>
> (let* ((x (+ 1 2)) (y (* x x)))
> (add x y))
>
> Given some suitably complicated program, the optimizer may run for
> quite a while before it discovers that x and y are constants. Will it
> then run my compiler-macro again? Or does it only run the macros once
> before the main optimizer runs?
This is implementation dependent. There is no guarantee that compiler
macros are used at all (although most CL implementations seem to do it).
You can only test this, or ask your vendor for documentation/help.
Pascal
--
2nd European Lisp and Scheme Workshop
July 26 - Glasgow, Scotland - co-located with ECOOP 2005
http://lisp-ecoop05.bknr.net/
> This is implementation dependent.
Does there *exist* a Lisp implementation that will run a compiler-macro
during or after optimization? I'll bet the answer is no. The standard
requires a minimal implementation, as Kent says, because anything more
would likely get in the implementors way.
I'm only asking out of curiosity, but I'm not curious enough to try all
implementations. :-)
Thanks.
"Pinku Surana" <·······@gmail.com> writes:
> > This is implementation dependent.
>
> Does there *exist* a Lisp implementation that will run a compiler-macro
> during or after optimization? I'll bet the answer is no.
I can easily imagine an implementation where compiler macros are
implemented AS an optimizer, and hence are semi-non-deterministic in
the sense that there may or may not be a specified ordering of
optimizers. (They may only have partial orders or none at all.)
> The standard requires a minimal implementation, as Kent says,
> because anything more would likely get in the implementors way.
The standard was explicitly permitted to allow either:
... (optimize (expand-compiler-macros form env) env)) ...
or
(defmacro define-compiler-macro (name bvl &body body)
(add-optimizer name #'(lambda ,bvl ,@body)))
...
(optimize form env) ;with no separate expand-compiler-macros
> I'm only asking out of curiosity, but I'm not curious enough to try all
> implementations. :-)
You couldn't anyway.
All implementations includes implementations not yet written. Even if
you could quantify over all extant implementations, one could be
written tomorrow to disagree.
> I can easily imagine an implementation where compiler macros are
> implemented AS an optimizer, and hence are semi-non-deterministic in
> the sense that there may or may not be a specified ordering of
> optimizers. (They may only have partial orders or none at all.)
Indeed, this is what I've built for my Scheme compiler. It seems to
work well enough.
Kent M Pitman wrote:
> I can easily imagine an implementation where compiler macros are
> implemented AS an optimizer, and hence are semi-non-deterministic in
> the sense that there may or may not be a specified ordering of
> optimizers. (They may only have partial orders or none at all.)
Kent -- You seem to use the term "optimizer" to denote some specific
entity that runs at some particular time during compilation. I'm
unfamiliar with this usage (LM?) and I can think of at least three
separate phases of typicaly CL compilation where optimization occurs.
However, the tmie when compiler-macros are executed (if the
implementation chooses to execute a particular compiler-macro) is
fairly well circumscribed by the ANS.
If you check the records, you can discover than JonL and I wrote the
first draft of the compiler-macro X3J13 action late one night during a
3-day X3J13 meeting, running right up against the committee's
self-imposed "No New Features!!!" deadline. IIRC we weren't real happy
to be standardizing a new API without any use history, but we felt that
establishing a hook for compiler-macrology was worth the risk.
With 16 years of hindsight, I believe the significant missing component
that would give compiler-macros the power they should have is access to
the environment. Compiler macros need access to type information, to
information about the extent of a cmoputed value. and other such things.
Unfortunately, in most implementations this information isn't available
-- at least, not fully available -- until later passes. Sorting out how
a programmer can specify "optimizers" that depend upon information that
(typically) isn't computed until later compilation phases remains an
unsolved problem.
"Later compilation phases" here means those phases where the compiler
is operating on internal data structures rather than rewritten Lisp
forms. All implementations I know of work that way, and I observe
wryly that it only gets in their way...
> All implementations includes implementations not yet written. Even if
> you could quantify over all extant implementations, one could be
> written tomorrow to disagree.
... except maybe for the one I haven't written yet.
"Steven M. Haflich" <···@alum.mit.edu> writes:
> Kent M Pitman wrote:
>
> > I can easily imagine an implementation where compiler macros are
> > implemented AS an optimizer, and hence are semi-non-deterministic in
> > the sense that there may or may not be a specified ordering of
> > optimizers. (They may only have partial orders or none at all.)
>
> Kent -- You seem to use the term "optimizer" to denote some specific
> entity that runs at some particular time during compilation.
Yes, the LispM had such.
> I'm unfamiliar with this usage (LM?)
Bingo. I wasn't trying to say all implementations do this. I was just
trying to establish a framework for understanding how things could vary
more than people were thinking.
> If you check the records, you can discover than JonL and I wrote the
> first draft of the compiler-macro X3J13 action late one night during a
> 3-day X3J13 meeting, running right up against the committee's
> self-imposed "No New Features!!!" deadline.
Heh... I recall a number of such efforts. I didn't mean to paint
myself as the only one doing lots of work on these things. I just
happen to remember it from my own point of view. Thanks for rounding
out the historical picture.
> IIRC we weren't real happy
> to be standardizing a new API without any use history, but we felt that
> establishing a hook for compiler-macrology was worth the risk.
No one was. But it WAS needed.
> With 16 years of hindsight, I believe the significant missing component
> that would give compiler-macros the power they should have is access to
> the environment.
Well, in fairness, we knew that then, too. We had all that
environment access stuff in there for a while (which showed up in
CLTL2, and then we backed out of later). As you surely recall (but
some reading may not know), we didn't flush that stuff because we
decided it was the wrong idea or there was no need, we flushed it
because it had exactly the problem you were worried about above when
you talked about last-minute design: it had too many bugs and we
weren't reach closure in fixing them. The hope had been that
implementors would complete our work and publish an interface
to that stuff later.
> Compiler macros need access to type information,
Some do. As you'll recall, one major concern was optimizing keyword
calling, and that can be done without type info.
> to
> information about the extent of a cmoputed value. and other such things.
> Unfortunately, in most implementations this information isn't available
> -- at least, not fully available -- until later passes. Sorting out how
> a programmer can specify "optimizers" that depend upon information that
> (typically) isn't computed until later compilation phases remains an
> unsolved problem.
Yes, I agree.
> "Later compilation phases" here means those phases where the compiler
> is operating on internal data structures rather than rewritten Lisp
> forms. All implementations I know of work that way, and I observe
> wryly that it only gets in their way...
>
> > All implementations includes implementations not yet written. Even if
> > you could quantify over all extant implementations, one could be
> > written tomorrow to disagree.
>
> ... except maybe for the one I haven't written yet.
Right. Let us know when it's ready.
"Pinku Surana" <·······@gmail.com> writes:
> (let* ((x (+ 1 2)) (y (* x x)))
> (add x y))
>
> Given some suitably complicated program, the optimizer may run for
> quite a while before it discovers that x and y are constants. Will it
> then run my compiler-macro again? Or does it only run the macros once
> before the main optimizer runs?
I believe that as it currently stands, you cannot expect help of a
compiler-macro definition for add to yield a constant out of the above
code.
The crucial point is not the compiler macro, but the function to
detect constants and the mean to obtain that constant value.
Whould you invoke (constantp form)
or (constantp form env)?
or a constant? detector of your own
inside your macro?
That alone already makes a lot of a difference.
After that, there a missing link in Common Lisp: you need
(constant-value form env) to get that constant value, so that you can
substitute the initial add form with the value of (+ (constant-value
first env) (constant-value second env)).
Sadly, constant-value is not (yet) in CL, and several pieces of
software suffer (witness the defconstant debate in SBCL during
cross-compilation: people typically put DEFCONSTANT inside EVAL-WHEN
when they want it during compilation. Yet the current value may still
be different from the value that'll get used after load-time (possibly
in another image).
And yet, I'm not even sure constant-value would be enough to solve a
large enough set of problems without raising other hairy issues.
Regards,
Jorg Hohle
Telekom/T-Systems Technology Center