Another one of my historical questions. I've found various discussion
of this issue in old posts but nothing that I could find addressed
this question: why was it that ' is defined in terms of the QUOTE
special operator but the implementation of ` is left open? As always,
I understand, that the answer might be, "because that's just the way
we did it" but if anyone recollects any more about the decision I'd be
interested to hear about it.
-Peter
--
Peter Seibel ·····@javamonkey.com
Lisp is the red pill. -- John Fraser, comp.lang.lisp
In article <··············@javamonkey.com>,
Peter Seibel <·····@javamonkey.com> wrote:
>Another one of my historical questions. I've found various discussion
>of this issue in old posts but nothing that I could find addressed
>this question: why was it that ' is defined in terms of the QUOTE
>special operator but the implementation of ` is left open? As always,
>I understand, that the answer might be, "because that's just the way
>we did it" but if anyone recollects any more about the decision I'd be
>interested to hear about it.
Because it doesn't *need* a special operator of its own. Adding a new
special operator requires changing the language implementation (Maclisp had
FEXPRs, which were functions that didn't evaluate their arguments, but they
couldn't be used to make the compiler treat the code specially, e.g. they
didn't have any way to modify or access the environment). Backquote was
originally a user-written reader-macro; all the operators it needed for its
expansion (QUOTE, CONS, LIST, LIST*, APPEND) were already in the language.
Some implementations use different operators than those I listed. They're
generally synonyms for these functions, but recognized specially by the
pretty-printer so that it can reconstruct the original backquote syntax.
Different implementations chose different internal symbol names for them,
and when Common Lisp was being defined there was no compelling reason to
force them all to a common mechanism. As far as users are concerned, all
that matters is the end result, not the internal details.
As a general rule, special operators should be kept to a minimum, since any
code-processing program needs to know about them and treat them specially.
If it's straightforward to accomplish something with a macro or function,
it shouldn't be a special operator.
--
Barry Margolin, ··············@level3.com
Level(3), Woburn, MA
*** DON'T SEND TECHNICAL QUESTIONS DIRECTLY TO ME, post them to newsgroups.
Please DON'T copy followups to me -- I'll assume it wasn't posted to the group.
Barry Margolin <··············@level3.com> writes:
> In article <··············@javamonkey.com>,
> Peter Seibel <·····@javamonkey.com> wrote:
> >Another one of my historical questions. I've found various discussion
> >of this issue in old posts but nothing that I could find addressed
> >this question: why was it that ' is defined in terms of the QUOTE
> >special operator but the implementation of ` is left open? As always,
> >I understand, that the answer might be, "because that's just the way
> >we did it" but if anyone recollects any more about the decision I'd be
> >interested to hear about it.
>
> Because it doesn't *need* a special operator of its own. Adding a
> new special operator requires changing the language implementation
> (Maclisp had FEXPRs, which were functions that didn't evaluate their
> arguments, but they couldn't be used to make the compiler treat the
> code specially, e.g. they didn't have any way to modify or access
> the environment). Backquote was originally a user-written
> reader-macro; all the operators it needed for its expansion (QUOTE,
> CONS, LIST, LIST*, APPEND) were already in the language.
Ah, of course.
> Some implementations use different operators than those I listed.
> They're generally synonyms for these functions, but recognized
> specially by the pretty-printer so that it can reconstruct the
> original backquote syntax. Different implementations chose different
> internal symbol names for them, and when Common Lisp was being
> defined there was no compelling reason to force them all to a common
> mechanism. As far as users are concerned, all that matters is the
> end result, not the internal details.
Hmmm. So the posts I did find on this topic when I googled mostly had
to do with people complaining that they couldn't write their own
portable pretty printers since there's no way to know what a given
backquoted expression is going to look like by the time you get it
back from the reader.
I.e. I always know that (first ''foo) is QUOTE but I don't know what
(first ``foo) is going to be. I'm not sure if that matters to anyone
other than pretty-printer authors.
> As a general rule, special operators should be kept to a minimum,
> since any code-processing program needs to know about them and treat
> them specially. If it's straightforward to accomplish something with
> a macro or function, it shouldn't be a special operator.
Right. That makes sense. But it does sort of leave the portable pretty
printer writers SOL doesn't it?
-Peter
--
Peter Seibel ·····@javamonkey.com
Lisp is the red pill. -- John Fraser, comp.lang.lisp
Peter Seibel <·····@javamonkey.com> wrote in message news:<··············@javamonkey.com>...
> I.e. I always know that (first ''foo) is QUOTE but I don't know what
> (first ``foo) is going to be. I'm not sure if that matters to anyone
> other than pretty-printer authors.
It matters to people who write macros that write code which has
backquote in it. Without a well-defined representation, bacquote
suffers from the Scheme disease: the target language of the backquote
is defined by the source code characters rather than an abstract data
structure. How ironic that Scheme actually defines the S-exp structure
for bacquote forms. :)
Now you can do some neat things with nested backquotes, but this is
insufficiently powerful; the only operations you have in a backquote
are unquoting and splicing unquoting. If you want something more
complicated, you put the logic into the expression that is unquoted.
Using nested backquotes, you can write code that writes backquotes,
but in some cases you run out of power. There are rare situations when
want to use the outer backquote to manipulate the inner backquote in
some way that is just not possible.
For example, suppose you want to conditionally insert an unquote into
a backquote. Depending on some input, you want to create either the
object `(foo ,@bar) or `(foo ,xyzzy) but without duplicating the
entire template for the two cases, of course! There is no way to
express your intent ``if this condition is true, I want a ,@bar in
here otherwise a ,xyzzy!'' because you are not even allowed to utter
the syntax ,@bar if it's not matched by a backquote. It has no
representation by itself, and inside a matching backquote, it is
totally opaque. You cannot portably ``cheat'' by writing ,@bar inside
some throwaway backquoet expression, and probe within the resulting
object to retrieve the inner object that corresponds to ,@bar.
···@ashi.footprints.net (Kaz Kylheku) writes:
> Peter Seibel <·····@javamonkey.com> wrote in message news:<··············@javamonkey.com>...
> > I.e. I always know that (first ''foo) is QUOTE but I don't know what
> > (first ``foo) is going to be. I'm not sure if that matters to anyone
> > other than pretty-printer authors.
>
> It matters to people who write macros that write code which has
> backquote in it. Without a well-defined representation, bacquote
> suffers from the Scheme disease: the target language of the
> backquote is defined by the source code characters rather than an
> abstract data structure. How ironic that Scheme actually defines the
> S-exp structure for bacquote forms. :)
Right; that's sort of what I was imagining but I couldn't think of any
specific examples of the kind of thing one would want to do. But the
gap between the reader syntax and the resulting data seemed a bit at
odds with The Lisp Way. I guess my original question was misphrased in
one sense: one doesn't really need more special operators to solve
these problems, just a standard representation for what you get when
Lisp reads a backquoted expression, which could presumably be a
BACKQUOTE macro with embedded UNQUOTE and UNQUOTE-SPLICING macros. I
suppose the trick is that to implement BACKQUOTE as a macro requires
being able to reliably walk code, etc.
ISTR you wrote a new backquote implemenation for CLISP a few months
back? Is it dependent on the internals of CLISP? If so, what would be
required (either in terms of programming of from the language
definition) to be able to write it portably?
-Peter
--
Peter Seibel ·····@javamonkey.com
Lisp is the red pill. -- John Fraser, comp.lang.lisp
Peter Seibel <·····@javamonkey.com> wrote in message news:<··············@javamonkey.com>...
> ···@ashi.footprints.net (Kaz Kylheku) writes:
>
> > Peter Seibel <·····@javamonkey.com> wrote in message news:<··············@javamonkey.com>...
>
> > > I.e. I always know that (first ''foo) is QUOTE but I don't know what
> > > (first ``foo) is going to be. I'm not sure if that matters to anyone
> > > other than pretty-printer authors.
> >
> > It matters to people who write macros that write code which has
> > backquote in it. Without a well-defined representation, bacquote
> > suffers from the Scheme disease: the target language of the
> > backquote is defined by the source code characters rather than an
> > abstract data structure. How ironic that Scheme actually defines the
> > S-exp structure for bacquote forms. :)
>
> Right; that's sort of what I was imagining but I couldn't think of any
> specific examples of the kind of thing one would want to do.
CLISP has internal functions called SYSTEM::ADD-BACKQUOTE and
SYSTEM::ADD-UNQUOTE which generates an unquote object. I recall that
they are used in the macrology of DEFSTRUCT to construct backquote
code which is later re-evaluated. In other words, some list that
contains a variable number of unquotes is generated and then spun into
a complete backquote.
This type of thing is useful because sometimes you want to intercept
the intermediate code that is generated by some macrology. If the
target language is the unexpanded backquote syntax (which can be
pretty-printed to the backquote read syntax) that intermediate code is
easier for a human being to understand.
> ISTR you wrote a new backquote implemenation for CLISP a few months
> back? Is it dependent on the internals of CLISP? If so, what would be
> required (either in terms of programming of from the language
> definition) to be able to write it portably?
It started as portable code, but then in the transition to CLISP, it
became a little bit chummy with the internals. Here are the main
issues that I ran into, in about the right order:
Firstly, the CLISP pretty printer had to be adjusted to grok the new
internal representation of backquotes. My original portable
implementation had a very quick and dirty pretty printing routine just
for debugging.
Secondly, certain other internal interfaces, notably the
aforementioned SYSTEM::ADD-UNQUOTE and SYSTEM::ADD-BACKQUOTE functions
had to be modified to work with the new material.
Third, there were issues with integrating it with the reader macros
for arrays, structures and other objects. When certain objects appear
subordinate to a backquote, unquotes are not allowed within them:
array notation, structs and others. Sam Steingold wanted these
situations to signal errors, so there had to be tight coupling between
the backquote reader macro and these other readers.
Fourth, some internals of my portable implementation had to be
rewritten because they used CL functions that are not available at the
bootstrap stage where backquotes are introduced into CLISP.
Lastly, a nasty ``bug'' showed up: the uncompiled compiler to emit
slightly different .FAS files from the compiled compiler, a condition
that is detected by ``make check''! Nothing actually broke; this is an
artificial constraint imposed by the project for maintaining the
sanity of the compiler. If you allow the FAS's to differ in any
detail, then real bugs could creep in undetected.
This happened because under the old backquote, multiple EQ
duplications of a backquote instance led to expansions that were EQ.
Whereas under the new backquote, EQ duplications of a backquote
evaluate to distinct, non-EQ expansions. This is a consequence of
doing the expansion at macroexpansion time rather than at read time.
Certain inlining optimizations in the compiler duplicate source code,
such as the optimizations that turn MAPCAR and its ilk into inlined
loops. When the closure body to the MAPCAR is produced by a backquote,
that backquote is duplicated.
The fix which was finally adopted was to change the compiler, so that
when it does those optimizations, it macro-expands the pieces that are
duplicated before sticking them into the output form.
Before that I wrote a quick and dirty fix whereby I cached the results
of backquote expansion in the source code itself, as a hidden third
parameter to the SYSTEM::BACKQUOTE macro. The caching would reduce the
expansions of EQ backquotes to be EQ, but at the cost of destructively
modifying the program!
In article <····························@posting.google.com>,
Kaz Kylheku <···@ashi.footprints.net> wrote:
>For example, suppose you want to conditionally insert an unquote into
>a backquote. Depending on some input, you want to create either the
>object `(foo ,@bar) or `(foo ,xyzzy) but without duplicating the
>entire template for the two cases, of course!
I think this is an unreasonable thing to attempt in the first place.
There's no requirement that programmers use backquote. Instead of `(foo
,bar) they could write (list 'foo bar) (as we did in the days before
backquote), or they could write (my-function bar), where MY-FUNCTION is
defined as:
(defun my-function (something)
`(foo ,something))
Do you really want to write programs that only work properly if a
particular syntax was used to express the data structure?
--
Barry Margolin, ··············@level3.com
Level(3), Woburn, MA
*** DON'T SEND TECHNICAL QUESTIONS DIRECTLY TO ME, post them to newsgroups.
Please DON'T copy followups to me -- I'll assume it wasn't posted to the group.
On 9 Oct 2003 10:26:12 -0700, Kaz Kylheku wrote:
> Peter Seibel <·····@javamonkey.com> wrote in message news:<··············@javamonkey.com>...
>> I.e. I always know that (first ''foo) is QUOTE but I don't know what
>> (first ``foo) is going to be. I'm not sure if that matters to anyone
>> other than pretty-printer authors.
> It matters to people who write macros that write code which has
> backquote in it.
It shouldn't. You don't "write code which has backquote in it", you
write code which has s-expressions in it, and if you care where those
expressions came from (e.g., whether I wrote `(a ,b c) or (list 'a b 'c)),
ISTM you're doing something very badly wrong.
You can't tell whether I write 'a or (quote a) or (intern "A") or any
number of other things, either.
> For example, suppose you want to conditionally insert an unquote into
> a backquote. Depending on some input, you want to create either the
> object `(foo ,@bar) or `(foo ,xyzzy) but without duplicating the
> entire template for the two cases, of course! There is no way to
> express your intent ``if this condition is true, I want a ,@bar in
> here otherwise a ,xyzzy!'' because you are not even allowed to utter
`(foo ,@(if <condition> bar (list xyzzy)))
--
Cogito ergo I'm right and you're wrong. -- Blair Houghton
(setq reply-to
(concatenate 'string "Paul Foley " "<mycroft" '(··@) "actrix.gen.nz>"))
Paul Foley <·······@actrix.gen.nz> wrote in message news:<··············@mycroft.actrix.gen.nz>...
> On 9 Oct 2003 10:26:12 -0700, Kaz Kylheku wrote:
>
> > Peter Seibel <·····@javamonkey.com> wrote in message news:<··············@javamonkey.com>...
> >> I.e. I always know that (first ''foo) is QUOTE but I don't know what
> >> (first ``foo) is going to be. I'm not sure if that matters to anyone
> >> other than pretty-printer authors.
>
> > It matters to people who write macros that write code which has
> > backquote in it.
>
> It shouldn't. You don't "write code which has backquote in it", you
> write code which has s-expressions in it, and if you care where those
> expressions came from (e.g., whether I wrote `(a ,b c) or (list 'a b 'c)),
> ISTM you're doing something very badly wrong.
You write code that has backquotes in in whenever you nest them:
(defmacro macro-writer (name)
`(defmacro ,name ()
`(...)))
> You can't tell whether I write 'a or (quote a) or (intern "A") or any
> number of other things, either.
You also can't tell whether I prepared machine code as a hexadecimal
string and had it punched directly into memory. That doesn't mean I
want to express myself that way. :)
I may care whether you wrote `(a ,b c) or (list 'a b 'c) when I need
to intercept what your code is doing, and I want it to be readable.
When large, complex backquotes are expanded into the equivalent list
construction code, it often looks like tossed salad, even when the
code is nicely condensed and optimized to take advantage of the slick
idioms that a programmer would use.
It would be nice if I could invoke MACROEXPAND no the above
MACRO-WRITER and see something that looks like a normal DEFMACRO form
with backquote in it, pretty printed properly and all that. Rather
than a DEFMACRO with a whole bunch of nested LIST, APPEND, LIST* and
whatnot.
In article <····························@posting.google.com>,
Kaz Kylheku <···@ashi.footprints.net> wrote:
>Paul Foley <·······@actrix.gen.nz> wrote in message
>news:<··············@mycroft.actrix.gen.nz>...
>> You can't tell whether I write 'a or (quote a) or (intern "A") or any
>> number of other things, either.
>
>You also can't tell whether I prepared machine code as a hexadecimal
>string and had it punched directly into memory. That doesn't mean I
>want to express myself that way. :)
>
>I may care whether you wrote `(a ,b c) or (list 'a b 'c) when I need
>to intercept what your code is doing, and I want it to be readable.
The language was designed to support processing the data, not the input
syntax. Since those are equivalent ways of expressing the same data
structure, it's expected that any program that deals with them will treat
them equivalently.
>It would be nice if I could invoke MACROEXPAND no the above
>MACRO-WRITER and see something that looks like a normal DEFMACRO form
>with backquote in it, pretty printed properly and all that. Rather
>than a DEFMACRO with a whole bunch of nested LIST, APPEND, LIST* and
>whatnot.
Since the pretty-printer is supplied by the implementation, it can
recognize the implementation-dependent operators that are used in the
backquote expansion, and display backquote syntax.
--
Barry Margolin, ··············@level3.com
Level(3), Woburn, MA
*** DON'T SEND TECHNICAL QUESTIONS DIRECTLY TO ME, post them to newsgroups.
Please DON'T copy followups to me -- I'll assume it wasn't posted to the group.
In article <················@news.level3.com>, Barry Margolin
<··············@level3.com> wrote:
> >I may care whether you wrote `(a ,b c) or (list 'a b 'c) when I need
> >to intercept what your code is doing, and I want it to be readable.
>
> The language was designed to support processing the data, not the input
> syntax. Since those are equivalent ways of expressing the same data
> structure, it's expected that any program that deals with them will treat
> them equivalently.
Let me try to restate what I think the OP was getting at:
I may care whether you wrote (list 'a 'b 'c) or (cons 'a (cons b '(c)))
(note that the semantics of these two renditions of `(a ,b c) are
different) and it would be nice if CL specified a "canonical" translation
for backquote forms like Scheme does.
E.
Barry Margolin wrote:
> Since the pretty-printer is supplied by the implementation, it can
> recognize the implementation-dependent operators that are used in the
> backquote expansion, and display backquote syntax.
>
Actually, this isn't exactly true. The semantics of backquote is not
completely determined by the ANSI spec. Different implementations are
allowed to interpret the same backquote expression in different ways. I
quote from section 2.4.6 of the Holy Scripture:
------
An implementation is free to interpret a backquoted form F1 as any form
F2 that, when evaluated, will produce a result that is the same under
equal as the result implied by the above definition, provided that the
side-effect behavior of the substitute form F2 is also consistent with
the description given above. The constructed copy of the template might
or might not share list structure with the template itself.
-------
Hence a printer that turns all S-expression-building forms into
backquote forms may produce an expression that has a different value if
read by a different implementation. Granted, the situations where this
matters are rare. But there are also examples like this:
(mapcan (lambda (x)
(cond ((prop x) (list '**))
(t (list `(foo ,x)))))
ll)
I like to discriminate between forms that evaluate to immutable
S-expressions and forms that evaluate to destructible lists, reserving
backquote for the former. So I would never write `((foo ,x)) in
the lambda-body. I don't want a pretty-printer overriding my stylistic
decisions when it prints some of my code back at me. And, of course, if
it prints (list '**) as `(**) then the printed code is wrong.
Face it, the absence of a standard definition of what backquote
expressions expand into is a blemish on Lisp. As evidence I'll cite the
following fact: as far as I know, every serious implementation of Lisp
_does_ have a special internal representation for backquote, which the
pretty printer relies on. What we don't have is a _portable_ internal
representation.
--
-- Drew McDermott
Yale Computer Science Department
Barry Margolin <··············@level3.com> wrote in message news:<················@news.level3.com>...
> In article <····························@posting.google.com>,
> Kaz Kylheku <···@ashi.footprints.net> wrote:
> >It would be nice if I could invoke MACROEXPAND no the above
> >MACRO-WRITER and see something that looks like a normal DEFMACRO form
> >with backquote in it, pretty printed properly and all that. Rather
> >than a DEFMACRO with a whole bunch of nested LIST, APPEND, LIST* and
> >whatnot.
>
> Since the pretty-printer is supplied by the implementation, it can
> recognize the implementation-dependent operators that are used in the
> backquote expansion, and display backquote syntax.
In the case I'm thinking of, the author, for lack of having a
backquote target language which his code generator could use, resorted
generating list construction code. That code will not likely be turned
into backquote syntax by the pretty printer! A target language for
backquotes would let that author generate backquote syntax which could
be intercepted and printed as backquote syntax.
Barry Margolin wrote:
> Because it doesn't *need* a special operator of its own.
I'll admit the following argument runs the danger of being somethi8ng
of a troll, but it is a troll with a technical point.
cl:quote doesn't need to be a special operator, either. It could have
been a macro:
(defmacro quote (object)
(list 'svref (vector object) 0))
We'd leave it to the sufficiently-smart compiler to eliminate the
runtime inefficiency.
While I certainly agree with the conclusion that quote ought be a
special form, and that backquote ought not, I'm suspicious of
arguments that rely on specialformitude because the set of
special forms is like the set of jello flavors available
to be nailed to a tree.
Steven M. Haflich <·················@alum.mit.edu> wrote:
+---------------
| cl:quote doesn't need to be a special operator, either.
| It could have been a macro:
|
| (defmacro quote (object)
| (list 'svref (vector object) 0))
+---------------
Uh... Got a little problem with bootstrapping there. How do you
get that 'SVREF in there in the first place without turtling
all the way down? ;-}
Just kidding. That can be fixed up easily enough:
(defmacro quote (object)
(list (load-time-value (intern "SVREF" "COMMON-LISP"))
(vector object)
0))
Here's another way to do it which probably executes faster but
may (or may not) consume more space:
(defmacro quote (object)
(let ((symbol (gensym)))
(setf (symbol-value symbol) object)
symbol))
-Rob
-----
Rob Warnock <····@rpw3.org>
627 26th Avenue <URL:http://rpw3.org/>
San Mateo, CA 94403 (650)572-2607
Peter Seibel <·····@javamonkey.com> writes:
> Another one of my historical questions. I've found various discussion
> of this issue in old posts but nothing that I could find addressed
> this question: why was it that ' is defined in terms of the QUOTE
> special operator but the implementation of ` is left open? As always,
> I understand, that the answer might be, "because that's just the way
> we did it" but if anyone recollects any more about the decision I'd be
> interested to hear about it.
As I recall, the reason was to make sure you didn't confuse yourself
by consing in a piece of backquote that was created in some other
context. Effectively, I think it was a hygiene concern. (I'm not sure
it was really necessary, incidentally. It might have been a wholly
imagined problem. Scheme doesn't seem to have run into difficulties
with it.)
And it also wanted to tell you if you got your commas too deep, etc.,
as early as possible. This WAS a real concern, especially when backquote
was new and people were not used to it. It was commonplace in the early
systems for people to put too commas too many deep and to end up with
+internal-/,-marker (or whatever it was called--something like that)
in their final application program data. I think there was the worry
that these would be left over by accident and then (as mentioned in the
previous paragraph) consed accidentally into other things where they'd
have unwanted active effect.
Keep in mind, too, that at the time backquote was created, back in
Maclisp days, there were several competing implementations, and it was
the implementation that dictated the semantics, not vice versa. It
was not like we had a community-approved design and then went seeking
implementations.
So to some extent, it was just arbitrary that the implementation that
got adopted happened to do readtime resolution of these things.