Hi,
I'm stuck in a compile-time macroexpansion problem.
I have a macro, which as a side-effect of macroexpansion, gathers
information. I want some of the gathered information to flow back into
generated code (at another place).
I wish an expansion to look similar to this
(setf (zeilenanalyse 'test)
(list
(function (lambda (analyse ...hidden-vars)
(when (COLUMN-OP protocol = "http")
(incf (analyse-count analyse) (COLUMN bytes integer)))))
'(PROTOCOL BYTES)))
where PROTOCOL and BYTES are discovered as a side-effect of the
macroexpansion of the COLUMN[-OP] macros during interpretion or
compilation.
I don't feel like this requires code walker. It seems to me like
having macros gather information during expansion is all there is
needed, while code-walkers are a major undertaking and cause other
problems.
The application domain is about performing analyses on files, each
line of which contains a number of columns. A typical example is a
comma-separated-values (.csv) file.
The main feature is to know statically, what are the columns that an
analysis operates upon. Optimized code can be produced knowing that,
as well as early compatibility-checking.
For the interested, the above COLUMN-OP can expand into
(string= .buffer. "http"
:start1 (aref .start-positions. 4) ;inlined column index
:end1 (aref .stop-positions. 4)) ;of column "protocol"
which is the fastest I could devise for this kind of application.
The huge files are read in chunks of 128KB.
A demo analysis looks like this:
(defclass test (zeilenanalyse)
((count :accessor analyse-count :initform 0)))
(define-zeilenanalyse ((analyse test))
"Sum up HTTP trafic (in bytes)"
(when (COLUMN-OP protocol = "http")
(incf (analyse-count analyse) (COLUMN bytes integer))))
You see how my approach is to define a tiny domain specific language
(DSL) which embeds into classic Lisp code for all control logic. The
DSL provides two macros: COLUMN and COLUMN-OP, as seen above.
COLUMN/-OP expand using other macros, as follows:
(defvar *columns-used* '())
(defmacro column-index (column)
(check-type column symbol)
(pushnew column *columns-used*)
;; Once we know the column's position, we index it directly
(if *precompile*
(or (gethash column *column-positions*)
(error "Unknown column ~S" column))
`(or (gethash ',column *column-positions*)
(error "Unknown column ~S" ',column))))
So far, my macro barely works as follows, but I'm having problems
with compile-time issues.
(defmacro define-zeilenanalyse (((var class)) &body body)
(check-type var symbol)
(setq *columns-used* nil) ; side-effect compiler environment
(setq *precompile* nil)
ronment
;;TODO does not work in compiled mode: need *columns-used* from
;; compiler environment
(let ((lambda-form `(lambda (,var .line. .start-positions.
.stop-positions.) ,@body)))
`(eval-when (load compile eval)
(let ((*precompile* nil)
(*columns-used* '())) ; superfluous for compilation
(format *trace-output* "Columns for ~S so far: ~S~%"
',class *columns-used*)
(setf (gethash ',class *analyse-funktionen*)
(list (function ,lambda-form)
',lambda-form
*columns-used*))
(format *trace-output* "Columns now: ~S~%" *columns-used*)
))))
I.e. it generates code like
(setf (zeilenanalyse 'test)
(list
#'(lambda (analyse ...)
(when (COLUMN-OP protocol = "http")
(incf (analyse-count analyse) (COLUMN bytes integer))))
*colunms-used*))
but this expects *columns-used* to have been set, which does not
happen when loading a compiled file. It would be better to replace
this by the actual columns.
How can I achieve this? (doubly-nested backquotes??)
Some years ago, I had a similar setting. But it was easy because the
information gathered as a side-effect of macroexpansion went into
another file (where another macro put it in place). I then used
defsystem to maintain dependencies. That worked very well. The above
does not.
Thanks for your help,
Jorg Hohle
Telekom/T-Systems Technology Center
From: Duane Rettig
Subject: Re: Help with eval/compile-time macroexpansion
Date:
Message-ID: <4adexqd7e.fsf@beta.franz.com>
Joerg Hoehle <······@users.sourceforge.net> writes:
> Hi,
>
> I'm stuck in a compile-time macroexpansion problem.
[ ... ]
> So far, my macro barely works as follows, but I'm having problems
> with compile-time issues.
> (defmacro define-zeilenanalyse (((var class)) &body body)
> (check-type var symbol)
> (setq *columns-used* nil) ; side-effect compiler environment
> (setq *precompile* nil)
> ;;TODO does not work in compiled mode: need *columns-used* from
> ;; compiler environment
> (let ((lambda-form `(lambda (,var .line. .start-positions.
=====^
> .stop-positions.) ,@body)))
> `(eval-when (load compile eval)
===========^
> (let ((*precompile* nil)
> (*columns-used* '())) ; superfluous for compilation
I only looked at this cursorily, so this may not be the only problem.
Note that this eval-when is not at top-level, because it is within
a let form. So it is likely not doing what you expected it to do.
See
http://www.franz.com/support/documentation/6.2/ansicl/subsubse/processi.htm
and especially note items 1, 2, 3, and 4, and the very last paragraph.
--
Duane Rettig ·····@franz.com Franz Inc. http://www.franz.com/
555 12th St., Suite 1450 http://www.555citycenter.com/
Oakland, Ca. 94607 Phone: (510) 452-2000; Fax: (510) 452-0182
In article <·············@beta.franz.com>,
Duane Rettig <·····@franz.com> wrote:
>Joerg Hoehle <······@users.sourceforge.net> writes:
>
>> Hi,
>>
>> I'm stuck in a compile-time macroexpansion problem.
>
> [ ... ]
>
>> So far, my macro barely works as follows, but I'm having problems
>> with compile-time issues.
>> (defmacro define-zeilenanalyse (((var class)) &body body)
>> (check-type var symbol)
>> (setq *columns-used* nil) ; side-effect compiler environment
>> (setq *precompile* nil)
>> ;;TODO does not work in compiled mode: need *columns-used* from
>> ;; compiler environment
>> (let ((lambda-form `(lambda (,var .line. .start-positions.
>=====^
>> .stop-positions.) ,@body)))
>> `(eval-when (load compile eval)
>===========^
>> (let ((*precompile* nil)
>> (*columns-used* '())) ; superfluous for compilation
>
>I only looked at this cursorily, so this may not be the only problem.
>Note that this eval-when is not at top-level, because it is within
>a let form. So it is likely not doing what you expected it to do.
The EVAL-WHEN is in the *expansion*, and it's not within the LET form when
the expansion is processed (unless he invokes the macro from within a LET
form).
--
Barry Margolin, ··············@level3.com
Genuity Managed Services, a Level(3) Company, Woburn, MA
*** DON'T SEND TECHNICAL QUESTIONS DIRECTLY TO ME, post them to newsgroups.
Please DON'T copy followups to me -- I'll assume it wasn't posted to the group.
From: Duane Rettig
Subject: Re: Help with eval/compile-time macroexpansion
Date:
Message-ID: <465plqa4d.fsf@beta.franz.com>
Barry Margolin <··············@level3.com> writes:
> In article <·············@beta.franz.com>,
> Duane Rettig <·····@franz.com> wrote:
> >Joerg Hoehle <······@users.sourceforge.net> writes:
> >
> >> Hi,
> >>
> >> I'm stuck in a compile-time macroexpansion problem.
> >
> > [ ... ]
> >
> >> So far, my macro barely works as follows, but I'm having problems
> >> with compile-time issues.
> >> (defmacro define-zeilenanalyse (((var class)) &body body)
> >> (check-type var symbol)
> >> (setq *columns-used* nil) ; side-effect compiler environment
> >> (setq *precompile* nil)
> >> ;;TODO does not work in compiled mode: need *columns-used* from
> >> ;; compiler environment
> >> (let ((lambda-form `(lambda (,var .line. .start-positions.
> >=====^
> >> .stop-positions.) ,@body)))
> >> `(eval-when (load compile eval)
> >===========^
> >> (let ((*precompile* nil)
> >> (*columns-used* '())) ; superfluous for compilation
> >
> >I only looked at this cursorily, so this may not be the only problem.
> >Note that this eval-when is not at top-level, because it is within
> >a let form. So it is likely not doing what you expected it to do.
>
> The EVAL-WHEN is in the *expansion*, and it's not within the LET form when
> the expansion is processed (unless he invokes the macro from within a LET
> form).
Oops. Well, I _told_ you it was a cursory look :-)
--
Duane Rettig ·····@franz.com Franz Inc. http://www.franz.com/
555 12th St., Suite 1450 http://www.555citycenter.com/
Oakland, Ca. 94607 Phone: (510) 452-2000; Fax: (510) 452-0182
In article <·············@dont.t-systems.UCE.spam.no.com>,
Joerg Hoehle <······@users.sourceforge.net> wrote:
>COLUMN/-OP expand using other macros, as follows:
>(defvar *columns-used* '())
This needs to be in an EVAL-WHEN so that the variable will be available at
compile time.
>(defmacro column-index (column)
> (check-type column symbol)
> (pushnew column *columns-used*)
> ;; Once we know the column's position, we index it directly
> (if *precompile*
> (or (gethash column *column-positions*)
> (error "Unknown column ~S" column))
> `(or (gethash ',column *column-positions*)
> (error "Unknown column ~S" ',column))))
This macro doesn't seem to be used code you posted. But if it will be used
at compile time, it also needs to be in an EVAL-WHEN.
>So far, my macro barely works as follows, but I'm having problems
>with compile-time issues.
>(defmacro define-zeilenanalyse (((var class)) &body body)
> (check-type var symbol)
> (setq *columns-used* nil) ; side-effect compiler environment
> (setq *precompile* nil)
>ronment
> ;;TODO does not work in compiled mode: need *columns-used* from
> ;; compiler environment
> (let ((lambda-form `(lambda (,var .line. .start-positions.
> .stop-positions.) ,@body)))
> `(eval-when (load compile eval)
> (let ((*precompile* nil)
> (*columns-used* '())) ; superfluous for compilation
> (format *trace-output* "Columns for ~S so far: ~S~%"
> ',class *columns-used*)
> (setf (gethash ',class *analyse-funktionen*)
> (list (function ,lambda-form)
> ',lambda-form
> *columns-used*))
> (format *trace-output* "Columns now: ~S~%" *columns-used*)
>))))
>I.e. it generates code like
>(setf (zeilenanalyse 'test)
> (list
> #'(lambda (analyse ...)
> (when (COLUMN-OP protocol = "http")
> (incf (analyse-count analyse) (COLUMN bytes integer))))
> *colunms-used*))
I don't see how you get this expansion from the above macro. You're
leaving out too much information, making it difficult for us to understand
what you're trying to do.
>but this expects *columns-used* to have been set, which does not
>happen when loading a compiled file. It would be better to replace
>this by the actual columns.
If it expects *COLUMNS-USED* to be set when you load the file, then the
expansion has to perform a SETQ assignment, not a LET-binding.
--
Barry Margolin, ··············@level3.com
Genuity Managed Services, a Level(3) Company, Woburn, MA
*** DON'T SEND TECHNICAL QUESTIONS DIRECTLY TO ME, post them to newsgroups.
Please DON'T copy followups to me -- I'll assume it wasn't posted to the group.
Joerg Hoehle <······@users.sourceforge.net> wrote in message news:<·············@dont.t-systems.UCE.spam.no.com>...
> Hi,
>
> I'm stuck in a compile-time macroexpansion problem.
>
> I have a macro, which as a side-effect of macroexpansion, gathers
> information. I want some of the gathered information to flow back into
> generated code (at another place).
Deja vu! Look for the Feb 27th thread that has "COMPILER-LET" in the
subject line. What you seem to want is local macros that stash some
information at expansion time to be used by other macros later in the
same expansion job.
You can use the SYMBOL-VALUE of a generated symbol as the stash where
you put the information, along these lines:
(defmacro gather-recall (&rest forms)
(let ((stash (gensym)))
(setf (symbol-value stash) nil)
`(macrolet ((gather (item)
(print 'gathering)
(push item (symbol-value ',stash))
(values))
(recall-it ()
(symbol-value ',stash)))
(symbol-macrolet ((recall (recall-it)))
,@forms))))
The macro allocates a symbol and initializes its value slot to NIL.
Then the forms that are passed in are wrapped in some local macros, so
you can do, e.g.
(gather-recall (gather 1) (gather 2) (gather vector) recall)
=> #(2 1)
The macro pushed the objects 1, 2 and VECTOR onto a list to produce
(VECTOR 2 1) and then produced that form as its expansion thanks to
the recall.
The indirection through RECALL-IT is necessary to obtain a
macroexpansion-time evaluation of the (SYMBOL-VALUE ...) form, and a
substitution of its result in place of the RECALL symbol macro.
We are depending on left-to-right order of expansion of the subforms.
If (gather vector) is expanded before (gather 2) and (gather 1) it
won't work! Or if recall is expanded first, we will get NIL, the
SYMBOL-VALUE of the gensym at that time.
> I don't feel like this requires code walker.
It does, but not one you have to write yourself! :) You just need
local macros to hook into your Lisp system's code walker.
···@ashi.footprints.net (Kaz Kylheku) writes:
> Joerg Hoehle <······@users.sourceforge.net> wrote in message news:<·············@dont.t-systems.UCE.spam.no.com>...
> > Hi,
> >
> > I'm stuck in a compile-time macroexpansion problem.
> >
> > I have a macro, which as a side-effect of macroexpansion, gathers
> > information. I want some of the gathered information to flow back into
> > generated code (at another place).
>
> Deja vu! Look for the Feb 27th thread that has "COMPILER-LET" in the
> subject line. What you seem to want is local macros that stash some
> information at expansion time to be used by other macros later in the
> same expansion job.
>
> You can use the SYMBOL-VALUE of a generated symbol as the stash where
> you put the information, along these lines:
>
> (defmacro gather-recall (&rest forms)
> (let ((stash (gensym)))
> (setf (symbol-value stash) nil)
> `(macrolet ((gather (item)
> (print 'gathering)
> (push item (symbol-value ',stash))
> (values))
> (recall-it ()
> (symbol-value ',stash)))
> (symbol-macrolet ((recall (recall-it)))
> ,@forms))))
>
> The macro allocates a symbol and initializes its value slot to NIL.
This is really bad form.
The symbol value might turn out (through symbol inheritance) to be
something you should not be affecting.
Better to at least use, e.g., (get ',stash 'gather-recall-cell)
rather than (symbol-value ',stash).
Incidentally, that will be pre-initialized to NIL since GET defaults to
NIL when a cell doesn't exist.
However, this whole design has the problem that, nested calls might not
do as you expect. There is nothing cleaning up these messes after.
Doing something 'correct' is non-trivial, I think, but doing something
incorrect is not generally a good substitute.
From: Kalle Olavi Niemitalo
Subject: Re: Help with eval/compile-time macroexpansion
Date:
Message-ID: <8765pj5qcx.fsf@Astalo.kon.iki.fi>
Kent M Pitman <······@world.std.com> writes:
> The symbol value might turn out (through symbol inheritance) to be
> something you should not be affecting.
I don't understand this. The macro makes the symbol with GENSYM
and never interns it in a package. How could anything inherit it?
Kalle Olavi Niemitalo <···@iki.fi> writes:
> Kent M Pitman <······@world.std.com> writes:
>
> > The symbol value might turn out (through symbol inheritance) to be
> > something you should not be affecting.
>
> I don't understand this.
(Probably because you assume I'm not making an error. ;)
> The macro makes the symbol with GENSYM and never interns it in a
> package. How could anything inherit it?
Ah, so it does. (I was thinking it did something else and only skimmed
the code.) Yes, this will work a bit better. It won't get screwed up
on nesting/cleanup issues. But...
I don't think there's a guarantee of left-to-right expansion, though,
nor even expansion at all (in some interpreted cases, in the case of compiled
code involving unreachable code, etc).
So I'm still a bit dubious about this macro. There are some Lisps
that, in interpreted code, expand macros only as they are executed and
do not expand those that are not executed. (Allegro might be one such,
if I recall correctly.) Does this run in interpreted Allegro?
From: Kalle Olavi Niemitalo
Subject: Re: Help with eval/compile-time macroexpansion
Date:
Message-ID: <87ptnqj2ba.fsf@Astalo.kon.iki.fi>
Kent M Pitman <······@world.std.com> writes:
> Kalle Olavi Niemitalo <···@iki.fi> writes:
>
> > I don't understand this.
>
> (Probably because you assume I'm not making an error. ;)
(I suspected there might be other kinds of "symbol inheritance"
than what happens between packages.)
> I don't think there's a guarantee of left-to-right expansion, though,
> nor even expansion at all (in some interpreted cases, in the case of compiled
> code involving unreachable code, etc).
I'm not sure unreachable code would be a problem. I consider
(when nil (foo)) and (progn #+(or) (foo) nil) pretty much
equivalent; if the code truly cannot be reached, then it should
not matter what it does or contains. This also holds in the
original poster's case, where he was going to use the collected
values to decide which columns to extract from input: if only
unreachable code uses some columns, then it is OK to skip them.
The interpreter sounds more difficult. :-( I suppose the
code could just blindly extract all fields if it is run
interpreted. The function would have to check whether it has
been minimally compiled (as in 3.2.2.2); I presume it could do
this with COMPILED-FUNCTION-P.
Out-of-order or repeated expansion of the GATHER invocations
should not be problems either, if the result is treated just as a
set. However, there seems to be no way to guarantee that RECALL
is expanded last. :-(
···@ashi.footprints.net (Kaz Kylheku) writes:
> > I have a macro, which as a side-effect of macroexpansion, gathers
> > information. I want some of the gathered information to flow back into
> > generated code (at another place).
> Deja vu! Look for the Feb 27th thread that has "COMPILER-LET" in the
Great!
> (defmacro gather-recall (&rest forms)
> (let ((stash (gensym)))
> (setf (symbol-value stash) nil)
> `(macrolet ((gather (item)
> (print 'gathering)
> (push item (symbol-value ',stash))
> (values))
> (recall-it ()
> (symbol-value ',stash)))
> (symbol-macrolet ((recall (recall-it)))
> ,@forms))))
>(gather-recall (gather 1) (gather 2) (gather vector) recall)
Why couldn't this be simplified to:
(defmacro gather-recall (&rest forms)
(let ((stash (gensym)))
(setf (symbol-value stash) nil)
`(macrolet ((gather (item)
(print 'gathering)
(push item (symbol-value ',stash))
(values))
(recall-it ()
(symbol-value ',stash)))
,@forms)))
(gather-recall (gather 1) (gather 2) (gather vector) (recall-it))
> (gather-recall (gather 1) (gather 2) (gather vector) recall)
> => #(2 1)
Curiously, it yields #(2 1 1) in clisp-2.28. But I believe newer
CLISP's dont do that anymore, because I had a bug [678194] fixed which
I believe is involved here (disallow macroexpansion of possible
declare form).
Curiously, my simplified version of your example, not using
symbol-macrolet, generates #(2 1) in CLISP-2.28. Go figure.
Based upon your model, I tried to use the following, which now works
in CLISP. However, in CormanLisp, (used-columns) expands to ().
(defmacro define-zeilenanalyse (((var class)) &body body)
(check-type var symbol)
(let ((lambda-form `(lambda (,var .line. .start-positions. .stop-positions.) ,@body)))
(setq *columns-used* '())
`(macrolet ((used-columns () (list 'QUOTE *columns-used*)))
(setf (gethash ',class *analyse-funktionen*)
(list (used-columns) (function ,lambda-form) ',lambda-form (used-columns))))))
; ^ test right-to-left and left-to-right ^
(define-zeilenanalyse ((analyse test))
(print (column action)))
I don't need a hidden 'stash' symbol, since *columns-used* is there for
exactly that purpose.
Now I also know why I had the vague impression that Christopher Browne's
reply to another post of mine in 1999 was sketching a path to a solution
<URL http://groups.google.com/groups?hl=en&lr=&ie=UTF-8&selm=87vh8hk0dj.fsf%402xtreme.net&rnum=8>
He made use of macrolet and double backquotes. In the above, I
refrained from using double backquotes (yuk!) and generated a QUOTE
form by hand. Thanks again for the hint and explanation!
I understand Kent Pitman's concern for left-to-right evaluation
order. Indeed, my macro above does not work in Corman Lisp. Maybe I
oversimplified, because the two forms of the GATHER macros work in
CormanLisp. OTOH, maybe the reason is that my macro depends on the
macroexpansion inside a (FUNCTION (lambda ...)) form, which seems to
happen later than the expansion of the nearby recall form.
Indeed, with CormanLisp
(gather-recall #'(lambda (x) (gather 1)) (gather vector) recall)
yields the empty vector #().
I believe it's perfectly legal for CormanLisp to exhibit this behaviour.
The core of the expansion that I hope for is modeled by:
(store-away 'test
(function (lambda (...) ... code using (column X) ...))
:columns-used '(X))
But maybe this cannot work with CormanLisp, and I cannot expect it to
work in this way.
So I'm left wondering whether one can portably hope for a working
solution,
a) perhaps by introducing additional levels of nesting so as to
enforce order of compilation (like Haskell programmers sometimes need
to do with evaluation),
b) or whether to work around the issue by using two separate forms in
the source code, like:
(define-analyse ((analyse test)) ...) -> generate #'(lambda form)
(record-columns test) -> store columns computed previously, which is
more error-prone (copy&paste => wrong mapping).
Still, the issue is interesting. I can relate it to Kenny Tilton's
cells and his usage of the self macro therein.
Regards,
Jorg Hohle
Telekom/T-Systems Technology Center
Hi again,
thanks for all the answers so far. Here are my comments on side-topics
that arose in the anwsers.
To Barry Margolin: sorry for not posting enough information. I was
afraid my post would become to long already. Here's the missing chain:
The macros COLUMN and COLUMN-OP themselves expand to a form using the
macro COLUMN-INDEX. Please allow me not to show the complete code,
because there are additional layers of macros expanding to other
macros, but here's a simplified model:
(defmacro column (column &optional (kind 'string) (default nil defaulting))
(check-type column symbol)
`(%with-column-index ,column ;let ((.index. (column-index ,column)))
,(ecase kind
(string
`(subseq
.line.
(%column-start ,column)
(%column-end ,column))))))
(defmacro %column-start (column)
(check-type column symbol)
`(aref .start-positions.
,(if *precompile*
`(column-index ,column)
'.index.)))
(eval-when (compile also)) was not the problem here. I know how to
deal with this issue by having (defvar *columns-used* ()) and all
those macros lay in a file separate from the forms using them (and
load the former prior to compiling the latter). That's a clean,
proper and reliable approach.
I could keep everything in a single file, but I dislike the added
verbosity of all those eval-when that this would require in the source
code, for little benefit over the multiple-file solution.
Incidentally, in case somebody doesn't like the hidden macros and
especially the hidden variables like .line., .index, .start-positions.
I *know* how to get rid of these: Instead of lots of defmacros, I'd
write a single one and plenty of MACROLET inside it and a few gensym
(or use that rebinding macro). But I don't want that.
What's the problem?
It would make my code more difficult to test portably and to modify.
I feel the reason related to why IIRC Kent himself some time ago
advocated use of DEFUN or DEFVAR etc. over lots of nested FLET/LABELS
as well as (LET (...) (DEFUN ...)) forms that people with Pascal/Ada
background presumably tend to write. IIRC he invoked patchability at
run-time. But also:
A nested macrolet is harder to assess than a top-level one.
(macroexpand-1 '(column X)) comes trivial with defmacro.
So it's just a matter of convenience and growing code incrementally.
I remember Henry Baker saying that the editor should support the
source level code transformations involved here, e.g. turning DEFUN
into FLET, or moving an inner LET outside of a DEFUN, etc. I never
wrote such code for Emacs and I'm not aware whether anyone else did.
I also cannot recall having that on the Symbolics...
As I recently wrote in another recent thread, my code is full of
pseudo test forms as follows:
(defmacro %column-end (column)
...)
;(macroexpand-1 '(%column-end src))
that I use with C-x c-e in Emacs, during both development and porting.
Macrolet doesn't allow this usage.
I'd be open for something which allows greater ease of use or
productivity. Have fun!
Regards,
Jorg Hohle
Telekom/T-Systems Technology Center