Ok, I think I finally found the stick that is getting me to seriously
consider lisp over python.
I have to crunch a large amount of logfile data. Generally, this
process involves:
1: parsing a number of log files, each with a slightly different format.
2: looking up values in a separate file.
3: bean-counting and aggregating something by group (in other words,
adding or incrementing objects in a hash table.)
4: pretty-printing the output in tsv and possibly LaTeX as a bonus.
Rather than continuing my pattern of cut-and-paste between multiple
python scripts, I thought that this would be a good candidate for a lisp
macro. Not that I'm highly wedded to the macro concept, just that a
macro seems easier to work with than some of my alternatives.
After a bit of searching I came up with this little snippit that does
about %50 of what I want for it to do, and leads the way to %80 of the
rest. However, after quite a bit of googling, and searching through
Practical Common Lisp, I don't quite understand all of it. (Taken
blatantly from http://kantz.com/jason/clim-primer/drawing.htm .)
(defmacro do-file ((path line-variable &key (key #'identity)) &body body)
"Iterate over the lines of the file, binding the line to the line-variable in each iteration"
(let ((str (gensym))
(var (gensym)))
`(with-open-file (,str ,path :direction :input)
(do ((,var (read-line ,str nil)
(read-line ,str nil)))
((not ,var))
(let ((,line-variable (funcall ,key ,var)))
,@body)))))
Just as example code, this can be invoked with (from memory, so there
may be a typo):
(do-file ("tmp.txt" bar) (format t "~a~%" bar))
Some of my issues are:
1: I understand the rebinding due to namespace issues. Binding the line
to a gensym variable seems much safer than the alternatives. However,
I'm not exactly clear on using the function identity here.
2: I don't fully grok keyword variables. My best interpretation is that
this specification permits me to do:
(do-file ("temp.txt" bar :key #'hello-world)...)
This would defeat the purpose of using this macro. Why put this
function in a place where it can be overwritten, as opposed to the body
of the macro? Is there an advantage that I'm missing? What would be
lost by doing (let ((,line-variable (identiy ,var))) or even (let
(,line-variable ,var))?
--
Kirk Job-Sluder
"The square-jawed homunculi of Tommy Hilfinger ads make every day an
existential holocaust." --Scary Go Round
From: Espen Vestre
Subject: Re: newbie needing help (accessing variables within macros)
Date:
Message-ID: <kwy89xrvn7.fsf@merced.netfonds.no>
Kirk Job Sluder <····@jobsluder.net> writes:
> This would defeat the purpose of using this macro. Why put this
> function in a place where it can be overwritten, as opposed to the body
> of the macro? Is there an advantage that I'm missing?
Reuse, I guess. Suppose you have a dozen different parsers that all
youse the common FOOBAR fileformat, and that #'parse-foobar parses
this format. Then you could write e.g.
(defun handle-foobar-file-1 (filename)
(do-file (filename foobar :key #'parse-foobar)
(treat-foobar-struct-1 foobar)))
(defun handle-foobar-file-2 (filename)
(do-file (filename foobar :key #'parse-foobar)
(treat-foobar-struct-2 foobar)))
...
btw here's a tip:
I've written a lot of code in this vein, and have often used a
combination of CLOS generic functions (often eql methods) and macros
to create a declarative expression of the different "rules" for
treating the data. Think "plugins", where plugin could be a new method
for new data type, new parser, new output function for new report
format etc., etc.
--
(espen)
Kirk Job Sluder <····@jobsluder.net> writes:
> Ok, I think I finally found the stick that is getting me to seriously
> consider lisp over python.
>
> I have to crunch a large amount of logfile data. Generally, this
> process involves:
> 1: parsing a number of log files, each with a slightly different format.
> 2: looking up values in a separate file.
> 3: bean-counting and aggregating something by group (in other words,
> adding or incrementing objects in a hash table.)
> 4: pretty-printing the output in tsv and possibly LaTeX as a bonus.
>
> Rather than continuing my pattern of cut-and-paste between multiple
> python scripts, I thought that this would be a good candidate for a lisp
> macro. Not that I'm highly wedded to the macro concept, just that a
> macro seems easier to work with than some of my alternatives.
>
> After a bit of searching I came up with this little snippit that does
> about %50 of what I want for it to do, and leads the way to %80 of the
> rest. However, after quite a bit of googling, and searching through
> Practical Common Lisp, I don't quite understand all of it. (Taken
> blatantly from http://kantz.com/jason/clim-primer/drawing.htm .)
>
> (defmacro do-file ((path line-variable &key (key #'identity)) &body body)
> "Iterate over the lines of the file, binding the line to the line-variable in each iteration"
> (let ((str (gensym))
> (var (gensym)))
> `(with-open-file (,str ,path :direction :input)
> (do ((,var (read-line ,str nil)
> (read-line ,str nil)))
> ((not ,var))
> (let ((,line-variable (funcall ,key ,var)))
> ,@body)))))
>
>
> Just as example code, this can be invoked with (from memory, so there
> may be a typo):
> (do-file ("tmp.txt" bar) (format t "~a~%" bar))
>
> Some of my issues are:
> 1: I understand the rebinding due to namespace issues. Binding the line
> to a gensym variable seems much safer than the alternatives. However,
> I'm not exactly clear on using the function identity here.
>
> 2: I don't fully grok keyword variables. My best interpretation is that
> this specification permits me to do:
>
> (do-file ("temp.txt" bar :key #'hello-world)...)
>
> This would defeat the purpose of using this macro. Why put this
> function in a place where it can be overwritten, as opposed to the body
> of the macro? Is there an advantage that I'm missing? What would be
> lost by doing (let ((,line-variable (identiy ,var))) or even (let
> (,line-variable ,var))?
It looks like the purpose of the "key" parameter is to allow
extra processing of each line. Examples could be to turn the line into
lowercase, split it into a list of fields, or even to split it into
fields, and then create an object/struct from the fields.
--
Raymond Wiker Mail: ·············@fast.no
Senior Software Engineer Web: http://www.fast.no/
Fast Search & Transfer ASA Phone: +47 23 01 11 60
P.O. Box 1677 Vika Fax: +47 35 54 87 99
NO-0120 Oslo, NORWAY Mob: +47 48 01 11 60
Kirk Job Sluder wrote:
> Ok, I think I finally found the stick that is getting me to seriously
> consider lisp over python.
>
> I have to crunch a large amount of logfile data. Generally, this
> process involves:
> 1: parsing a number of log files, each with a slightly different format.
> 2: looking up values in a separate file.
> 3: bean-counting and aggregating something by group (in other words,
> adding or incrementing objects in a hash table.)
> 4: pretty-printing the output in tsv and possibly LaTeX as a bonus.
>
> Rather than continuing my pattern of cut-and-paste between multiple
> python scripts, I thought that this would be a good candidate for a lisp
> macro. Not that I'm highly wedded to the macro concept, just that a
> macro seems easier to work with than some of my alternatives.
>
> After a bit of searching I came up with this little snippit that does
> about %50 of what I want for it to do, and leads the way to %80 of the
> rest. However, after quite a bit of googling, and searching through
> Practical Common Lisp, I don't quite understand all of it. (Taken
> blatantly from http://kantz.com/jason/clim-primer/drawing.htm .)
>
> (defmacro do-file ((path line-variable &key (key #'identity)) &body body)
> "Iterate over the lines of the file, binding the line to the line-variable in each iteration"
> (let ((str (gensym))
> (var (gensym)))
> `(with-open-file (,str ,path :direction :input)
> (do ((,var (read-line ,str nil)
> (read-line ,str nil)))
> ((not ,var))
> (let ((,line-variable (funcall ,key ,var)))
> ,@body)))))
>
>
> Just as example code, this can be invoked with (from memory, so there
> may be a typo):
> (do-file ("tmp.txt" bar) (format t "~a~%" bar))
>
> Some of my issues are:
> 1: I understand the rebinding due to namespace issues. Binding the line
> to a gensym variable seems much safer than the alternatives. However,
> I'm not exactly clear on using the function identity here.
What is not clear? (identity X) -> X. identity is useful when you are
giving the user the option of a callback (in your case KEY) and you want
to avoid: (let ((thing (if callback (funcall callback stuff) stuff)))...
ie, it just makes things a little easier on the tool author.
>
> 2: I don't fully grok keyword variables. My best interpretation is that
> this specification permits me to do:
>
> (do-file ("temp.txt" bar :key #'hello-world)...)
>
> This would defeat the purpose of using this macro. Why put this
> function in a place where it can be overwritten, as opposed to the body
> of the macro? Is there an advantage that I'm missing? What would be
> lost by doing (let ((,line-variable (identiy ,var))) or even (let
> (,line-variable ,var))?
>
I believe this is in the spirit of something like FIND-IF, which also
accepts a :key parameter: it is just a little convenience, and arguably
overkill, but (a) Lisp is sometimes like that and (b) consider:
(find-if 'oddp people :key 'age)
vs
(find-if (lambda (p) (oddp (age p))) people)
Way too many parentheses. :)
In your case, the tool author is guessing that (a) in a sufficient
number of cases the user will want some preprocessing of each line and
(b) that it can safely be expressed as a standalone parameter to be
inflexibly applied just as the macroexpansion decides. ie, if it was the
kind of thing that would sometimes have to go this way and sometimes
have to go that way, then there would be few cases where the option
could be used, so why bother?
kenny
--
Cells? : http://www.common-lisp.net/project/cells/
Cello? : http://www.common-lisp.net/project/cello/
Cells-Gtk? : http://www.common-lisp.net/project/cells-gtk/
Why Lisp? http://lisp.tech.coop/RtL%20Highlight%20Film
"Doctor, I wrestled with reality for forty years, and I am happy to
state that I finally won out over it." -- Elwood P. Dowd