Elisp Lesson: Repeated Find Replace
Xah Lee, 2007-10-09
(A HTML version with links and colors is at:
http://xahlee.org/emacs/elisp_repeat_replace.html
)
This page shows a example of writing a emacs lisp function that cleans
up a file's content by repeated application of Find-replace operation.
If you don't know elisp, first take a gander at Emacs Lisp Basics.
---------------------------------------
The Problem
Summary
I want to write a command such that it repeatedly does find-replace on
several find-replace pairs on the current file.
Detail
I have a website of Math Surface Gallery, which contains a Java applet
called JavaView that allows people to view 3D objects interactively.
(i.e. live rotation with the mouse) For example, this is one of the
java applet page: Costa surface applet. There are about 70 of such
surfaces. Each of these surface has a raw data file that the java
applet reads. For example, for the Costa surface above, the raw data
file is: costa.mgs.gz. These files are just Mathematica graphics in
plain text, and compressed with gzip.
The content of the file looks like this:
Graphics3D[{{
Polygon[{{3.552, -0.001061, 2.689}, {3.552, 0.03079, 2.689},
{3.025, 0.02634, 2.524}, {3.025, -0.001061, 2.524}}],
Polygon[{{3.552, 0.03079, 2.689}, {3.550, 0.1250, 2.689},
{3.023, 0.1074, 2.524}, {3.025, 0.02634, 2.524}}],
Polygon[{...}],
...
}}]
Since the file contains thousands or tens of thousands of polygons, it
can get large, and takes a while for the java applet to load it from
the net. One way to reduce file size is to reduce the number of
polygons. But given a file, spaces and end-of-line characters can be
deleted, and the decimal numbers can be safely truncated to 3 digits.
So, typically, i open the file, do global find-and-replace operations
(“Alt-x query-replace”) by replacing “, ” to just “,”, and delete line
endings (replacing “\n” by nothing), delete multiple spaces. To
truncate decimals to 3 places, i use the “Alt+x query-replace-regexp”
with pattern “\([0-9]\)\.\([0-9][0-9][0-9]\)[0-9]+” and replace it
with “\1.\2”.
After a while, this process gets repetitious. It would be nice, to
have a emacs command, so that when invoked, it will perform all these
find-and-replace operations on the current file in one-shot. This
would reduce some 50 keystrokes and eye-balling into a single
brainless button press.
---------------------------------------
Solution
Here's the solution:
(defun replace-mgs ()
"Does several string replacement on current buffer, for Mathematica
graphics file format used by JavaView, that has suffix “.mgs”. The
goal of these replacement is to reduce file size."
(interactive)
(goto-char (point-min))
(while (search-forward "\n" nil t) (replace-match "" nil t))
(goto-char (point-min))
(while (search-forward-regexp " +" nil t) (replace-match " " nil
t))
(goto-char (point-min))
(while (search-forward ", " nil t) (replace-match "," nil t))
(goto-char (point-min))
(while (search-forward-regexp "\\([0-9]\\)\\.\\([0-9][0-9][0-9]\\)
[0-9]+" nil t)
(replace-match "\\1.\\2" t nil))
)
This function is relatively simple. It does a series of replacement
using the “while” loop, each time moving the cursor to the beginning
of file. The gist is the “search-forward”, “search-forward-regexp”,
and “replace-match”.
The “search-forward” function takes a string and moves the cursor to
the end of the string that matches. “search-forward-regexp” does
similar. The “replace-match” simply replaces the text matched by the
last search.
One interesting aspect about “search-forward-regexp” is that you must
use 2 backslashes to represent one backslash. This is because
backslash in emacs string needs a backslash to represent it. Then,
this string is passed to emacs's regex engine.
Another thing of interest is that the first 2 optional parameters to
“replace-match” function is “fixedcase” and “literal”, both are
booleans. If “fixedcase” is non-nil, then emacs will not alter the
case of the replacement. (ohterwise it decides smartly based on the
case of the matched text) If “literal” is non-nil, then emacs will
interprete the replacement string as literal. (in our case, we want
“literal” to be “nil” when we use search-forward-regexp.)
Emacs is beautiful!
Addendum: here's the Mathematica code to export graphics into a text
file forcing all numbers to be printed in a simple “d.dddd” format.
Otherwise, Mathematica may print numbers in various forms such as
“2.25`*^-9”, “\(7.2389`\)”, “3.141592653589793238462643383279503`20”.
writeToFileRounded[expr_Graphics3D,fileName_?StringQ,prec_:
4]:=Module[{},
OpenWrite[fileName];
WriteString[fileName,"Graphics3D["];
WriteString[fileName,
StringReplace[
········@
NumberForm[·····@SetPrecision[Chop[expr,10^-(prec
+1)],prec],
ExponentFunction\[Rule](If[-
Infinity<#<Infinity,Null,#]&)],
"],"->"],\n"]];
WriteString[fileName,"]"];
Close[fileName]
];
writeToFileRounded[surf,"helicoid.ma",4]
(*the first argument is a Graphics3D object, the second is a name to
save to, the third is number of decimal places for the coordinate
values.*)
Xah
···@xahlee.org
∑ http://xahlee.org/
Xah Lee <···@xahlee.org> writes:
> (defun replace-mgs ()
> "Does several string replacement on current buffer, for Mathematica
> graphics file format used by JavaView, that has suffix $B!H(B.mgs$B!I(B. The
> goal of these replacement is to reduce file size."
In elisp, the first line of a function's docstring will show up in
apropos searches and the like; (lispref)Function Documentation talks a
bit about this: "The first line of the documentation string should
stand on its own, because `apropos' displays just this first line. It
should consist of one or two complete sentences that summarize the
function's purpose."
I'd suggest something like
(defun replace-mgs ()
"Reduce size of an mgs file by removing whitespace and truncating numbers.
An mgs file contains a series of Mathematica graphics commands; newlines
can be removed, and multiple spaces collapsed into one. Numbers can be
truncated at the thousandths column."
As for your code, I don't know why you are indenting interactive
differently than the further forms; it's a little bit unusual
(although I'll sometimes put in a blank line between it and the next
form, if I want to highlight it).
And, although it may complicate your tutorial, I'd suggest your code
use either save-excursion or explicitly save the point-marker at the
beginning and restore it at the end. Something like
(save-excursion
(goto-char (point-min))
(while ...)
(goto-char (point-min))
(while ...)
...)
That way, the user can apply it from anywhere within the file, and
sort of recognize where they end up.
--
I'm awfully glad I'm a Beta, because I don't work so hard.
Brian Palmer wrote:
«In elisp, the first line of a function's docstring will show up in
apropos searches and the like; (lispref)Function Documentation talks
a...»
Thanks. Yeah i know... fixed.
i'll ramble a bit in the following.
«As for your code, I don't know why you are indenting interactive
differently than the further forms;»
just sloppy... never really pay much attention about any so-called
“coding style” (which often means the code fomatting habit) Actualy, i
consider the rampant reference and concern about “coding style”, is a
egregious fuck up that can be attributed significantly to unix and C.
The damage is far and wide, and also influenced negatively the lisp
community.
In general, a programer should never have to press any returns, tabs,
etc for the purpose of formatting his code. The editor should
automatically wrap the code properly for display formatting. Many
language, esp those turds from Unix/C's family (tcsh,Perl,C++,Java)
has a syntax that this is impossible for this at a lex level. For Lisp
the lang it is is possible, but the thinking isn't there.
In mathematica, i never have to spend time to fiddle with code
formatting. (and when i do actually insert a indent or return, it is
intentional and means something (usually indicating a semantic/
algorithmic/code unit/break)) It's quite interesting to note that
Mathematica not only formats codes automatically, but the fact that
its code can contain 2-dimentional type-set mathematics (i.e.
fractions, roots, powers, subs, parens, nesting, integrals, ...
variously combined.), and auto-wrap such type-set expressions on the
fly. This feature, started in Mathematica version 3 about 1997, is
today a decade-old technology. Most coders today (e.g. Perl) are still
arguing and wallowing about the fine points of how many spaces a
indent should be. (not just innanely in newsgroups, but there are
entire literature (e.g. guides) devoted to it. Motherfucking morons
and holes.)
As to lisp, it would be nice, if a programer can press a button in
emacs, then the current code block would be formatted by a simple
lexical analysis. (similar to how fill-paragraph would work) I think
it is relatively trivial to code his command, but to my surprise, it
is not done. I was told by one Scheme expert Taylor R. Campbell (aka
Riastradh, author of paren-edit mode) that this is non-trivial, but i
couldn't believe it and maybe he misunderstood what i wanted about
this command.
In fact when i gained more lisp experience in mid 2000s, i was
surprised to find that the concept of auto-wrap is basically non-
existant among lispers. Lispers, to a lesser degree than C/Perl
morons, still do discuss and debate now and then about formatting
trivia.
Further readings:
• Unix, RFC, and Line Truncation, Xah Lee, 2002
http://xahlee.org/UnixResource_dir/writ/truncate_line.html
• The Harm of hard-wrapping Lines, Xah Lee, 2005
http://xahlee.org/UnixResource_dir/writ/hard-wrap.html
• A Text Editor Feature: Syntax Tree Walk, Xah Lee, 2006
http://xahlee.org/emacs/syntax_tree_walk.html
Xah
···@xahlee.org
∑ http://xahlee.org/
On Oct 11, 12:50 am, Brian Palmer <·······@rescomp.stanford.edu>
wrote:
> Xah Lee <····@xahlee.org> writes:
> > (defun replace-mgs ()
> > "Does several string replacement on current buffer, for Mathematica
> > graphics file format used by JavaView, that has suffix “.mgs”. The
> > goal of these replacement is to reduce file size."
>
> In elisp, the first line of a function's docstring will show up in
> apropos searches and the like; (lispref)Function Documentation talks a
> bit about this: "The first line of the documentation string should
> stand on its own, because `apropos' displays just this first line. It
> should consist of one or two complete sentences that summarize the
> function's purpose."
>
> I'd suggest something like
> (defun replace-mgs ()
> "Reduce size of an mgs file by removing whitespace and truncating numbers.
> An mgs file contains a series of Mathematica graphics commands; newlines
> can be removed, and multiple spaces collapsed into one. Numbers can be
> truncated at the thousandths column."
>
> As for your code, I don't know why you are indenting interactive
> differently than the further forms; it's a little bit unusual
> (although I'll sometimes put in a blank line between it and the next
> form, if I want to highlight it).
>
> And, although it may complicate your tutorial, I'd suggest your code
> use either save-excursion or explicitly save the point-marker at the
> beginning and restore it at the end. Something like
>
> (save-excursion
> (goto-char (point-min))
> (while ...)
> (goto-char (point-min))
> (while ...)
> ...)
>
> That way, the user can apply it from anywhere within the file, and
> sort of recognize where they end up.
>
> --
> I'm awfully glad I'm a Beta, because I don't work so hard.
On Thu, 11 Oct 2007 18:06:56 -0700, Xah Lee <···@xahlee.org> wrote:
>As to lisp, it would be nice, if a programer can press a button in
>emacs, then the current code block would be formatted by a simple
>lexical analysis. (similar to how fill-paragraph would work) I think
>it is relatively trivial to code his command, but to my surprise, it
>is not done. I was told by one Scheme expert Taylor R. Campbell (aka
>Riastradh, author of paren-edit mode) that this is non-trivial, but i
>couldn't believe it and maybe he misunderstood what i wanted about
>this command.
So the question that remains is : if it is trivial, why didn't you
write that function ? (No intent to be aggressive by that question,
just I'd like to know.)
Xah wrote: «As to lisp, it would be nice, if a programer can press a
button in emacs, then the current code block would be formatted by a
simple lexical analysis. (similar to how fill-paragraph would work) I
think it is relatively trivial to code his command, but to my
surprise, it is not done. I was told by one Scheme expert Taylor R.
Campbell (aka Riastradh, author of paren-edit mode) that this is non-
trivial, but i couldn't believe it and maybe he misunderstood what i
wanted about this command.»
someone wrote:
«So the question that remains is : if it is trivial, why didn't you
write that function ? (No intent to be aggressive by that question,
just I'd like to know.)»
trivial in the sense to anyone who has elementary experience writing a
parser, or is a lisp expert (say, with 1 year of full-time coding
lisp).
i have never wrote any parser or lexer, nor am a professional lisper.
For me to create a auto-formater for lisp code, it will probably take
a day, which may become days when running into fine points. (just for
a proof-of-concept version)
However, here's the reason how it is really trivial.
Simply count the levels of nesting of parens. For example, this code:
(defun previous-user-buffer ()
"Switch to the next user buffer in cyclic order."
(interactive)
(previous-buffer)
(let ((i 0))
(while (and (string-match "^*" (buffer-name)) (< i 10))
(setq i (1+ i)) (previous-buffer) )))
each left paren has a level of nesting. Say, n=0, n=1, n=2...etc. A
simplest version of auto-format (or fill-paragraph for lisp) is to
start a new line for each left paren, with n being the number of
indent. So, the above code would be formatted like this (using 1 space
for indent in this exmaple):
(defun previous-user-buffer
() "Switch to the next user buffer in cyclic order."
(interactive)
(previous-buffer)
(let
(
(i 0))
(while
(and
(string-match "^*"
(buffer-name))
(< i 10))
(setq i
(1+ i))
(previous-buffer))))
Now, this is probably be too much indent. So, we can modify it by
reduce the indenting by 2. e.g. the number of indentation for level n
left paren is n/2. So, the above code would be formatted like this:
(defun previous-user-buffer () "Switch to the next user buffer in
cyclic order." (interactive)
(previous-buffer)
(let
(
(i 0))
(while
(and
(string-match "^*"
(buffer-name))
(< i 10))
(setq i
(1+ i))
(previous-buffer))))
Ok, this is probably not a good idea. Ok, so instead of reducing by n/
2, we use a huristics such that: if a complete sexp is less than 70
char, then render the whole sexp in one line (abort any analysis or
indenting of code inside that sexp)
here's how the code would look with this rule:
123456789 123456789 123456789 123456789 123456789 123456789 123456789
(defun previous-user-buffer
() "Switch to the next user buffer in cyclic order."
(interactive)
(previous-buffer)
(let ((i 0))
(while
(and (string-match "^*" (buffer-name)) (< i 10))
(setq i (1+ i))
(previous-buffer))))
... looks pretty good. I don't know how well this would work out for
more complex code... but i think idea is there. Adding to the
heuristics might be special rules dealing with the doc string and
other special non-regular lisp syntaxes (such as “'”, “#”, etc).
(However, such special rule should be kept as minimal as possible)
on the whole, a simple formating by lexical analysis in not going to
be as pretty as manual and customized formating done by lispers as the
lisp code base exists now. However, it is my opinion, if lispers
adapts to such a uniform, simple, machine-produced auto-formating (by
a fill-paragraph command geared for lisp), the impact on lisp
community consider as whole, will be tremendously positive in a way
that that would fundamentally influence the algorithms lisp coders
actually produce.
----
This essay is now archived at:
http://xahlee.org/UnixResource_dir/writ/hard-wrap.html
Xah
···@xahlee.org
∑ http://xahlee.org/
On Oct 11, 9:02 pm, subtenante <··············@gmail.com> wrote:
> On Thu, 11 Oct 2007 18:06:56 -0700,XahLee<····@xahlee.org> wrote:
> >As to lisp, it would be nice, if a programer can press a button in
> >emacs, then the current code block would be formatted by a simple
> >lexical analysis. (similar to how fill-paragraph would work) I think
> >it is relatively trivial to code his command, but to my surprise, it
> >is not done. I was told by one Scheme expert Taylor R. Campbell (aka
> >Riastradh, author of paren-edit mode) that this is non-trivial, but i
> >couldn't believe it and maybe he misunderstood what i wanted about
> >this command.
>
> So the question that remains is : if it is trivial, why didn't you
> write that function ? (No intent to be aggressive by that question,
> just I'd like to know.)
correction:
in my previous post about lisp code formatting, the final form should
be:
(defun previous-user-buffer
() "Switch to the next user buffer in cyclic order."
(interactive)
(previous-buffer)
(let
((i 0))
(while
(and
(string-match "^*" (buffer-name))
(< i 10))
(setq i (1+ i))
(previous-buffer))))
Xah
···@xahlee.org
∑ http://xahlee.org/
In article <························@e34g2000pro.googlegroups.com>,
Xah Lee <···@xahlee.org> wrote:
> correction:
>
> in my previous post about lisp code formatting, the final form should
> be:
>
> (defun previous-user-buffer
> () "Switch to the next user buffer in cyclic order."
> (interactive)
> (previous-buffer)
> (let
> ((i 0))
> (while
> (and
> (string-match "^*" (buffer-name))
> (< i 10))
> (setq i (1+ i))
> (previous-buffer))))
>
> Xah
> ···@xahlee.org
> ∑ http://xahlee.org/
But that's not Lisp style formatting.
Lisp style formatting would be something like:
(defun previous-user-buffer ()
"Switch to the next user buffer in cyclic order."
(interactive)
(previous-buffer)
(let ((i 0))
(while (and (string-match "^*" (buffer-name))
(< i 10))
(setq i (1+ i))
(previous-buffer))))
On the Lisp Machine the Zmacs command 'Format Code'
will format the code for you. It 'knows' about
things like quotes, backquotes, macros, ...
It takes a whole expressions and formats it
into a buffer. Note that it does more than
just indenting lines. It can take a single like code
and reformats it across several lines according
to usual Lisp formatting conventions.
>
> On the Lisp Machine the Zmacs command 'Format Code'
> will format the code for you. It 'knows' about
> things like quotes, backquotes, macros, ...
> It takes a whole expressions and formats it
> into a buffer. Note that it does more than
> just indenting lines. It can take a single like code
> and reformats it across several lines according
> to usual Lisp formatting conventions.
LOL
or since he is obviously using EMACS how about M-x indent-form
In article <·················@pandora.upc.no>,
"John Thingstad" <··············@chello.no> wrote:
> >
> > On the Lisp Machine the Zmacs command 'Format Code'
> > will format the code for you. It 'knows' about
> > things like quotes, backquotes, macros, ...
> > It takes a whole expressions and formats it
> > into a buffer. Note that it does more than
> > just indenting lines. It can take a single like code
> > and reformats it across several lines according
> > to usual Lisp formatting conventions.
>
> LOL
Don't LOL.
> or since he is obviously using EMACS how about M-x indent-form
Which M-x indent-form are you talking about? My GNU Emacs does not have
that.
I'm also not about indenting forms, but formatting it.
If you have
(defun foo (bar baz) "some comment" (when (and bar baz) (print bar)
(print baz)))
In Zmacs M-x format code
will reformat the code to
(defun foo (bar baz)
"some comment"
(when (and bar baz)
(print bar)
(print baz)))
On Oct 13, 12:44 pm, Rainer Joswig <······@lisp.de> wrote:
> In article <·················@pandora.upc.no>,
> "John Thingstad" <··············@chello.no> wrote:
>
> > > On the Lisp Machine the Zmacs command 'Format Code'
> > > will format the code for you. It 'knows' about
> > > things like quotes, backquotes, macros, ...
> > > It takes a whole expressions and formats it
> > > into a buffer. Note that it does more than
> > > just indenting lines. It can take a single like code
> > > and reformats it across several lines according
> > > to usual Lisp formatting conventions.
>
> > LOL
>
> Don't LOL.
OMG TROFLMAO FTW!
> > or since he is obviously using EMACS how about M-x indent-form
>
> Which M-x indent-form are you talking about? My GNU Emacs does not have
> that.
>
> I'm also not about indenting forms, but formatting it.
>
> If you have
>
> (defun foo (bar baz) "some comment" (when (and bar baz) (print bar)
> (print baz)))
>
> In Zmacs M-x format code
> will reformat the code to
>
> (defun foo (bar baz)
> "some comment"
> (when (and bar baz)
> (print bar)
> (print baz)))
Recent Gnu Emacsen have a (non-interactive) pp-buffer function which
kinda sorta does something like this. You could wrap it in an
interactive function:
(defun format-defun ()
(interactive)
(save-excursion
(save-restriction
(beginning-of-defun)
(let ((start (point)))
(end-of-defun)
(narrow-to-region start (point))
(pp-buffer)))))
The good news is that you now have a command that will consistently
reformat your code to something resembling Lisp style. The bad news
is that it has a pretty deranged idea of where lines should be broken:
(defun foo
(bar baz)
"some comment"
(when
(and bar baz)
(print bar)
(print baz)))
This is the type of thing that makes you wonder if an open source Fred
couldn't be developed into an IDE front-end for other Lisps in
addition to MCL itself. Which is a nice way of saying "throw up your
hands in frustration at how brain dead Emacs is". I mean, that's
certainly not the style in which the pp code itself is formatted.
In article <························@v29g2000prd.googlegroups.com>,
"Thomas F. Burdick" <········@gmail.com> wrote:
...
> > If you have
> >
> > (defun foo (bar baz) "some comment" (when (and bar baz) (print bar)
> > (print baz)))
> >
> > In Zmacs M-x format code
> > will reformat the code to
> >
> > (defun foo (bar baz)
> > "some comment"
> > (when (and bar baz)
> > (print bar)
> > (print baz)))
>
> Recent Gnu Emacsen have a (non-interactive) pp-buffer function which
> kinda sorta does something like this. You could wrap it in an
> interactive function:
>
> (defun format-defun ()
> (interactive)
> (save-excursion
> (save-restriction
> (beginning-of-defun)
> (let ((start (point)))
> (end-of-defun)
> (narrow-to-region start (point))
> (pp-buffer)))))
>
> The good news is that you now have a command that will consistently
> reformat your code to something resembling Lisp style. The bad news
> is that it has a pretty deranged idea of where lines should be broken:
>
> (defun foo
> (bar baz)
> "some comment"
> (when
> (and bar baz)
> (print bar)
> (print baz)))
Probably, not what one wants. ;-)
> This is the type of thing that makes you wonder if an open source Fred
> couldn't be developed into an IDE front-end for other Lisps in
> addition to MCL itself. Which is a nice way of saying "throw up your
> hands in frustration at how brain dead Emacs is". I mean, that's
> certainly not the style in which the pp code itself is formatted.
It is a good idea.
BUT: Fred runs only on MCL and is very much tied to the Mac.
It also is not that advanced is some respects. It is
simple to use and that's a huge plus - but it lacks lots
of stuff. It is fun to hack, but as I said don't
think that you can port it to GTK or something like that.
On the Mac one might have some success with OpenMCL's
version of Hemlock.
Rainer Joswig wrote:
«On the Lisp Machine the Zmacs command 'Format Code' will format the
code for you. It 'knows' about things like quotes, backquotes,
macros, ... It takes a whole expressions and formats it into a
buffer. Note that it does more than just indenting lines. It can take
a single like code and reformats it across several lines according to
usual Lisp formatting conventions.»
http://en.wikipedia.org/wiki/Zmacs
http://en.wikipedia.org/wiki/LispWorks
That's great. Good to know.
since emacs has a entire parser and compiler of elisp ... why doesn't
emacs have a auto-wrap/pretty-print command (at least for elisp)?
Xah
···@xahlee.org
∑ http://xahlee.org/
Xah Lee <···@xahlee.org> writes:
> someone wrote:
> �So the question that remains is : if it is trivial, why didn't you
> write that function ? (No intent to be aggressive by that question,
> just I'd like to know.)�
>
> trivial in the sense to anyone who has elementary experience writing a
> parser, or is a lisp expert (say, with 1 year of full-time coding
> lisp).
>
> i have never wrote any parser or lexer, nor am a professional lisper.
> For me to create a auto-formater for lisp code, it will probably take
> a day, which may become days when running into fine points. (just for
> a proof-of-concept version)
Why write one when it is included?
C-x h TAB
indents the whole buffer in the correct style for the language.
Bye,
Tassilo
--
Chuck Norris is not Politically Correct. He is just Correct. Always.