From: Damien Kick
Subject: Question about design, defmacro, macrolet, and &environment
Date:
Message-ID: <ov3cd2gw3k.fsf@email.mot.com>
;;;; I've been experimenting with the following lispification of
;;;; Expect. Any help/guidance would be very appreciated.
(defpackage "PLAYGROUND"
(:nicknames "PG")
(:export "EXPECT" "WITH-SPAWN" "EXPECT-IT"
"EXPECT-1" "WITH-SPAWN-1" "DEFAULT-VAR" "EXPECT-1-LEMMA"))
(in-package "PLAYGROUND")
;;; Let's start with the utilities.
(defmacro with-gensyms (syms &body body)
`(let ,(mapcar #'(lambda (s)
`(,s (gensym)))
syms)
,@body))
;;; First attempt is simplistic. Requires expect and expect-it.
(defun expect (x)
(format t "The top-level expect. X => ~A~%" x))
(defmacro with-spawn ((var val) &body body)
`(let ((,var ,val))
(macrolet
((expect-it ()
`(expect ,',var)))
,@body)))
;;; First attempt at using the same name, expect, instead of two,
;;; expect and expect-it.
;; Broken. Can this be fixed? Is there some way to have the macrolet
;; expect expand only once and, from then on, turn into a function
;; call?
(defmacro with-spawn-broken ((var val) &body body)
`(let ((,var ,val))
(macrolet
((expect ()
`(funcall #'expect ,',var)))
,@body)))
;;; First attempt at fixing with-spawn-broken by using the environment
;;; stuff. It too is broken.
(defmacro with-spawn-1 ((var val) &body body)
`(let ((,var ,val))
(macrolet
((default-var () `(t ,',var)))
,@body)))
(defmacro expect-1 (&optional var &environment env)
(if var
`(expect-1-lemma ,var)
;; The macroexpansion of '(default-var) does not seem to be
;; doing what I want it to do. If this is done within
;; (with-spawn-1 (foo 'x) ...), I'm hoping to get query => (t
;; foo). Apparently, I'm always getting query => default-var.
(let ((query (macroexpand '(default-var) env)))
(if (eq (car query) t)
`(expect-1-lemma ,(cdr query))
`(expect-1-lemma)))))
(defun expect-1-lemma (var)
(format t "The top-level expect-1-lemma. X => ~A~%" var))
;;;; Now, this is my attempt to use with-spawn-1 with clisp-hs/ilisp...
i i i i i i i ooooo o ooooooo ooooo ooooo
I I I I I I I 8 8 8 8 8 o 8 8
I \ `+' / I 8 8 8 8 8 8
\ `-+-' / 8 8 8 ooooo 8oooo
`-__|__-' 8 8 8 8 8
| 8 o 8 8 o 8 8
------+------ ooooo 8oooooo ooo8ooo ooooo 8
Copyright (c) Bruno Haible, Michael Stoll 1992, 1993
Copyright (c) Bruno Haible, Marcus Daniels 1994-1997
Copyright (c) Bruno Haible, Pierpaolo Bernardi, Sam Steingold 1998
Copyright (c) Bruno Haible, Sam Steingold 1999-2002
;;;[... snip ...]
[17]> ;;;Compile /home/kick/tmp/f00f.lisp
Compiling file /home/kick/tmp/f00f.lisp ...
Wrote file /home/kick/tmp/f00f.fas
0 errors, 0 warnings
#P"/home/kick/tmp/f00f.fas" ;
NIL ;
NIL
[16]>
;; Loading file /home/kick/tmp/f00f.fas ...
;; Loaded file /home/kick/tmp/f00f.fas
T
[17]> (defpackage :pg-user (:use :cl :pg))
#<PACKAGE PG-USER>
[18]> (in-package :pg-user)
#<PACKAGE PG-USER>
PG-USER[19]> (pprint
(macroexpand-1
'(with-spawn-1 (foo 'x)
(expect-1))))
(LET ((FOO 'X)) (MACROLET ((DEFAULT-VAR NIL `(T FOO))) (EXPECT-1)))
;;;Why doesn't pprint indent this?
PG-USER[20]> (LET ((FOO 'X))
(MACROLET
((DEFAULT-VAR NIL `(T FOO)))
(macroexpand '(EXPECT-1))))
(EXPECT-1-LEMMA) ;
T
;;;It would seem that the macrolet is not being expanded within
;;;expect-1 like I thought it should be.
PG-USER[21]>
(with-spawn-1 (foo 'x)
(expect-1))
*** - EVAL: the function FOO is undefined
;;;At this point, I'm just too confused by the macroexpansion from
;;;above to be able to make sense of this.
1. Break PG-USER[22]>
PG-USER[23]>
From: Damien Kick
Subject: Re: Question about design, defmacro, macrolet, and &environment
Date:
Message-ID: <ovwuaefgtw.fsf@email.mot.com>
BTW, I should have explained that the following is an attempt to
distill my exploration of the interface(s). In other words, the toy
versions of with-spawn and expect would eventually be replaced with
something like the following
(with-spawn ((id "telnet" "someplace.cool.org"))
(expect "login:")) ; implies (expect "login:" id)
At the moment, I'm just toying with using macros to produce a version
of EXPECT that requires an argument unless it is used within a
WITH-SPAWN by using the technique described here
<http://groups.google.com/groups?q=WITH-CONTEXT-1&hl=en&lr=&ie=UTF-8&oe=UTF-8&selm=R-Cdned1bP-Nt--iU-KYvA%40dls.net&rnum=1>.
But, as I detailed in the previous post, this is not working for me
for some reason.
Or should I just use something like the following?
(defun expect (stuff &optional (stream *spawn*))
(do-something-cool))
(let ((*spawn* (spawn "telnet" "someplace.cool.org")))
(expect "login:"))
It would be a lot simpler. For some reason, though, I like the idea
of using WITH-SPAWN as a sort of code walker to rewrite EXPECT to
supply the parameter instead of using &OPTIONAL so that optionality is
context sensitive.
Damien Kick <······@email.mot.com> writes:
> ;;;; I've been experimenting with the following lispification of
> ;;;; Expect. Any help/guidance would be very appreciated.
>
> (defpackage "PLAYGROUND"
> (:nicknames "PG")
> (:export "EXPECT" "WITH-SPAWN" "EXPECT-IT"
> "EXPECT-1" "WITH-SPAWN-1" "DEFAULT-VAR" "EXPECT-1-LEMMA"))
>
> (in-package "PLAYGROUND")
>
> ;;; Let's start with the utilities.
>
> (defmacro with-gensyms (syms &body body)
> `(let ,(mapcar #'(lambda (s)
> `(,s (gensym)))
> syms)
> ,@body))
>
> ;;; First attempt is simplistic. Requires expect and expect-it.
>
> (defun expect (x)
> (format t "The top-level expect. X => ~A~%" x))
>
> (defmacro with-spawn ((var val) &body body)
> `(let ((,var ,val))
> (macrolet
> ((expect-it ()
> `(expect ,',var)))
> ,@body)))
>
> ;;; First attempt at using the same name, expect, instead of two,
> ;;; expect and expect-it.
>
> ;; Broken. Can this be fixed? Is there some way to have the macrolet
> ;; expect expand only once and, from then on, turn into a function
> ;; call?
> (defmacro with-spawn-broken ((var val) &body body)
> `(let ((,var ,val))
> (macrolet
> ((expect ()
> `(funcall #'expect ,',var)))
> ,@body)))
>
> ;;; First attempt at fixing with-spawn-broken by using the environment
> ;;; stuff. It too is broken.
>
> (defmacro with-spawn-1 ((var val) &body body)
> `(let ((,var ,val))
> (macrolet
> ((default-var () `(t ,',var)))
> ,@body)))
>
> (defmacro expect-1 (&optional var &environment env)
> (if var
> `(expect-1-lemma ,var)
> ;; The macroexpansion of '(default-var) does not seem to be
> ;; doing what I want it to do. If this is done within
> ;; (with-spawn-1 (foo 'x) ...), I'm hoping to get query => (t
> ;; foo). Apparently, I'm always getting query => default-var.
> (let ((query (macroexpand '(default-var) env)))
> (if (eq (car query) t)
> `(expect-1-lemma ,(cdr query))
> `(expect-1-lemma)))))
>
> (defun expect-1-lemma (var)
> (format t "The top-level expect-1-lemma. X => ~A~%" var))
>
> ;;;; Now, this is my attempt to use with-spawn-1 with clisp-hs/ilisp...
>
> i i i i i i i ooooo o ooooooo ooooo ooooo
> I I I I I I I 8 8 8 8 8 o 8 8
> I \ `+' / I 8 8 8 8 8 8
> \ `-+-' / 8 8 8 ooooo 8oooo
> `-__|__-' 8 8 8 8 8
> | 8 o 8 8 o 8 8
> ------+------ ooooo 8oooooo ooo8ooo ooooo 8
>
> Copyright (c) Bruno Haible, Michael Stoll 1992, 1993
> Copyright (c) Bruno Haible, Marcus Daniels 1994-1997
> Copyright (c) Bruno Haible, Pierpaolo Bernardi, Sam Steingold 1998
> Copyright (c) Bruno Haible, Sam Steingold 1999-2002
>
> ;;;[... snip ...]
>
> [17]> ;;;Compile /home/kick/tmp/f00f.lisp
> Compiling file /home/kick/tmp/f00f.lisp ...
>
> Wrote file /home/kick/tmp/f00f.fas
> 0 errors, 0 warnings
> #P"/home/kick/tmp/f00f.fas" ;
> NIL ;
> NIL
> [16]>
> ;; Loading file /home/kick/tmp/f00f.fas ...
> ;; Loaded file /home/kick/tmp/f00f.fas
> T
> [17]> (defpackage :pg-user (:use :cl :pg))
> #<PACKAGE PG-USER>
> [18]> (in-package :pg-user)
> #<PACKAGE PG-USER>
> PG-USER[19]> (pprint
> (macroexpand-1
> '(with-spawn-1 (foo 'x)
> (expect-1))))
> (LET ((FOO 'X)) (MACROLET ((DEFAULT-VAR NIL `(T FOO))) (EXPECT-1)))
>
> ;;;Why doesn't pprint indent this?
>
> PG-USER[20]> (LET ((FOO 'X))
> (MACROLET
> ((DEFAULT-VAR NIL `(T FOO)))
> (macroexpand '(EXPECT-1))))
> (EXPECT-1-LEMMA) ;
> T
>
> ;;;It would seem that the macrolet is not being expanded within
> ;;;expect-1 like I thought it should be.
>
> PG-USER[21]>
> (with-spawn-1 (foo 'x)
> (expect-1))
> *** - EVAL: the function FOO is undefined
>
> ;;;At this point, I'm just too confused by the macroexpansion from
> ;;;above to be able to make sense of this.
>
> 1. Break PG-USER[22]>
> PG-USER[23]>
From: Peter Seibel
Subject: Re: Question about design, defmacro, macrolet, and &environment
Date:
Message-ID: <m38ymu8do9.fsf@javamonkey.com>
Damien Kick <······@email.mot.com> writes:
> ;;;; I've been experimenting with the following lispification of
> ;;;; Expect. Any help/guidance would be very appreciated.
Let's see if I can beat Kenny to the punch. When you're having trouble
with macros your best bet is to follow this procedure.
Step 1. Write the code that the macro is going to let you write,
that is, a call to the macro.
Step 2. Write the code that you'd have to write by hand to implement
the desired behavior of the code from step 1.
Step 3. Now write a macro that converts what you wrote in Step 1 to
what you wrote in Step 2.
Step 4. Check for hygine problems--if you use local variables in the
macro expansion, make sure they can't shadow variables in in
the calling environment. (GENSYM is your friend here.) Also
check that in the expanded code that none of the forms
passed as arguments to the macro get evaluated more times
that a reasonable person would expect.
The best time to ask questions on c.l.l is after you've done steps 1
and 2. And show us what you came up with in those steps.
For example, if DOTIMES didn't exist we could write it by this process:
Step 1 -- I want to be able to write this:
(dotimes (i 10) (print i)) ;; should print the numbers from 0 to 9
Step 2 -- I could get the same effect by writing this:
(do ((i 0 (1+ i)))
((>= i 10))
(print i))
Step 3 -- Seems like we can use a simple backquote template:
(defmacro dotimes ((var count) &body body)
`(do ((,var 0 (1+ ,var)))
((>= ,var ,count))
,@body))
Step 4 -- Whoops: hygine problem. If I write this:
(dotimes (i (random 100)) (print i))
it's going to call (random 100) each time it tests for the end of
the loop. We can fix that by evaluating the count form once. But
we'll need a variable to hold it in. To avoid another kind of hygine
problem we should use a gensym'd name:
(defmacro dotimes ((var count) &body body)
(let ((count-value (gensym)))
`(do ((,var 0 (1+ ,var))
(,count-value ,count))
((>= ,var ,count-value))
,@body)))
Done. (But not tested--may be typos in there.)
-Peter
--
Peter Seibel ·····@javamonkey.com
Lisp is the red pill. -- John Fraser, comp.lang.lisp
From: Damien Kick
Subject: Re: Question about design, defmacro, macrolet, and &environment
Date:
Message-ID: <ovptg5o2j9.fsf@email.mot.com>
Peter Seibel <·····@javamonkey.com> writes:
> The best time to ask questions on c.l.l is after you've done steps 1
> and 2. And show us what you came up with in those steps.
I'll try and expand on this a bit.
Last time, when I wrote the following, I was attempting to understand
what was happening during the macroexpansion.
> PG-USER[20]> (LET ((FOO 'X))
> (MACROLET
> ((DEFAULT-VAR NIL `(T FOO)))
> (macroexpand '(EXPECT-1))))
> (EXPECT-1-LEMMA) ;
> T
>
> ;;;It would seem that the macrolet is not being expanded within
> ;;;expect-1 like I thought it should be.
>
I did not expect to find (EXPECT-1-LEMMA) but rather (EXPECT-1-LEMMA
FOO).
If I change the DEFMACRO to the following (I had tried this but did
not post it)
(defmacro expect-1 (&optional var &environment env)
(if var
`(expect-1-lemma ,var)
;; The macroexpansion of '(default-var) does not seem to be
;; doing what I want it to do. If this is done within
;; (with-spawn-1 (foo 'x) ...), I'm hoping to get query => (t
;; foo). Apparently, I'm always getting query => default-var.
(let ((query (macroexpand '(default-var) env)))
; (if (eq (car query) t)
; `(expect-1-lemma ,(cdr query))
; `(expect-1-lemma))
`(format t "~A ~A ~A~%" ,(length query) ,(car query)
,(cdr query))
)))
I find the following happens during the expansion.
PG-USER[24]>
;; Loading file /home/kick/tmp/f00f.fas ...
;; Loaded file /home/kick/tmp/f00f.fas
T
PG-USER[25]> (LET ((FOO 'X))
(MACROLET
((DEFAULT-VAR NIL `(T FOO)))
(macroexpand '(EXPECT-1))))
(FORMAT T "~A ~A ~A~%" 1 DEFAULT-VAR NIL) ;
T
PG-USER[26]>
I was expecting that query would contain (T FOO). I don't understand
why it is returning (DEFAULT-VAR), as if there was no enclosing
MACROLET. This is really the meat of my question.
From: Thomas A. Russ
Subject: Re: Question about design, defmacro, macrolet, and &environment
Date:
Message-ID: <ymillqtwbwl.fsf@sevak.isi.edu>
Damien Kick <······@email.mot.com> writes:
>
> I was expecting that query would contain (T FOO). I don't understand
> why it is returning (DEFAULT-VAR), as if there was no enclosing
> MACROLET. This is really the meat of my question.
Because MACROLET has lexical scoping, not dynamic scoping.
--
Thomas A. Russ, USC/Information Sciences Institute
From: Peter Seibel
Subject: Re: Question about design, defmacro, macrolet, and &environment
Date:
Message-ID: <m3ad795lm8.fsf@javamonkey.com>
Damien Kick <······@email.mot.com> writes:
> Peter Seibel <·····@javamonkey.com> writes:
>
> > The best time to ask questions on c.l.l is after you've done steps 1
> > and 2. And show us what you came up with in those steps.
>
> I'll try and expand on this a bit.
>
> Last time, when I wrote the following, I was attempting to understand
> what was happening during the macroexpansion.
>
> > PG-USER[20]> (LET ((FOO 'X))
> > (MACROLET
> > ((DEFAULT-VAR NIL `(T FOO)))
> > (macroexpand '(EXPECT-1))))
> > (EXPECT-1-LEMMA) ;
> > T
>
> > ;;;It would seem that the macrolet is not being expanded within
> > ;;;expect-1 like I thought it should be.
Okay, so I think your immediate problem is that MACROLET defines a
*local*, i.e. lexical macro binding. Since there's no reference to
DEFAULT-VAR within the lexical scope of the MACROLET form, there's no
way the local binding of DEFAULT-VAR is going to have any affect. (You
seem to be expecting that the local definition of DEFAULT-VAR is going
to affect the call to MACROEXPAND-1 *inside* the definition of
EXPECT-1; that's not how it works.)
More generally, I think you still need to pull back and show us 1) the
code you want to be able to write and 2) the code you would have to
write to do achive the desired result without the macro. Maybe I'm
just dense but I have no idea what a program that uses EXPECT is going
to look like (1) and what it would do (2). I suspect you're making
this more complicated than it needs to be but without knowing what
you're trying to do, at a high level, it's hard to say.
-Peter
--
Peter Seibel ·····@javamonkey.com
Lisp is the red pill. -- John Fraser, comp.lang.lisp
From: Damien Kick
Subject: Re: Question about design, defmacro, macrolet, and &environment
Date:
Message-ID: <ovllqtnp4l.fsf@email.mot.com>
Peter Seibel <·····@javamonkey.com> writes:
> Damien Kick <······@email.mot.com> writes:
>
> > Peter Seibel <·····@javamonkey.com> writes:
> >
> > > The best time to ask questions on c.l.l is after you've done steps 1
> > > and 2. And show us what you came up with in those steps.
> >
> > I'll try and expand on this a bit.
> >
> > Last time, when I wrote the following, I was attempting to understand
> > what was happening during the macroexpansion.
> >
> > > PG-USER[20]> (LET ((FOO 'X))
> > > (MACROLET
> > > ((DEFAULT-VAR NIL `(T FOO)))
> > > (macroexpand '(EXPECT-1))))
> > > (EXPECT-1-LEMMA) ;
> > > T
> >
> > > ;;;It would seem that the macrolet is not being expanded within
> > > ;;;expect-1 like I thought it should be.
>
> Okay, so I think your immediate problem is that MACROLET defines a
> *local*, i.e. lexical macro binding. Since there's no reference to
> DEFAULT-VAR within the lexical scope of the MACROLET form, there's no
> way the local binding of DEFAULT-VAR is going to have any affect. (You
> seem to be expecting that the local definition of DEFAULT-VAR is going
> to affect the call to MACROEXPAND-1 *inside* the definition of
> EXPECT-1; that's not how it works.)
<nod> Yeah, I think I just figured this out... kinda. I was not fully
understanding The Post
<http://groups.google.com/groups?q=WITH-CONTEXT-1&hl=en&lr=&ie=UTF-8&oe=UTF-8&selm=R-Cdned1bP-Nt--iU-KYvA%40dls.net&rnum=1>.
I first discovered this here.
PG-USER[96]>
(defmacro try-macroexpanding-default-var (&environment env)
(let ((query (macroexpand '(default-var) env)))
`(format t "query => ~A~%" query)))
TRY-MACROEXPANDING-DEFAULT-VAR
PG-USER[97]>
(macrolet
((default-var () `(list t 'foo)))
(try-macroexpanding-default-var))
*** - EVAL: variable QUERY has no value
It was then that I noticed that the actual pattern being shown in The
Post was rather something like the following.
PG-USER[100]>
(defmacro try-macroexpanding-default-var (&environment env)
(let ((query (macroexpand '(default-var) env)))
(format t "query => ~A~%" query))) ; not actually returned
TRY-MACROEXPANDING-DEFAULT-VAR
PG-USER[101]>
(macrolet
((default-var () `(list t 'foo)))
(try-macroexpanding-default-var))
query => (LIST T 'FOO)
NIL
I obviously do not fully understand the interactions between MACROLET,
DEFMACRO, MACROEXPAND, &ENVIRONMENT, etc.
> More generally, I think you still need to pull back and show us 1)
> the code you want to be able to write and 2) the code you would have
> to write to do achive the desired result without the macro.
Fair enough.
> Maybe I'm just dense but I have no idea what a program that uses
> EXPECT is going to look like (1) and what it would do (2).
Okay, well forget about EXPECT. Let us just rephrase the question.
Suppose I have a function/macro, FOO, that takes a single required
parameter.
(foo 'x)
(foo 'y)
Supose I want to write a macro, WITH-FOO-DEFAULT, such that with the
following form
(with-foo-default (x 69)
(foo))
one gets the following after macroexpansion.
(let ((x 69))
(foo x))
Or
(with-foo-default (x 69)
(do-something-cool)
(do-something-awful)
(do-something-with (foo))
(do-something-with (foo 'y))
(do-something-with '(foo)))
gets expanded into
(let ((x 69))
(do-something-cool)
(do-something-awful)
(do-something-with (foo x))
(do-something-with (foo 'y))
(do-something-with '(foo)))
However,
PG-USER[108]> (foo)
*** - EVAL/APPLY: too few arguments given to FOO
1. Break PG-USER[109]>
This is what I was attempting to do. I was hoping that this context
sensitive macroexpansion described in The Post would be the key to
what I wanted but I've obviously missed a crucial point.
In previous posts, I was attempting to show various failed/suboptimal
attempts at achieving this. For example, an implementation of the
following is obvious to me
PG-USER[121]>
(defmacro with-foo-default ((var val) &body body)
`(let ((,var ,val))
(macrolet
((foo-it () `(foo ,',var)))
,@body)))
WITH-FOO-DEFAULT
PG-USER[122]>
(with-foo-default (x 69)
(foo-it))
69
PG-USER[123]>
However, I still confused on the following point
PG-USER[127]>
(macroexpand
`(with-foo-default (x 69)
(foo-it)))
(LET ((X 69)) (MACROLET ((FOO-IT NIL `(FOO X))) (FOO-IT))) ;
T
PG-USER[128]>
(LET ((X 69))
(MACROLET ((FOO-IT NIL `(FOO X)))
(macroexpand '(FOO-IT))))
(FOO-IT) ; How can I expand (FOO-IT)?
NIL
PG-USER[129]>
(LET ((X 69))
(macroexpand '(MACROLET ((FOO-IT NIL `(FOO X))) (FOO-IT))))
(MACROLET ((FOO-IT NIL `(FOO X))) (FOO-IT)) ; Not neither...
NIL
How can I futher expand the combination of the marco and its MACROLET?
I'd like to be able to see the final result of
(let ((x 69))
(foo x))
Or is this not possible?
> I suspect you're making this more complicated than it needs to be
> but without knowing what you're trying to do, at a high level, it's
> hard to say.
Yeah, I suspect the same thing. I can get nearly the same effect with
(defvar *default-foo-bar* nil)
(defun foo (&optional (bar *default-foo-bar*))
(if (nil bar)
(error "Gimme a foo-bar!~%")
(foo-lemma bar)))
(let ((*default-foo-bar* 69))
(foo)
(foo 'y))
(foo) ; 1. valid but will error
But it isn't the same. Doesn't use lexical scope. Not supplying
something (number 1 in the above comments) is still valid at compile
time.
From: Damien Kick
Subject: Re: Question about design, defmacro, macrolet, and &environment
Date:
Message-ID: <ovhe1hnnbl.fsf@email.mot.com>
Damien Kick <······@email.mot.com> writes:
> [...] This is what I was attempting to do. I was hoping that this
> context sensitive macroexpansion described in The Post would be the
> key to what I wanted but I've obviously missed a crucial point.
*This* is what I wanted to do.
PG-USER[151]>
(fmakunbound 'foo)
FOO
PG-USER[152]>
(defun foo-lemma (x) x)
FOO-LEMMA
PG-USER[153]>
(defmacro foo (x)
`(foo-lemma ,x))
FOO
PG-USER[154]>
(defmacro with-foo-default ((var val) &body body)
`(let ((,var ,val))
(macrolet
((foo (&optional (explicit-var nil explicit-var-supplied-p))
(if explicit-var-supplied-p
`(foo-lemma ,explicit-var)
`(foo-lemma ,',var))))
,@body)))
WITH-FOO-DEFAULT
PG-USER[155]>
(with-foo-default (x 69)
(foo))
*** - EVAL/APPLY: too few arguments given to FOO-LEMMA
1. Break PG-USER[156]>
PG-USER[157]>
(defmacro with-foo-default ((var val) &body body)
`(let ((,var ,val))
(macrolet
((foo (&optional (explicit-var nil explicit-var-supplied-p))
(if explicit-var-supplied-p
`(foo-lemma ,explicit-var)
`(foo-lemma ,',var))))
,@body)))
WITH-FOO-DEFAULT
PG-USER[158]>
(with-foo-default (x 69)
(foo))
69
PG-USER[159]> (foo 71)
71
PG-USER[160]> (foo)
*** - The macro FOO may not be called with 0 arguments: (FOO)
1. Break PG-USER[161]>
PG-USER[162]>
Now the only problem is that one can not
(funcall #'foo 69)
but must
(funcall #'foo-lemma 69)
Can't figure out how to do something like the following.
PG-USER[149]>
(defmacro with-foo-default ((var val) &body body)
`(let ((,var ,val))
(macrolet
((foo (&optional (var nil var-supplied-p))
(if var-supplied-p
`(foo ,',var) ; This won't work because it will
`(foo)))) ; recurse infinitely
,@body)))
WITH-FOO-DEFAULT
PG-USER[150]>
(with-foo-default (x 69)
(foo))
*** - Lisp stack overflow. RESET
Still confused on the following point, too.
> PG-USER[127]>
> (macroexpand
> `(with-foo-default (x 69)
> (foo-it)))
> (LET ((X 69)) (MACROLET ((FOO-IT NIL `(FOO X))) (FOO-IT))) ;
> T
> PG-USER[128]>
> (LET ((X 69))
> (MACROLET ((FOO-IT NIL `(FOO X)))
> (macroexpand '(FOO-IT))))
> (FOO-IT) ; How can I expand (FOO-IT)?
> NIL
> PG-USER[129]>
> (LET ((X 69))
> (macroexpand '(MACROLET ((FOO-IT NIL `(FOO X))) (FOO-IT))))
> (MACROLET ((FOO-IT NIL `(FOO X))) (FOO-IT)) ; Not neither...
> NIL
>
> How can I futher expand the combination of the marco and its MACROLET?
> I'd like to be able to see the final result of
>
> (let ((x 69))
> (foo x))
>
> Or is this not possible?
From: Peter Seibel
Subject: Re: Question about design, defmacro, macrolet, and &environment
Date:
Message-ID: <m3oevp3t8z.fsf@javamonkey.com>
Damien Kick <······@email.mot.com> writes:
> Still confused on the following point, too.
>
> > PG-USER[127]>
> > (macroexpand
> > `(with-foo-default (x 69)
> > (foo-it)))
> > (LET ((X 69)) (MACROLET ((FOO-IT NIL `(FOO X))) (FOO-IT))) ;
> > T
> > PG-USER[128]>
> > (LET ((X 69))
> > (MACROLET ((FOO-IT NIL `(FOO X)))
> > (macroexpand '(FOO-IT))))
> > (FOO-IT) ; How can I expand (FOO-IT)?
> > NIL
> > PG-USER[129]>
> > (LET ((X 69))
> > (macroexpand '(MACROLET ((FOO-IT NIL `(FOO X))) (FOO-IT))))
> > (MACROLET ((FOO-IT NIL `(FOO X))) (FOO-IT)) ; Not neither...
> > NIL
> >
> > How can I futher expand the combination of the marco and its MACROLET?
> > I'd like to be able to see the final result of
> >
> > (let ((x 69))
> > (foo x))
> >
> > Or is this not possible?
Not easily. I believe implementations may provide a way to recursively
expand all the macros in a form (they almost have to have such a thing
internally, in order to implement minimal compilation but may not
expose it via any public API). Failing that you need to write a code
walker.
-Peter
--
Peter Seibel ·····@javamonkey.com
Lisp is the red pill. -- John Fraser, comp.lang.lisp
From: Damien Kick
Subject: Re: Question about design, defmacro, macrolet, and &environment
Date:
Message-ID: <ovr80ed8zv.fsf@email.mot.com>
Peter Seibel <·····@javamonkey.com> writes:
> Damien Kick <······@email.mot.com> writes:
>
> > > [...] How can I futher expand the combination of the marco and
> > > its MACROLET? [...]
>
> Not easily. I believe implementations may provide a way to
> recursively expand all the macros in a form (they almost have to
> have such a thing internally, in order to implement minimal
> compilation but may not expose it via any public API). Failing that
> you need to write a code walker.
Actually, as was pointed out to me in e-mail responses to my posts,
and as is mentioned in the CLHS description of MACROEXPAND
<http://www.lispworks.com/reference/HyperSpec/Body/f_mexp_.htm#macroexpand>,
MACROEXPAND takes an optional environment object. This can be used in
conjunction with a DEFMACRO to capture the lexical environment, as
shown in the examples from the CLHS:
(defmacro expand (form &environment env)
(multiple-value-bind (expansion expanded-p)
(macroexpand form env)
`(values ',expansion ',expanded-p)))
Thanks to everyone for all the help/suggestions.
From: Nikodemus Siivola
Subject: Re: Question about design, defmacro, macrolet, and &environment
Date:
Message-ID: <boeldt$ihi$1@nyytiset.pp.htv.fi>
Damien Kick <······@email.mot.com> wrote:
> (with-foo-default (x 69)
> (foo))
>
> (let ((x 69))
> (foo x))
One wonders what you want to do with the X: do you wish it to be
visible as a binding in the body?
Eg:
(with-foo-default (x 42)
(foo)
(bar x))
? Assuming so, and that FOO is not a symbol from the COMMON-LISP
package, you almost had it right at the end:
> (defvar *default-foo-bar* nil)
> (defun foo (&optional (bar *default-foo-bar*))
> (if (nil bar)
> (error "Gimme a foo-bar!~%")
> (foo-lemma bar)))
>
> (let ((*default-foo-bar* 69))
> (foo)
> (foo 'y))
>
> (foo) ; 1. valid but will error
Just do that in the lexical scope:
(defmacro with-foo ((binding expr) &body forms)
`(let ((,binding ,expr))
(flet ((foo (&optional (opt ,binding))
(foo opt)))
,@forms)))
(macroexpand '(with-foo (x 42)
(bar x)
(foo)
(foo 69)))
=> (LET ((X 42))
(FLET ((FOO (&OPTIONAL (OPT X))
(FOO OPT)))
(BAR X)
(FOO) ;; <- if you *really* need to see (foo x) here, you (I think) need a code-walker
(FOO 69)))
What you're doing -- if I follow you correctly -- is quite
simple.
You seem to be getting tangled up because you're trying to deal with
two different issues at the same time: the scoping rules of the
various forms, and the way backquoting works.
The way I've managed so far is something like this:
1) When writing a macro, use backquote only in simple templates at first,
and handle more complex cases with regular list-manipulation functions.
Once the macro works, rewrite it with backquote if it wasn't like that
already -- and if it seems appropriate.
I'm too lazy to systematically practise "scales" as Peter Seibel suggested,
so I do periodical reviews instead. ;)
2) When writing a macro, unless I _know_ how a form in my macro relates
to scope I double-check in CLHS. Of course, sometimes I think I know,
but turns out I didn't really...
Cheers,
-- Nikodemus
From: Damien Kick
Subject: Re: Question about design, defmacro, macrolet, and &environment
Date:
Message-ID: <ovvfpqdhak.fsf@email.mot.com>
Nikodemus Siivola <······@random-state.net> writes:
> Just do that in the lexical scope:
>
> (defmacro with-foo ((binding expr) &body forms)
> `(let ((,binding ,expr))
> (flet ((foo (&optional (opt ,binding))
> (foo opt)))
> ,@forms)))
>
> (macroexpand '(with-foo (x 42)
> (bar x)
> (foo)
> (foo 69)))
>
>
> => (LET ((X 42))
> (FLET ((FOO (&OPTIONAL (OPT X))
> (FOO OPT)))
> (BAR X)
> (FOO) ;; <- if you *really* need to see (foo x) here [...]
> (FOO 69)))
>
> What you're doing -- if I follow you correctly -- is quite
> simple.
I like this much better than what I had in the previous post. I don't
really need to see (FOO X) in the macroexpansion; I just want (FOO) to
"see" it in the call. It had not even occurred to me to use FLET in
this way. Thank you for the suggestion.
From: Damien Kick
Subject: Re: Question about design, defmacro, macrolet, and &environment
Date:
Message-ID: <ovr809noov.fsf@email.mot.com>
Peter Seibel <·····@javamonkey.com> writes:
> [...] I have no idea what a program that uses EXPECT is going to
> look like (1) and what it would do (2).
FWIW, this is that with which I've been playing as an excuse to get
some more experience with Common Lisp. As always, I would appreciate
any feedback as c.l.l is the only place I've any chance to meet
lispniks.
;;;; First version of my excuse to play with CMU CL, using to
;;;; implement Don Libe's Expect. *SPAWN* is similiar to Expect's
;;;; "spawn_id". EXPECT-STRING is a stupified version of Expect's
;;;; "expect" as it does not allow for regular expressions. Also,
;;;; none of this allows for controlling multiple "spawn"s or
;;;; "expect"ing from more than one "spawn".
(defpackage "LONE-TRIP-INVESTMENTS"
(:nicknames "LTI")
(:use "COMMON-LISP" "EXTENSIONS")
(:export "WITH-SPAWN-ID" "WITH-SPAWN-STREAM" "*SPAWN*"
"*TRACE-EXPECT-STRING-LEMMA*" "SPAWN" "EXPECT-STRING"
"EXPECT-SOMEONE-SOMEWHERE-PROMPT"))
(in-package "LONE-TRIP-INVESTMENTS")
;;; Helper/utility functions; i.e. generally not exported but rather
;;; only used internally.
(declaim (inline synonym-value))
(defun synonym-value (symbol synonym value &key (test #'eql))
"If the value of the symbol is a synonym (or abbreviation) for some
other value, return the real value. Otherwise, just use the value of
the symbol."
;; Should any of this attempt to be 'setf'able? In other words,
;; somehow make use of generalized variables?
;; (setf (synonym-value #|something|#) #|something-else|#)
;; But I wouldn't know what that should look like.
(if (funcall test symbol synonym) value symbol))
;;; The rest of the stuff...
(defmacro with-spawn-process ((id exec-name . exec-args) &body code)
`(let ((,id (spawn ,exec-name ',exec-args)))
(unwind-protect
(progn ,@code)
(process-close ,id))))
(defmacro with-spawn-stream ((bi exec-name . exec-args) &body code)
(let ((id (gensym "SPAWN-PROCESS-")))
`(with-spawn-process (,id ,exec-name ,@exec-args)
(let ((,bi (process-pty ,id)))
,@code))))
(defvar *spawn* nil)
(defvar *trace-expect-string-lemma* nil)
(defun spawn (name &optional args)
"The CMU CL implementation of Don Libes spawn."
(run-program name args :wait nil :pty t :input t :output t :error t))
(defun expect-string (string &optional stream &key (echo t))
"Expects to find the literal string on the stream."
(labels
((lemma (string in end match echo pos)
(when *trace-expect-string-lemma*
(format t "expect-string lemma ~A~%"
(list string in end match echo pos)))
(unless (= pos end)
(let ((actual (read-char in))
(expect (char string pos)))
(write-char actual match)
(when echo
(write-char actual echo)
(force-output echo))
(let ((next (if (char= actual expect) (1+ pos) 0)))
(lemma string in end match echo next))))))
(values string
(let ((in (or stream (process-pty *spawn*)))
(end (length string))
(echo (synonym-value echo t *standard-output*)))
(with-output-to-string (match)
(lemma string in end match echo 0))))))
(defun expect-someone-somewhere-prompt (&optional stream)
"Expects to find Hashim's prompt on the MGTS server."
(let ((stream (or stream (process-pty *spawn*))))
(expect-string "tekelec:[" stream)
(expect-string "]" stream)
(expect-string "%" stream)))
(defun send (string &optional stream)
(let ((stream (or stream (process-pty *spawn*))))
(write-string string stream)
(force-output stream)
string))
;;; **********************************************************************
;;; Testing functions, variables, etc.
;;; **********************************************************************
(defvar *test-user-name* "someone")
(defvar *test-password* "something")
(declaim (inline make-adjustable-string))
(defun make-adjustable-string ()
(make-array '(0)
:element-type 'base-char :fill-pointer 0 :adjustable t))
(declaim (inline string-cat))
(defun string-cat (&rest args)
(apply #'concatenate 'string args))
(declaim (inline make-test-expect-string))
(defun make-test-expect-string ()
"telnet somewhere
Trying 136.182.32.250...
Connected to somewhere.
Escape character is '^]'.
SunOS 5.6
login: kick
Password: ")
(declaim (inline make-expected-match-string))
(defun make-expected-match-string ()
"telnet somewhere
Trying 136.182.32.250...
Connected to somewhere.
Escape character is '^]'.
SunOS 5.6
login:")
(declaim (inline make-expected-expect-string))
(defun make-expected-expect-string ()
"login:")
(defun call-expect-string-tester (f)
(let ((expect "login:"))
(with-input-from-string (phake-spawn-in (make-test-expect-string))
(multiple-value-bind (what-was-expected what-was-matched)
(funcall f expect phake-spawn-in)
(assert (string= what-was-expected
(make-expected-expect-string)))
(assert (string= what-was-matched
(make-expected-match-string)))))))
(defun test-expect-string ()
(call-expect-string-tester
#'(lambda (string stream)
(expect-string string stream :echo nil)))
(assert (string= (with-output-to-string (phake-echo-stream)
(call-expect-string-tester
#'(lambda (string stream)
(expect-string string stream
:echo phake-echo-stream))))
(make-expected-match-string)))
(call-expect-string-tester
#'(lambda (string stream)
(expect-string string stream)))
t)
(defun test-telnet-somewhere (&optional (user-name *test-user-name*)
(password *test-password*))
(with-spawn-process (*spawn* "telnet" "somewhere")
(expect-string "login:")
(send (string-cat (string user-name) (string #\Newline)))
(expect-string "assword:")
(send (string-cat (string password) (string #\Newline)))
(expect-someone-somewhere-prompt)
(send (string-cat "ls" (string #\Newline)))
(expect-someone-somewhere-prompt)
t))
;;;; What follows is the diff of the first draft and what I changed to
;;;; support defaults differently.
--- lti.lisp.cll.bak Thu Nov 13 17:02:44 2003
+++ lti.lisp.cll Sat Nov 15 18:14:12 2003
@@ -1,16 +1,14 @@
-;;;; First version of my excuse to play with CMU CL, using to
-;;;; implement Don Libe's Expect. *SPAWN* is similiar to Expect's
-;;;; "spawn_id". EXPECT-STRING is a stupified version of Expect's
-;;;; "expect" as it does not allow for regular expressions. Also,
-;;;; none of this allows for controlling multiple "spawn"s or
-;;;; "expect"ing from more than one "spawn".
+;;;; This is a second version that removes *SPAWN* and any code on
+;;;; which it depended. Instead, I use WITH-SPAWN-STREAM to create
+;;;; local functions that supply the default values of STREAM for
+;;;; EXPECT and SPAWN. Somehow, this seems more cl-style (&optional
+;;;; flame-war-about-what-is-cl-style) to me than using *SPAWN*.
(defpackage "LONE-TRIP-INVESTMENTS"
(:nicknames "LTI")
(:use "COMMON-LISP" "EXTENSIONS")
- (:export "WITH-SPAWN-ID" "WITH-SPAWN-STREAM" "*SPAWN*"
- "*TRACE-EXPECT-STRING-LEMMA*" "SPAWN" "EXPECT-STRING"
- "EXPECT-SOMEONE-SOMEWHERE-PROMPT"))
+ (:export "WITH-SPAWN-ID" "WITH-SPAWN-STREAM" "*TRACE-EXPECT-STRING-LEMMA*"
+ "SPAWN" "EXPECT-STRING" "EXPECT-SOMEONE-SOMEWHERE-PROMPT"))
(in-package "LONE-TRIP-INVESTMENTS")
@@ -40,19 +38,31 @@
(let ((id (gensym "SPAWN-PROCESS-")))
`(with-spawn-process (,id ,exec-name ,@exec-args)
(let ((,bi (process-pty ,id)))
- ,@code))))
+ (flet ((expect-string (string &optional (stream ,bi) &key (echo t))
+ (expect-string string stream :echo echo))
+ (send (string &optional (stream ,bi))
+ (send string stream)))
+ ,@code)))))
-(defvar *spawn* nil)
(defvar *trace-expect-string-lemma* nil)
+(declaim (inline spawn))
(defun spawn (name &optional args)
"The CMU CL implementation of Don Libes spawn."
(run-program name args :wait nil :pty t :input t :output t :error t))
-(defun expect-string (string &optional stream &key (echo t))
+;; I wonder how long it will take someone on c.l.l to propose a
+;; version of EXPECT-STRING that uses LOOP. As I've been introduced
+;; to CL by Graham's _ANSI Common Lisp_, I have been introduced to an
+;; anti-LOOP cl-view and appreciate having been introduced to a
+;; pro-LOOP view by c.l.l. However, I'm still not comfortable with
+;; LOOP and wish I knew of a tutorial for it.
+(defun expect-string (string stream &key (echo t))
"Expects to find the literal string on the stream."
(labels
((lemma (string in end match echo pos)
+ ;; If/when CMU CL allows one to trace local functions, I will
+ ;; be able to remove the following WHEN.
(when *trace-expect-string-lemma*
(format t "expect-string lemma ~A~%"
(list string in end match echo pos)))
@@ -66,24 +76,21 @@
(let ((next (if (char= actual expect) (1+ pos) 0)))
(lemma string in end match echo next))))))
(values string
- (let ((in (or stream (process-pty *spawn*)))
- (end (length string))
+ (let ((end (length string))
(echo (synonym-value echo t *standard-output*)))
(with-output-to-string (match)
- (lemma string in end match echo 0))))))
+ (lemma string stream end match echo 0))))))
-(defun expect-someone-somewhere-prompt (&optional stream)
+(defun expect-someone-somewhere-prompt (stream)
"Expects to find Hashim's prompt on the MGTS server."
- (let ((stream (or stream (process-pty *spawn*))))
- (expect-string "tekelec:[" stream)
- (expect-string "]" stream)
- (expect-string "%" stream)))
-
-(defun send (string &optional stream)
- (let ((stream (or stream (process-pty *spawn*))))
- (write-string string stream)
- (force-output stream)
- string))
+ (expect-string "tekelec:[" stream)
+ (expect-string "]" stream)
+ (expect-string "%" stream))
+
+(defun send (string stream)
+ (write-string string stream)
+ (force-output stream)
+ string)
;;; **********************************************************************
;;; Testing functions, variables, etc.
@@ -157,12 +164,19 @@
(defun test-telnet-somewhere (&optional (user-name *test-user-name*)
(password *test-password*))
- (with-spawn-process (*spawn* "telnet" "somewhere")
+ (with-spawn-stream (stream "telnet" "somewhere")
(expect-string "login:")
(send (string-cat (string user-name) (string #\Newline)))
(expect-string "assword:")
(send (string-cat (string password) (string #\Newline)))
- (expect-someone-somewhere-prompt)
+ ;; It would be nice to be able to somehow get WITH-SPAWN-STREAM to
+ ;; generate FLET versions of functions, like
+ ;; EXPECT-SOMEONE-SOMEWHERE-PROMPT, that take STREAM as an
+ ;; optional parameter, as is done with EXPECT and SEND. I wonder
+ ;; how much extra work would be required to accomplish this for
+ ;; any function, FOO. Or would it be better to do this with
+ ;; closures?
+ (expect-someone-somewhere-prompt stream)
(send (string-cat "ls" (string #\Newline)))
- (expect-someone-somewhere-prompt)
+ (expect-someone-somewhere-prompt stream)
t))
;;;; Applying these changes, I get a second version that removes
;;;; *SPAWN* and any code on which it depended. Instead, I use
;;;; WITH-SPAWN-STREAM to create local functions that supply the
;;;; default values of STREAM for EXPECT and SPAWN. Somehow, this
;;;; seems more cl-style (&optional flame-war-about-what-is-cl-style)
;;;; to me than using *SPAWN*.
(defpackage "LONE-TRIP-INVESTMENTS"
(:nicknames "LTI")
(:use "COMMON-LISP" "EXTENSIONS")
(:export "WITH-SPAWN-ID" "WITH-SPAWN-STREAM" "*TRACE-EXPECT-STRING-LEMMA*"
"SPAWN" "EXPECT-STRING" "EXPECT-SOMEONE-SOMEWHERE-PROMPT"))
(in-package "LONE-TRIP-INVESTMENTS")
;;; Helper/utility functions; i.e. generally not exported but rather
;;; only used internally.
(declaim (inline synonym-value))
(defun synonym-value (symbol synonym value &key (test #'eql))
"If the value of the symbol is a synonym (or abbreviation) for some
other value, return the real value. Otherwise, just use the value of
the symbol."
;; Should any of this attempt to be 'setf'able? In other words,
;; somehow make use of generalized variables?
;; (setf (synonym-value #|something|#) #|something-else|#)
;; But I wouldn't know what that should look like.
(if (funcall test symbol synonym) value symbol))
;;; The rest of the stuff...
(defmacro with-spawn-process ((id exec-name . exec-args) &body code)
`(let ((,id (spawn ,exec-name ',exec-args)))
(unwind-protect
(progn ,@code)
(process-close ,id))))
(defmacro with-spawn-stream ((bi exec-name . exec-args) &body code)
(let ((id (gensym "SPAWN-PROCESS-")))
`(with-spawn-process (,id ,exec-name ,@exec-args)
(let ((,bi (process-pty ,id)))
(flet ((expect-string (string &optional (stream ,bi) &key (echo t))
(expect-string string stream :echo echo))
(send (string &optional (stream ,bi))
(send string stream)))
,@code)))))
(defvar *trace-expect-string-lemma* nil)
(declaim (inline spawn))
(defun spawn (name &optional args)
"The CMU CL implementation of Don Libes spawn."
(run-program name args :wait nil :pty t :input t :output t :error t))
;; I wonder how long it will take someone on c.l.l to propose a
;; version of EXPECT-STRING that uses LOOP. As I've been introduced
;; to CL by Graham's _ANSI Common Lisp_, I have been introduced to an
;; anti-LOOP cl-view and appreciate having been introduced to a
;; pro-LOOP view by c.l.l. However, I'm still not comfortable with
;; LOOP and wish I knew of a tutorial for it.
(defun expect-string (string stream &key (echo t))
"Expects to find the literal string on the stream."
(labels
((lemma (string in end match echo pos)
;; If/when CMU CL allows one to trace local functions, I will
;; be able to remove the following WHEN.
(when *trace-expect-string-lemma*
(format t "expect-string lemma ~A~%"
(list string in end match echo pos)))
(unless (= pos end)
(let ((actual (read-char in))
(expect (char string pos)))
(write-char actual match)
(when echo
(write-char actual echo)
(force-output echo))
(let ((next (if (char= actual expect) (1+ pos) 0)))
(lemma string in end match echo next))))))
(values string
(let ((end (length string))
(echo (synonym-value echo t *standard-output*)))
(with-output-to-string (match)
(lemma string stream end match echo 0))))))
(defun expect-someone-somewhere-prompt (stream)
"Expects to find Hashim's prompt on the MGTS server."
(expect-string "tekelec:[" stream)
(expect-string "]" stream)
(expect-string "%" stream))
(defun send (string stream)
(write-string string stream)
(force-output stream)
string)
;;; **********************************************************************
;;; Testing functions, variables, etc.
;;; **********************************************************************
(defvar *test-user-name* "someone")
(defvar *test-password* "something")
(declaim (inline make-adjustable-string))
(defun make-adjustable-string ()
(make-array '(0)
:element-type 'base-char :fill-pointer 0 :adjustable t))
(declaim (inline string-cat))
(defun string-cat (&rest args)
(apply #'concatenate 'string args))
(declaim (inline make-test-expect-string))
(defun make-test-expect-string ()
"telnet somewhere
Trying 136.182.32.250...
Connected to somewhere.
Escape character is '^]'.
SunOS 5.6
login: kick
Password: ")
(declaim (inline make-expected-match-string))
(defun make-expected-match-string ()
"telnet somewhere
Trying 136.182.32.250...
Connected to somewhere.
Escape character is '^]'.
SunOS 5.6
login:")
(declaim (inline make-expected-expect-string))
(defun make-expected-expect-string ()
"login:")
(defun call-expect-string-tester (f)
(let ((expect "login:"))
(with-input-from-string (phake-spawn-in (make-test-expect-string))
(multiple-value-bind (what-was-expected what-was-matched)
(funcall f expect phake-spawn-in)
(assert (string= what-was-expected
(make-expected-expect-string)))
(assert (string= what-was-matched
(make-expected-match-string)))))))
(defun test-expect-string ()
(call-expect-string-tester
#'(lambda (string stream)
(expect-string string stream :echo nil)))
(assert (string= (with-output-to-string (phake-echo-stream)
(call-expect-string-tester
#'(lambda (string stream)
(expect-string string stream
:echo phake-echo-stream))))
(make-expected-match-string)))
(call-expect-string-tester
#'(lambda (string stream)
(expect-string string stream)))
t)
(defun test-telnet-somewhere (&optional (user-name *test-user-name*)
(password *test-password*))
(with-spawn-stream (stream "telnet" "somewhere")
(expect-string "login:")
(send (string-cat (string user-name) (string #\Newline)))
(expect-string "assword:")
(send (string-cat (string password) (string #\Newline)))
;; It would be nice to be able to somehow get WITH-SPAWN-STREAM to
;; generate FLET versions of functions, like
;; EXPECT-SOMEONE-SOMEWHERE-PROMPT, that take STREAM as an
;; optional parameter, as is done with EXPECT and SEND. I wonder
;; how much extra work would be required to accomplish this for
;; any function, FOO. Or would it be better to do this with
;; closures?
(expect-someone-somewhere-prompt stream)
(send (string-cat "ls" (string #\Newline)))
(expect-someone-somewhere-prompt stream)
t))
From: Damien Kick
Subject: Re: Question about design, defmacro, macrolet, and &environment
Date:
Message-ID: <ovn0avoly8.fsf@email.mot.com>
Well, some more style questions...
;; This was the first version. It uses tail recursion. This is a
;; very Scheme-ish style. I wonder how long it will take someone on
;; c.l.l to propose a version of EXPECT-STRING that uses LOOP. As
;; I've been introduced to CL by Graham's _ANSI Common Lisp_, I have
;; been introduced to an anti-LOOP cl-view and appreciate having been
;; introduced to a pro-LOOP view by c.l.l. However, I'm still not
;; comfortable with LOOP and wish I knew of a tutorial for it.
(defun expect-string--rec (string stream &key (echo t))
(labels ((lemma (string in end match echo pos)
;; If/when CMU CL allows one to trace local functions, I
;; will be able to remove the following WHEN.
(when *trace-expect-string-lemma*
(format t "expect-string lemma ~A~%"
(list string in end match echo pos)))
(unless (= pos end)
(let ((actual (read-char in))
(expect (char string pos)))
(write-char actual match)
(when echo
(write-char actual echo)
(force-output echo))
(let ((next (if (char= actual expect)
(1+ pos)
0)))
(lemma string in end match echo next))))))
(values string
(let ((end (length string))
(echo (synonym-value echo t *standard-output*)))
(with-output-to-string (match)
(lemma string stream end match echo 0))))))
;; I've rewritten the function to use DO instead of recursion.
(defun expect-string (string stream &key (echo t))
"Expects to find the literal STRING on the STREAM."
(values string
(let ((end (length string))
(echo (synonym-value echo t *standard-output*)))
(with-output-to-string (match)
(do ((actual)
(pos 0 (if (char= actual (char string pos))
(1+ pos)
0)))
((= pos end))
;; Okay, I know that I shouldn't be bothered by the
;; SETQ, by I still am. So, I try a couple different
;; versions. Are any of the variations worth the
;; effort? Probably not.
(setq actual (read-char stream))
(write-char actual match)
(when echo
(write-char actual echo)
(force-output echo)))))))
;; Remove the need for the explicit SETQ by doing all of the work in
;; the step forms. However, now we need to use EQL to test the actual
;; character read against the expected character because ACTUAL will
;; be NIL the first time through.
(defun expect-string-1 (string stream &key (echo t))
(values string
(let ((end (length string))
(echo (synonym-value echo t *standard-output*)))
(with-output-to-string (match)
;; Using the LAMBDA form allows me to achieve the side
;; effects as part of the step form of ACTUAL.
(do ((actual nil (funcall #'(lambda (x)
(write-char x match)
(when echo
(write-char x echo)
(force-output echo))
x)
(read-char stream)))
;; Because ACTUAL will be NIL in the first pass, we
;; use EQL instead of CHAR=. Is this potentially
;; slower?
(pos 0 (if (eql actual (char string pos))
(1+ pos)
0)))
((= pos end)))))))
;; Uses DO* instead of DO. Because of this, we know that ACTUAL will
;; be assigned before the test performed in the step form of POS, and
;; so we do not need to handle a case in which ACTUAL may be NIL. Of
;; course, I've read that DO* is frequently considered to be bad
;; style, and so I want to be suspisious about it. However, it kind
;; of doesn't really look all that bad. What am I missing? Would a
;; LOOP version be better than all of these?
(defun expect-string-2 (string stream &key (echo t))
(values string
(let ((end (length string))
(echo (synonym-value echo t *standard-output*)))
(with-output-to-string (match)
;; Again, use the LAMBDA form to achieve the side
;; effects in the step form.
(do* ((actual nil (funcall #'(lambda (x)
(write-char x match)
(when echo
(write-char x echo)
(force-output echo))
x)
(read-char stream)))
;; Because we know that ACTUAL will be assigned
;; before this point, we can use CHAR= as ACTUAL
;; will never be NIL.
(pos 0 (if (char= actual (char string pos))
(1+ pos)
0)))
((= pos end)))))))
From: Edi Weitz
Subject: Re: Question about design, defmacro, macrolet, and &environment
Date:
Message-ID: <873ccniynf.fsf@bird.agharta.de>
On 16 Nov 2003 18:52:47 -0600, Damien Kick <······@email.mot.com> wrote:
> As I've been introduced to CL by Graham's _ANSI Common Lisp_, I have
> been introduced to an anti-LOOP cl-view and appreciate having been
> introduced to a pro-LOOP view by c.l.l. However, I'm still not
> comfortable with LOOP and wish I knew of a tutorial for it.
Have you tried these?
<http://www.ai.sri.com/~pkarp/loop.html>
<http://www.gigamonkeys.com/book/appendix-loop-for-black-belts.html>
Edi.
From: Damien Kick
Subject: Re: Question about design, defmacro, macrolet, and &environment
Date:
Message-ID: <ovbrra4kf6.fsf@email.mot.com>
Edi Weitz <···@agharta.de> writes:
> Have you tried these?
>
> <http://www.ai.sri.com/~pkarp/loop.html>
> <http://www.gigamonkeys.com/book/appendix-loop-for-black-belts.html>
Thanks for the links. I'll definitely give them a look.
Incorporating suggestions sent in a very helpful e-mail from Paul
Foley (any errors are my own), in which he also gave me a version to
study that uses LOOP, I've got the following two iterative
implementations of EXPECT-STRING. IMO, the code is quite a bit less
convoluted when it uses echo streams and broadcast streams.
(defun expect-string (string stream &key (echo t))
"Expects to find STRING on STREAM."
(values string
(with-output-to-string (match)
(let ((io (make-echo-stream
stream
(let ((out (synonym-value
echo t *standard-output*)))
(if out
(make-broadcast-stream match out)
match)))))
(do ((pos 0 (if (char= (read-char io) (char string pos))
(1+ pos)
0)))
((= pos (length string))))))))
(defun expect-string--loop (string stream &key (echo t))
"Expects to find STRING on STREAM."
(values string
(with-output-to-string (match)
;; Is it good/bad style to use keywords--e.g. :with,
;; :until, etc.--to avoid interning loop sub-commands
;; as symbols?
(loop :with io = (make-echo-stream
stream
(let ((out (synonym-value
echo t *standard-output*)))
(if out
(make-broadcast-stream match out)
match)))
:with pos = 0
:until (= pos (length string)) :do
(setq pos (if (char= (read-char io) (char string pos))
(1+ pos)
0))))))
Somehow, I don't think that this is a good example to highlight LOOP,
as the DO version has nearly the same shape; i.e. the LOOP version
does not seem to add anything in this example. Of course, I'm only
beginning to try and learn LOOP, and I'm sure there are other examples
that do show its power. I just haven't encountered them yet.
Just curious. This reads forever until it gets the expected string? at
which point it returns two values, the string and effectively a copy of
the same string? I may have missed something. I know you are just asking
about loop, but it is hard to suggest another version without
understanding the the first version.
kenny
Damien Kick wrote:
> Well, some more style questions...
>
> ;; This was the first version. It uses tail recursion. This is a
> ;; very Scheme-ish style. I wonder how long it will take someone on
> ;; c.l.l to propose a version of EXPECT-STRING that uses LOOP. As
> ;; I've been introduced to CL by Graham's _ANSI Common Lisp_, I have
> ;; been introduced to an anti-LOOP cl-view and appreciate having been
> ;; introduced to a pro-LOOP view by c.l.l. However, I'm still not
> ;; comfortable with LOOP and wish I knew of a tutorial for it.
> (defun expect-string--rec (string stream &key (echo t))
> (labels ((lemma (string in end match echo pos)
> ;; If/when CMU CL allows one to trace local functions, I
> ;; will be able to remove the following WHEN.
> (when *trace-expect-string-lemma*
> (format t "expect-string lemma ~A~%"
> (list string in end match echo pos)))
> (unless (= pos end)
> (let ((actual (read-char in))
> (expect (char string pos)))
> (write-char actual match)
> (when echo
> (write-char actual echo)
> (force-output echo))
> (let ((next (if (char= actual expect)
> (1+ pos)
> 0)))
> (lemma string in end match echo next))))))
> (values string
> (let ((end (length string))
> (echo (synonym-value echo t *standard-output*)))
> (with-output-to-string (match)
> (lemma string stream end match echo 0))))))
>
> ;; I've rewritten the function to use DO instead of recursion.
> (defun expect-string (string stream &key (echo t))
> "Expects to find the literal STRING on the STREAM."
> (values string
> (let ((end (length string))
> (echo (synonym-value echo t *standard-output*)))
> (with-output-to-string (match)
> (do ((actual)
> (pos 0 (if (char= actual (char string pos))
> (1+ pos)
> 0)))
> ((= pos end))
> ;; Okay, I know that I shouldn't be bothered by the
> ;; SETQ, by I still am. So, I try a couple different
> ;; versions. Are any of the variations worth the
> ;; effort? Probably not.
> (setq actual (read-char stream))
> (write-char actual match)
> (when echo
> (write-char actual echo)
> (force-output echo)))))))
>
> ;; Remove the need for the explicit SETQ by doing all of the work in
> ;; the step forms. However, now we need to use EQL to test the actual
> ;; character read against the expected character because ACTUAL will
> ;; be NIL the first time through.
> (defun expect-string-1 (string stream &key (echo t))
> (values string
> (let ((end (length string))
> (echo (synonym-value echo t *standard-output*)))
> (with-output-to-string (match)
> ;; Using the LAMBDA form allows me to achieve the side
> ;; effects as part of the step form of ACTUAL.
> (do ((actual nil (funcall #'(lambda (x)
> (write-char x match)
> (when echo
> (write-char x echo)
> (force-output echo))
> x)
> (read-char stream)))
> ;; Because ACTUAL will be NIL in the first pass, we
> ;; use EQL instead of CHAR=. Is this potentially
> ;; slower?
> (pos 0 (if (eql actual (char string pos))
> (1+ pos)
> 0)))
> ((= pos end)))))))
>
> ;; Uses DO* instead of DO. Because of this, we know that ACTUAL will
> ;; be assigned before the test performed in the step form of POS, and
> ;; so we do not need to handle a case in which ACTUAL may be NIL. Of
> ;; course, I've read that DO* is frequently considered to be bad
> ;; style, and so I want to be suspisious about it. However, it kind
> ;; of doesn't really look all that bad. What am I missing? Would a
> ;; LOOP version be better than all of these?
> (defun expect-string-2 (string stream &key (echo t))
> (values string
> (let ((end (length string))
> (echo (synonym-value echo t *standard-output*)))
> (with-output-to-string (match)
> ;; Again, use the LAMBDA form to achieve the side
> ;; effects in the step form.
> (do* ((actual nil (funcall #'(lambda (x)
> (write-char x match)
> (when echo
> (write-char x echo)
> (force-output echo))
> x)
> (read-char stream)))
> ;; Because we know that ACTUAL will be assigned
> ;; before this point, we can use CHAR= as ACTUAL
> ;; will never be NIL.
> (pos 0 (if (char= actual (char string pos))
> (1+ pos)
> 0)))
> ((= pos end)))))))
--
http://tilton-technology.com
Why Lisp? http://alu.cliki.net/RtL%20Highlight%20Film
Your Project Here! http://alu.cliki.net/Industry%20Application
From: Damien Kick
Subject: Re: Question about design, defmacro, macrolet, and &environment
Date:
Message-ID: <ovisliok8y.fsf@email.mot.com>
Kenny Tilton <·······@nyc.rr.com> writes:
> Kenny Tilton wrote:
> > Just curious. This reads forever until it gets the expected string?
> > at which point it returns two values, the string and effectively a
> > copy of the same string? I may have missed something. I know you are
> > just asking about loop, but it is hard to suggest another version
> > without understanding the the first version.
It returns the string that was expected and the "buffer" that was read
while trying to find the expected string.
* (with-input-from-string (stream "ab123c")
(expect-string "123" stream :echo nil))
"123"
"ab123"
*
It is supposed to be something of a toy CL version of Don Libe's
Expect <http://expect.nist.gov/>. If you look at my previous posts in
this thread, you'll find I'm using EXPECT-STRING as part of some code
to telnet into a UNIX box.
[····@gsdapp01 2gtt/lti]% cmucl
CMU Common Lisp 18e, running on gsdapp01
With core: /usr/test/awo/user_work/kick/sparc-sun-solaris2.6/lib/cmucl/lib/lisp.core
Dumped on: Tue, 2003-04-08 13:23:10-05:00 on achat
See <http://www.cons.org/cmucl/> for support information.
Loaded subsystems:
Python 1.1, target SPARCstation/Solaris 2
CLOS 18e (based on PCL September 16 92 PCL (f))
* (load "lti.sparcf")
; Loading #p"/usr/vob/2gtt/lti/lti.sparcf".
T
* (lti::test-telnet-somewhere)
Trying 136.182.32.250...
Connected to somewhere.
Escape character is '^]'.
SunOS 5.6
login: someone
Password:
Last login: Thu Nov 13 16:27:59 from 10.17.193.21
Sun Microsystems Inc. SunOS 5.6 Generic August 1997
tekelec:[/tekelec/users/someone] 1 % ls
[... directory contents snipped ...]
tekelec:[/tekelec/users/someone] 2 %
T
* (quit)
[····@gsdapp01 2gtt/lti]%
I'm using EXPECT-STRING to wait for "login:" and then sending
"someone". The first return value is actually probably not useful.
Maybe I should not return it (or make it the second value). However,
the second return value is for if you need to do more processing on
the stuff that proceeded the expected string.
;; Assuming we've sent the "ls" and are waiting for the prompt
;; to come back to us.
(multiple-value-bind (string buffer)
(expect-string prompt stream :echo nil)
(count-files-in-ls buffer))
Of course, it would probably be more useful to not have PROMPT
included in BUFFER.
> what will it be on the first time thru? what you are thinking is that
> this won't work:
>
>
> (do ((actual (read-char)(read-char))
> (pos (if (char= actual ...etc
>
> because actual is an undeclared variable (not because something has
> not happened yet).
>
>
> do* would make actual visible to the second do binding, but if you have:
>
> (do ((actual nil (read-char))
> (pos (if (char= actual ...etc
>
> you'll still have a null actual on the first pass.
What I had was actually
(do* ((actual nil (read-char ...))
(pos 0 (if (char= actual ...) ...)))
...)
I don't the IF until after the step form for ACTUAL has been
evaluated. The init form for POS is 0.
> ps. have you tested on?:
>
> (with-input-from-string (i "oooops")
> (x-s-loop "oops" i))
Unfortunately, the DO version I posted without the SETQ does one too
many READ-CHARs. The version with DO* works just fine.
* (with-input-from-string (stream "ooooops")
(expect-string-2 "oops" stream))
ooooops
"oops"
"ooooops"
*
Damien Kick wrote:
> Kenny Tilton <·······@nyc.rr.com> writes:
> What I had was actually
>
> (do* ((actual nil (read-char ...))
> (pos 0 (if (char= actual ...) ...)))
> ...)
>
> I don't the IF until after the step form for ACTUAL has been
> evaluated. The init form for POS is 0.
Indeed. Not sure how I screwed that up.
>
>
>>ps. have you tested on?:
>>
>> (with-input-from-string (i "oooops")
>> (x-s-loop "oops" i))
>
>
> Unfortunately, the DO version I posted without the SETQ does one too
> many READ-CHARs. The version with DO* works just fine.
>
> * (with-input-from-string (stream "ooooops")
> (expect-string-2 "oops" stream))
> ooooops
> "oops"
> "ooooops"
> *
You can run, but you cannot hide:
(with-input-from-string (i "oooops")
(expect-string-2 "ooops" i))
The point is that when you decide you do not have a match you just reset
pos to zero and start matching from there. but you might at that point
be well into a stream which contains a match starting just after the
rejected starting pos of the failed search.
kt
--
http://tilton-technology.com
Why Lisp? http://alu.cliki.net/RtL%20Highlight%20Film
Your Project Here! http://alu.cliki.net/Industry%20Application
From: Damien Kick
Subject: Re: Question about design, defmacro, macrolet, and &environment
Date:
Message-ID: <ovllqdttzr.fsf@email.mot.com>
Kenny Tilton <·······@nyc.rr.com> writes:
> You can run, but you cannot hide:
>
> (with-input-from-string (i "oooops")
> (expect-string-2 "ooops" i))
>
> The point is that when you decide you do not have a match you just
> reset pos to zero and start matching from there. but you might at
> that point be well into a stream which contains a match starting
> just after the rejected starting pos of the failed search.
Doh! You got me <grimace>. <grumble how-many-times =
"2">Backtracking</grumble>. My first EXPECT-STRING worked just fine
for my simple needs of finding "login:", "password:", etc, so I was
avoiding the bigger issue for the moment.
Don Libe's Expect allows one to use regular expressions with its
"expect". For example
expect -re "a*"
I didn't want to deal with a full-blown version of my CL expect, which
would've also handled Kenny's "oooops" example, just yet because I
haven't found a satisfactory regexp library (I'll explain more in a
bit), I didn't need a full blown version to handle logging into a box,
and I didn't want to start out by writing yet another regexp library
(and doing it less effectively than those that have come before me).
The problem I have with all of the prefab regexp libraries that I've
found on the net so far is that the all seem to match regexps against
strings. However, I'm dealing with streams. I don't want a match to
fail if the required characters have not made it across the network
yet nor do I want to have to restart a regexp search from the
beginning after getting another character, e.g.
;; Pseudo-lisp (probably has errors) and my first LOOP
(loop :with buffer = (make-array #| fill pointer stuff |#)
:with matched-p = nil
:until matched-p :do
;; Is there an implicit PROGN here? I'll have to look.
(progn
(vector-push-extend (read-char io) buffer)
(when (regexp-match pattern buffer)
(setq matched-p t))))
Of course, maybe I should just start here and supply a more efficient
algorithm later.
Does anyone know of a regexp library for CL that would allow me to do
something like
(regexp-match pattern stream)
in which either STREAM or REGEXP-MATCH would do the buffering in such
a way to allow a match-in-progress to keep its current position while
looking to get more characters from STREAM if its runs off the end of
the current buffer? Is there some kind of buffering stream that
supports more than a single character peek? Nothing in the stream
dictionary
<http://www.lispworks.com/reference/HyperSpec/Body/c_stream.htm> seems
to fit the bill. I would also be cool if PATTERN was specified via
some kind of sexp notation, in a fashion simililar to META
<http://www.cliki.net/Meta>. <laugh> Not like I'm asking for much,
though. Should I simply start with META and extend it to do the
buffering that I need as well as getting it to handle full regular
expressions? Or has someone already done something similiar and I
just haven't found it?
Damien Kick <······@email.mot.com> writes:
> Does anyone know of a regexp library for CL that would allow me to do
> something like
>
> (regexp-match pattern stream)
>
> in which either STREAM or REGEXP-MATCH would do the buffering in such
> a way to allow a match-in-progress to keep its current position while
> looking to get more characters from STREAM if its runs off the end of
> the current buffer? Is there some kind of buffering stream that
> supports more than a single character peek? Nothing in the stream
> dictionary
> <http://www.lispworks.com/reference/HyperSpec/Body/c_stream.htm> seems
> to fit the bill.
I don't see why you'd need to peek at all, for what you're doing.
Just use plain old READ-CHAR, and let it hang if there's more coming
over the line. Extending something like CL-PPCRE to do that couldn't
be too hard.
> I would also be cool if PATTERN was specified via
> some kind of sexp notation, in a fashion simililar to META
> <http://www.cliki.net/Meta>.
BTW, CL-PPCRE supports an sexp syntax, in addition to the traditional
line noise one.
--
/|_ .-----------------------.
,' .\ / | No to Imperialist war |
,--' _,' | Wage class war! |
/ / `-----------------------'
( -. |
| ) |
(`-. '--.)
`. )----'
From: Damien Kick
Subject: Re: Question about design, defmacro, macrolet, and &environment
Date:
Message-ID: <ovhe11tgw4.fsf@email.mot.com>
···@fallingrocks.OCF.Berkeley.EDU (Thomas F. Burdick) writes:
> I don't see why you'd need to peek at all, for what you're doing.
> Just use plain old READ-CHAR, and let it hang if there's more coming
> over the line. [...]
Well, I mentioned peek because of the following from the essay on META
<http://home.pipeline.com/~hbaker1/Prag-Parse.html>, the only parsing
library of which I am aware that already handles streams as well as
strings and used a sexp syntax.
Before we can delve further into match and match-type, we must
make clear what it is we are trying to parse--whether it be a
Common Lisp character stream, a character string, or a Lisp
list. If the source of the text we are trying to parse is an
internal source, like a character string or a Lisp list, then
we are in a position to be able to back up. If the source is a
standard Common Lisp character stream, however, then we can
only look ("peek") one character ahead.
I was hoping that I would be able to use META with a buffering stream
so that META could look ("peek") more than one character ahead with
much of the internal framework unchanged.
> [...] Extending something like CL-PPCRE to do that couldn't be too
> hard.
>
> Damien Kick <······@email.mot.com> writes:
>
> > I would also be cool if PATTERN was specified via
> > some kind of sexp notation, in a fashion simililar to META
> > <http://www.cliki.net/Meta>.
>
> BTW, CL-PPCRE supports an sexp syntax, in addition to the traditional
> line noise one.
Well, that would be better than trying to write my own from scratch.
It definitely seems to be a high quality implementation, if its
web-site is any indication. Since it also supports a sexp syntax, it
looks like the only thing missing from my wish-list are functions that
work on streams as well as strings. I had glanced at CL-PPCRE but had
managed to miss the sexp support. Thanks for the suggestion.
From: Edi Weitz
Subject: Re: Question about design, defmacro, macrolet, and &environment
Date:
Message-ID: <87brr8fuxy.fsf@bird.agharta.de>
On 18 Nov 2003 12:24:40 -0600, Damien Kick <······@email.mot.com> wrote:
> Don Libe's Expect allows one to use regular expressions with its
> "expect". For example
>
> expect -re "a*"
>
> [...] The problem I have with all of the prefab regexp libraries
> that I've found on the net so far is that the all seem to match
> regexps against strings. However, I'm dealing with streams. I
> don't want a match to fail if the required characters have not made
> it across the network yet nor do I want to have to restart a regexp
> search from the beginning after getting another character, e.g.
>
> ;; Pseudo-lisp (probably has errors) and my first LOOP
> (loop :with buffer = (make-array #| fill pointer stuff |#)
> :with matched-p = nil
> :until matched-p :do
> ;; Is there an implicit PROGN here? I'll have to look.
> (progn
> (vector-push-extend (read-char io) buffer)
> (when (regexp-match pattern buffer)
> (setq matched-p t))))
>
>
> [...] Does anyone know of a regexp library for CL that would allow
> me to do something like
>
> (regexp-match pattern stream)
Some random thoughts:
1. I haven't used expect for some time but from "man expect" I gather
that it pretty much does what your pseudo-code above does. The
manpage says: "Each time new output arrives, it is compared to each
pattern in the order they are listed."
2. If you want a regex engine to match against streams instead of
strings you should carefully describe the semantics you want. Take
your "a*" from above and change it to (in Perl syntax) /(?s)a.*/
(or /a[.\n]*/ if you prefer). With a regex like this your engine
_must_ read the stream until EOF if it (as usually) tries to find
the longest match. Is that what you want?
3. Most regex engines around use NFA engines which implies
backtracking amongst other things. If your regex is something like
/(.*)\1x/ then I think the engine has no choice but to keep track
of all input it has read from the stream so far and retry from the
beginning each time new input arrives, i.e. it'd do something like
your pseudo-code above, again.
4. If you want to read the stream one character at a time without
"looking back" you might want to look at DFA engines. You'll loose
some features like back-references or lookahead but maybe you don't
need them for an expect-like tool. I'm not sure but isn't nregex a
DFA implementation? It's in the PortableAllegroServe sources, IIRC.
HTH,
Edi.
From: Damien Kick
Subject: Re: Question about design, defmacro, macrolet, and &environment
Date:
Message-ID: <ovhe0ybkjw.fsf@email.mot.com>
Edi Weitz <···@agharta.de> writes:
> On 18 Nov 2003 12:24:40 -0600, Damien Kick <······@email.mot.com> wrote:
>
> > [...] The problem I have with all of the prefab regexp libraries
> > that I've found on the net so far is that the all seem to match
> > regexps against strings. [...]
> >
> > ;; Pseudo-lisp (probably has errors) and my first LOOP
> > (loop :with buffer = (make-array #| fill pointer stuff |#)
> > :with matched-p = nil
> > :until matched-p :do
> > ;; Is there an implicit PROGN here? I'll have to look.
> > (progn
> > (vector-push-extend (read-char io) buffer)
> > (when (regexp-match pattern buffer)
> > (setq matched-p t))))
> >
> > [...] Does anyone know of a regexp library for CL that would allow
> > me to do something like
> >
> > (regexp-match pattern stream)
>
> Some random thoughts:
>
> 1. I haven't used expect for some time but from "man expect" I gather
> that it pretty much does what your pseudo-code above does. The
> manpage says: "Each time new output arrives, it is compared to each
> pattern in the order they are listed."
I would hope that it doesn't use a Shlemiel the Painter algorithm.
> 2. If you want a regex engine to match against streams instead of
> strings you should carefully describe the semantics you
> want. Take your "a*" from above and change it to (in Perl
> syntax) /(?s)a.*/ (or /a[.\n]*/ if you prefer). With a regex
> like this your engine _must_ read the stream until EOF if it (as
> usually) tries to find the longest match. Is that what you want?
It must be what I want because I've asked for it <smile>. I don't see
why (regexp-match ".*" stream) being a synonym for (read-to-eof
stream) should be a problem. I doubt that anyone would really ever
want to use a regular expression such as this but I don't see why one
would want to disallow it.
> 3. Most regex engines around use NFA engines which implies
> backtracking amongst other things. If your regex is something
> like /(.*)\1x/ then I think the engine has no choice but to keep
> track of all input it has read from the stream so far and retry
> from the beginning each time new input arrives, i.e. it'd do
> something like your pseudo-code above, again.
If someone gave a regexp that required lots-and-lots of backtracking,
then it would have to do it, as it would with any string. However, if
the engine was attempting to match something simple, (regexp-match
"ooops" stream), then it would not have to continue to go back to the
very start of the string, i.e. the beginning of the stream, everytime
a character failed the match. I would imagine an interface similar to
the following.
(let ((string "oooooooooooooooooooooooops123"))
;; When it fails at (char string 5) through (char string 25), it
;; doesn't have to go all the way back to (char string 1) each
;; time. But we probably want to hold onto those
;; already-confirmed-as-not-matching subseqs so we know what
;; we've been reading from the stream to get the match.
(with-input-from-string (stream string)
(regexp-match "oooops" stream))
=> 20, 26, "oooooooooooooooooooooooops", #<The stream>
Or something like that. Am I missing something?
> 4. If you want to read the stream one character at a time without
> "looking back" you might want to look at DFA engines. You'll
> loose some features like back-references or lookahead but maybe
> you don't need them for an expect-like tool. I'm not sure but
> isn't nregex a DFA implementation? It's in the
> PortableAllegroServe sources, IIRC.
Unless I'm mistaken, this is what the META library already implements
for matching on streams. I'll try looking for nregex, too, though.
From: Edi Weitz
Subject: Re: Question about design, defmacro, macrolet, and &environment
Date:
Message-ID: <87k75ucvlz.fsf@bird.agharta.de>
On 20 Nov 2003 19:02:43 -0600, Damien Kick <······@email.mot.com> wrote:
> If someone gave a regexp that required lots-and-lots of
> backtracking, then it would have to do it, as it would with any
> string. However, if the engine was attempting to match something
> simple, (regexp-match "ooops" stream), then it would not have to
> continue to go back to the very start of the string, i.e. the
> beginning of the stream, everytime a character failed the match. I
> would imagine an interface similar to the following.
>
> (let ((string "oooooooooooooooooooooooops123"))
> ;; When it fails at (char string 5) through (char string 25), it
> ;; doesn't have to go all the way back to (char string 1) each
> ;; time. But we probably want to hold onto those
> ;; already-confirmed-as-not-matching subseqs so we know what
> ;; we've been reading from the stream to get the match.
> (with-input-from-string (stream string)
> (regexp-match "oooops" stream))
> => 20, 26, "oooooooooooooooooooooooops", #<The stream>
Yes, there are clever ways to do this. CL-PPCRE uses the Boyer-Moore
algorithm to search for constant strings. For a much simpler example
see the function READ-UNTIL in HTML-TEMPLATE which reads from a stream
until a certain string has been found. It only needs to peek one
character ahead and doesn't have to look back at earlier
characters. (Not that I have invented this - this is a pretty common
technique and the CS guys probably have a name for it.)
> Or something like that. Am I missing something?
I don't think so. It's just that for "simple" regular expressions
(those that could be implemented by a DFA) the stream approach might
buy you something while for more complicated regexes you might loose
the ability to do more optimizations. (E.g., if the regex engine has
random access to the whole string it can first check if it can match
the end of the string and probably give up before it starts
backtracking - CL-PPCRE does that, amongst other things.)
IMHO it won't hurt for an expect-like application to limit the regular
expressions it supports to a subset that can be efficiently
implemented on streams, i.e. something that could be done with a DFA
engine or with techniques from META.
Cheers,
Edi.
From: Coby Beck
Subject: Re: Question about design, defmacro, macrolet, and &environment
Date:
Message-ID: <bofdpg$7eu$1@otis.netspace.net.au>
"Peter Seibel" <·····@javamonkey.com> wrote in message
···················@javamonkey.com...
> Step 4 -- Whoops: hygine problem. If I write this:
>
> (dotimes (i (random 100)) (print i))
>
> it's going to call (random 100) each time it tests for the end of
> the loop. We can fix that by evaluating the count form once. But
Is that really a part of what everyone means when talking about hygiene? I
always thought that multiple evaluation and variable capture were separate
evils and only the later was a hygiene issue....?
--
Coby Beck
(remove #\Space "coby 101 @ big pond . com")
From: Peter Seibel
Subject: Re: Question about design, defmacro, macrolet, and &environment
Date:
Message-ID: <m3y8us3cm8.fsf@javamonkey.com>
"Coby Beck" <·····@mercury.bc.ca> writes:
> "Peter Seibel" <·····@javamonkey.com> wrote in message
> ···················@javamonkey.com...
> > Step 4 -- Whoops: hygine problem. If I write this:
> >
> > (dotimes (i (random 100)) (print i))
> >
> > it's going to call (random 100) each time it tests for the end of
> > the loop. We can fix that by evaluating the count form once. But
>
> Is that really a part of what everyone means when talking about hygiene? I
> always thought that multiple evaluation and variable capture were separate
> evils and only the later was a hygiene issue....?
Dunno. I guess "hygenic" macros only intend to "solve" the variable
capture problem, so by that measure you're probably right.
I'll be happy to hear other suggestions for a term that encompases
unintended variable capture, multiple evaluation and out-of-order
evaluation. Quite happy in fact, because hygine--even if it were
deemed to refer to all three--is not my favorite term.
-Peter
--
Peter Seibel ·····@javamonkey.com
Lisp is the red pill. -- John Fraser, comp.lang.lisp
From: Daniel Barlow
Subject: Re: Question about design, defmacro, macrolet, and &environment
Date:
Message-ID: <87r80k8d1m.fsf@noetbook.telent.net>
Peter Seibel <·····@javamonkey.com> writes:
> I'll be happy to hear other suggestions for a term that encompases
> unintended variable capture, multiple evaluation and out-of-order
> evaluation. Quite happy in fact, because hygine--even if it were
> deemed to refer to all three--is not my favorite term.
"Broken"? ;-)
-dan
--
http://web.metacircles.com/cirCLe_CD - Free Software Lisp/Linux distro
"Peter Seibel" <·····@javamonkey.com> wrote in message
···················@javamonkey.com...
> evaluation. Quite happy in fact, because hygine--even if it were
> deemed to refer to all three--is not my favorite term.
Me either, it feels too much like the term "immaculate conception" with all
its religious overtones!
--
Coby Beck
(remove #\Space "coby 101 @ big pond . com")
From: ·············@comcast.net
Subject: Re: Question about design, defmacro, macrolet, and &environment
Date:
Message-ID: <n0b518pd.fsf@comcast.net>
Peter Seibel <·····@javamonkey.com> writes:
> Dunno. I guess "hygenic" macros only intend to "solve" the variable
> capture problem, so by that measure you're probably right.
The only solve the unintended variable capture problem. Presumably
intended variable capture is not a problem.
> I'll be happy to hear other suggestions for a term that encompases
> unintended variable capture, multiple evaluation and out-of-order
> evaluation. Quite happy in fact, because hygine--even if it were
> deemed to refer to all three--is not my favorite term.
Um, how about `unintended variable capture'?
From: Peter Seibel
Subject: Re: Question about design, defmacro, macrolet, and &environment
Date:
Message-ID: <m3brrlyut2.fsf@javamonkey.com>
·············@comcast.net writes:
> Peter Seibel <·····@javamonkey.com> writes:
>
>
> > Dunno. I guess "hygenic" macros only intend to "solve" the variable
> > capture problem, so by that measure you're probably right.
>
> The only solve the unintended variable capture problem. Presumably
> intended variable capture is not a problem.
Yes.
> > I'll be happy to hear other suggestions for a term that encompases
> > unintended variable capture, multiple evaluation and out-of-order
> > evaluation. Quite happy in fact, because hygine--even if it were
> > deemed to refer to all three--is not my favorite term.
>
> Um, how about `unintended variable capture'?
I was looking for a term that *also* covered unintended multiple and
out-of-order evaluation.
-Peter
--
Peter Seibel ·····@javamonkey.com
Lisp is the red pill. -- John Fraser, comp.lang.lisp
From: james anderson
Subject: Re: Question about design, defmacro, macrolet, and &environment
Date:
Message-ID: <3FAEDDAC.AFBEB013@setf.de>
Peter Seibel wrote:
>
>
>
> I was looking for a term that *also* covered unintended multiple and
> out-of-order evaluation.
>
transparency
coherence
...
From: Barry Margolin
Subject: Re: Question about design, defmacro, macrolet, and &environment
Date:
Message-ID: <CwOrb.426$lK3.273@news.level3.com>
In article <··············@javamonkey.com>,
Peter Seibel <·····@javamonkey.com> wrote:
>I was looking for a term that *also* covered unintended multiple and
>out-of-order evaluation.
How about "lack of referential transparency", i.e. the fact that details of
the implementation are exposed to the caller.
--
Barry Margolin, ··············@level3.com
Level(3), Woburn, MA
*** DON'T SEND TECHNICAL QUESTIONS DIRECTLY TO ME, post them to newsgroups.
Please DON'T copy followups to me -- I'll assume it wasn't posted to the group.