Hi! I wrote a simple lexer using Lisp. I tried to make it in Lisp
style, involving one of the major Lisp advantages - ability to generate
code. In my case lexer code is generated using simple lexer
specification. But my code seems akward to me. Could you kindly review
the code to advise me how can it be simplified/improved?
(defmacro add (a b)
`(setq ,a (+ ,a ,b)))
;; this function returns a fuction which for the specified string
;; generates char-by-char output.
(defun make-char-producer (str)
(let ((s str) (i 0))
(lambda ()
(cond
((>= i (length s)) nil)
(t (add i 1) (char s (1- i))))))))
;; this function returns generated lexer function
(defun make-lexer (lexer-spec char-producer)
`(let ((state nil))
(labels (,@(mapcar (lambda (s) (list (car s) nil
(make-state-function (cdr s) char-producer))) lexer-spec))
(setq state #'s-start)
(lambda () (let ((res nil))
(loop while (null res) do (setq res (funcall state)))
res)))))
;; this function generates function for each state
(defun make-state-function (conditions next-char)
`(let ((c (funcall ,next-char)))
(cond
,@(mapcar (lambda (edge) (list
(list (first edge) 'c)
(list 'setq 'state (second edge))
(third edge)))
conditions)
(t (error "unexpected char")))))
;; here is a specification of a simple lexer which parses interegrs
delimited
;; by spaces.
;; the format of specification is:
;; ( (state-a (char-condition-1 new-state-1 output-1)
;; (char-condition-2 new-state-2 output-2))
;; (state-b (char-condition-3 new-state-3 output-3)
;; (char-condition-4 new-state-4 output-4)))
(setq int-space-spec '((s-start
(null #'s-start 't-eof)
(digit-char-p #'s-digit nil)
(whitespace-char-p #'s-start nil))
(s-digit
(null #'s-start 't-int-num)
(digit-char-p #'s-digit nil)
(whitespace-char-p #'s-start 't-int-num))))
;; here is an example of usage: int-space-lexer variable now contains a
lexer function
(setq int-space-lexer (eval (make-lexer int-space-spec
(make-char-producer "123 32"))))
(funcall int-space-lexer) ;; this returns t-int
(funcall int-space-lexer) ;; this returns t-int
(funcall int-space-lexer) ;; this returns t-eof
Hi Anton,
> (defmacro add (a b)
> `(setq ,a (+ ,a ,b)))
Clever programmers will always reinvent Lisp...
http://www.lispworks.com/documentation/HyperSpec/Body/m_incf_.htm
Incidentally, http://lispdoc.com/ has nice searches for these things...
> ;; this function returns a fuction which for the specified string
> ;; generates char-by-char output.
> (defun make-char-producer (str)
> (let ((s str) (i 0))
> (lambda ()
> (cond
> ((>= i (length s)) nil)
> (t (add i 1) (char s (1- i))))))))
I've gotta run and can't do more than skim, but it seems you might want
to look at with-input-from-string.
> ;; the format of specification is:
> ;; ( (state-a (char-condition-1 new-state-1 output-1)
> ;; (char-condition-2 new-state-2 output-2))
> ;; (state-b (char-condition-3 new-state-3 output-3)
> ;; (char-condition-4 new-state-4 output-4)))
Looks a bit like Dave Roberts' state machine macro, unless I'm reading
you superficially.
http://www.findinglisp.com/blog/2004/06/basic-automaton-macro.html
http://www.findinglisp.com/blog/2004/06/automaton-meets-loop.html
> ;; here is an example of usage: int-space-lexer variable now contains a
> lexer function
> (setq int-space-lexer (eval (make-lexer int-space-spec
> (make-char-producer "123 32"))))
Your use of backquote struck my eye and I wondered if there were a
macro going on. Turns out you're using EVAL. (Which is quite often
EVIL, in terms of maintainability. Macros can be too, but EVAL has a
far higher standard of scrutiny.)
Have you looked at _Practical Common Lisp_?
http://gigamonkeys.com/book/
Tayssir
Thanks a lot, Tayssir!
I didnt know about incf, but I knew about mapcan (Dave Roberts
confessed in second post that he didnt) :)
Dave's state machine uses push model, while my lexer is pull driven.
But looking at Dave's macro I'm sure my lexer should be a macro too.
In article <························@j73g2000cwa.googlegroups.com>,
"Anton V. Belyaev" <·············@gmail.com> wrote:
> Hi! I wrote a simple lexer using Lisp. I tried to make it in Lisp
> style, involving one of the major Lisp advantages - ability to generate
> code. In my case lexer code is generated using simple lexer
> specification. But my code seems akward to me. Could you kindly review
> the code to advise me how can it be simplified/improved?
>
> (defmacro add (a b)
> `(setq ,a (+ ,a ,b)))
INCF
> ;; this function returns a fuction which for the specified string
> ;; generates char-by-char output.
> (defun make-char-producer (str)
> (let ((s str) (i 0))
> (lambda ()
> (cond
> ((>= i (length s)) nil)
> (t (add i 1) (char s (1- i))))))))
>
> ;; this function returns generated lexer function
> (defun make-lexer (lexer-spec char-producer)
> `(let ((state nil))
> (labels (,@(mapcar (lambda (s) (list (car s) nil
> (make-state-function (cdr s) char-producer))) lexer-spec))
> (setq state #'s-start)
> (lambda () (let ((res nil))
> (loop while (null res) do (setq res (funcall state)))
> res)))))
DESTRUCTURING-BIND
>
> ;; this function generates function for each state
> (defun make-state-function (conditions next-char)
> `(let ((c (funcall ,next-char)))
> (cond
> ,@(mapcar (lambda (edge) (list
> (list (first edge) 'c)
> (list 'setq 'state (second edge))
> (third edge)))
> conditions)
> (t (error "unexpected char")))))
DESTRUCTURING-BIND
Nested `( `( ...
> ;; here is a specification of a simple lexer which parses interegrs
> delimited
> ;; by spaces.
> ;; the format of specification is:
> ;; ( (state-a (char-condition-1 new-state-1 output-1)
> ;; (char-condition-2 new-state-2 output-2))
> ;; (state-b (char-condition-3 new-state-3 output-3)
> ;; (char-condition-4 new-state-4 output-4)))
> (setq int-space-spec '((s-start
> (null #'s-start 't-eof)
> (digit-char-p #'s-digit nil)
> (whitespace-char-p #'s-start nil))
> (s-digit
> (null #'s-start 't-int-num)
> (digit-char-p #'s-digit nil)
> (whitespace-char-p #'s-start 't-int-num))))
(deflexer name
(state-1 ...)
(state-2 ...))
> ;; here is an example of usage: int-space-lexer variable now contains a
> lexer function
> (setq int-space-lexer (eval (make-lexer int-space-spec
Get rid of EVAL.
see: COMPILE
> (make-char-producer "123 32"))))
>
> (funcall int-space-lexer) ;; this returns t-int
> (funcall int-space-lexer) ;; this returns t-int
> (funcall int-space-lexer) ;; this returns t-eof
--
http://lispm.dyndns.org/
Thank you, Rainer Joswig!
(deflexer name
(state-1 ...)
(state-2 ...))
yes, you're right. writing like this exposes another advantage of Lisp:
ability to extend the language.
"Anton V. Belyaev" <·············@gmail.com> writes:
> (defun make-char-producer (str)
> (let ((s str) (i 0))
> (lambda ()
> (cond
> ((>= i (length s)) nil)
> (t (add i 1) (char s (1- i))))))))
I don't see any need to introduce a new binding for str here. You
could just use str directly instead of s. I would be tempted to write
your function like this (untested), which isn't much shorter, but uses
various tricks to make it look smaller:
(defun make-char-producer (string &aux (i 0))
(lambda ()
(when (< i (length string))
(prog1 (char s i)
(incf i)))))
"Anton V. Belyaev" <·············@gmail.com> writes:
> ;; this function returns a fuction which for the specified string
> ;; generates char-by-char output.
> (defun make-char-producer (str)
> (let ((s str) (i 0))
> (lambda ()
> (cond
> ((>= i (length s)) nil)
> (t (add i 1) (char s (1- i))))))))
Why not just use WITH-INPUT-FROM-STRING or MAKE-STRING-INPUT-STREAM and
then just use READ-CHAR on the resulting stream? That would also give
you the flexibility to run the lexer on any kind of stream, not just
strings.
--
Thomas A. Russ, USC/Information Sciences Institute