I've found a nice YACC implementation for Lisp:
http://www.pps.jussieu.fr/~jch/software/cl-yacc/
now I need a lex implementation. I want to define a lexer similar to the
orignal lex format with regular expressions, e.g. something for lexing a
subset of C:
(define-lexer my-lexer
(definitions
:substitutions (whitespaces "[ \t\n]*"))
(rules
(({whitespaces} "[0-9]+" {whitespaces})
(values integer value))
(({whitespaces} ";" {whitespaces})
(values punctuator value))
(({whitespaces} "=" {whitespaces})
(values punctuator value))
(({whitespaces} "\." {whitespaces})
(values punctuator value))
(({whitespaces} "\++" {whitespaces})
(values punctuator value))
(({whitespaces} "\+" {whitespaces})
(values punctuator value))
(({whitespaces} "[a-z,A-Z,_]+[a-z,A-Z,_,0-9]*" {whitespaces})
(values identifier value))))
which I want to use like this:
(with-input-from-string (s "foo=10; bar=foo+++++foo;")
(loop do
(multiple-value-bind (s v) (my-lexer s))
(unless s (loop-finish))
(format t "~a ~a~%")))
identifier foo
punctuator =
integer 10
punctuator ;
identifier bar
punctuator =
identifier foo
punctuator ++
punctuator ++
punctuator +
identifier foo
punctuator ;
--
Frank Buss, ··@frank-buss.de
http://www.frank-buss.de, http://www.it4-systems.de
You can use the CLAWK package. It has a lexer package that "produces
lexers that are compatible with the parsers produced with Xanalys's
DEFPARSER macro."
Here's a link:
http://www.geocities.com/mparker762/clawk
--
Fred Gilham ······@csl.sri.com
When an economist criticizes any human institution because it has
failed to convey to mankind an incommunicable attribute of God, we can
safely dismiss that economist. The trouble is, there remains a market
for these economists in academia. I regard this fact as one more piece
of evidence for the failure of tax-subsidized education. -- Gary North
Fred Gilham wrote:
> You can use the CLAWK package. It has a lexer package that "produces
> lexers that are compatible with the parsers produced with Xanalys's
> DEFPARSER macro."
>
> Here's a link:
> http://www.geocities.com/mparker762/clawk
Thanks, it works. I've changed the packages to the ASDF system and included
it in the Common Lisp Application Builder project
(http://www.lispbuilder.org), because this could be the base for many other
nice things, like the DXF parser for the 3D viewer or a C header file
parser for better CFFI generation.
The ASDF installable packages can be downloaded at
http://sourceforge.net/project/showfiles.php?group_id=159740
A short example how to use it:
(defpackage #:calculator
(:use :lispbuilder-regex :lispbuilder-lexer :lispbuilder-yacc :cl))
(in-package #:calculator)
(deflexer calculator-lexer
("([:digit:]+\\.)?[:digit:]+"
(return (values 'number (num %0))))
("\\+"
(return (values '+ nil)))
("-"
(return (values '- nil)))
("\\*"
(return (values '* nil)))
("/"
(return (values '/ nil)))
("\\("
(return (values '|(| nil)))
("\\)"
(return (values '|)| nil)))
("[:space:]"))
(define-parser calculator-parser
(:start-symbol expression)
(:terminals (number + - * / |(| |)|))
(:precedence ((:left * /) (:left + -)))
(expression
(expression + expression #'(lambda (a op b) (+ a b)))
(expression - expression #'(lambda (a op b) (- a b)))
(expression * expression #'(lambda (a op b) (* a b)))
(expression / expression #'(lambda (a op b) (/ a b)))
term)
(term
number
(- term #'(lambda (op a) (- a)))
(|(| expression |)| #'(lambda (a b c) b))))
(defun calculate (expression)
(parse-with-lexer (calculator-lexer expression) calculator-parser))
(calculate "12 + 47*3 - 3 * (-2.75+40)") -> 41.25
--
Frank Buss, ··@frank-buss.de
http://www.frank-buss.de, http://www.it4-systems.de
Frank,
I modified clex.lisp to add support for case-insensitive lexing. Code
at http://wagerlabs.com/clex.lisp.
I looked at the lexer in cl-awk but picked clex for some valid reason
that I don't remember now.
Thanks, Joel
Joel Reymont wrote:
> I modified clex.lisp to add support for case-insensitive lexing. Code
> at http://wagerlabs.com/clex.lisp.
I didn't found documentation how to use your code. Does it support group
matching, like this:
(defun get-group (regs index string)
(let ((group-start (register-start regs index))
(group-end (register-end regs index)))
(subseq string group-start group-end)))
(defun test ()
(let ((matcher (compile-str "[:alpha:]+[:space:]*([:alpha:]+).*"))
(test-string "the Second word"))
(multiple-value-bind (matched-p start len regs)
(scan-str matcher test-string)
(format t "group0: ~a, group1: ~a~%"
(get-group regs 0 test-string)
(get-group regs 1 test-string)))))
(test) -> group0: the Second word, group1: Second
The lexer has a simpler syntax, like with flex (%0, %1 etc.) for selecting
matched groups.
> I looked at the lexer in cl-awk but picked clex for some valid reason
> that I don't remember now.
At least CLAWK looks stable, with many testcases and documentation and it
is released under BSD licence, which makes it easer to integrate it in
other projects than with LGPL.
--
Frank Buss, ··@frank-buss.de
http://www.frank-buss.de, http://www.it4-systems.de
On Sun, 19 Feb 2006 08:28:11 +0100, Frank Buss <··@frank-buss.de> wrote:
> I want to define a lexer similar to the orignal lex format with
> regular expressions
<http://groups.google.com/group/comp.lang.lisp/msg/deaf25edb6f2ef92>
<http://common-lisp.net/pipermail/cl-ppcre-devel/2004-June/000041.html>
Cheers,
Edi.
--
European Common Lisp Meeting 2006: <http://weitz.de/eclm2006/>
Real email: (replace (subseq ·········@agharta.de" 5) "edi")
In case someone else comes upon this thread at some later date, it is
worth noting that there was a long thread titled "We need a Lex/Yacc
for Common Lisp (Re: Processing (SG|X)ML)" that contains complementary
information to the options that were presented in this thread:
http://groups.google.com/group/comp.lang.lisp/browse_frm/thread/42469370877bcbde/8ad01039569139e8?hl=en#8ad01039569139e8
For example, Peter Seibel had some code for a Java lexer:
http://www.gigamonkeys.com/lisp/java-lexer.lisp
--
Bill Clementson
········@gmail.com writes:
> In case someone else comes upon this thread at some later date, it is
> worth noting that there was a long thread titled "We need a Lex/Yacc
> for Common Lisp (Re: Processing (SG|X)ML)" that contains complementary
> information to the options that were presented in this thread:
> http://groups.google.com/group/comp.lang.lisp/browse_frm/thread/42469370877bcbde/8ad01039569139e8?hl=en#8ad01039569139e8
>
> For example, Peter Seibel had some code for a Java lexer:
> http://www.gigamonkeys.com/lisp/java-lexer.lisp
Note, for that to work, you'll need the accompanying parser generator:
<http://www.gigamonkeys.com/lisp/parser.lisp>
Some day I hope to return to that project (one of the last I worked on
before I started writing my book) and turn it into an actual thing.
-Peter
--
Peter Seibel * ·····@gigamonkeys.com
Gigamonkeys Consulting * http://www.gigamonkeys.com/
Practical Common Lisp * http://www.gigamonkeys.com/book/