From: Emre Sevinc
Subject: Trying to convert a string into a list - code review
Date:
Message-ID: <87slyuw8wg.fsf@ileriseviye.org>
To all Lisp meister who would be concerned,
I have a small text file "grammar-input.txt" containing
nothing but the line:
[PP [Spec (right)] [P' [P across] [NP the bridge]]]
I'm trying to read this file into a variable, convert
the "["s, "]"s and "'"s into "(", ")"s, and "*"s and
then build a list using the result. Here's my attempt
(without any error handling and assuming a very smart
user who created a correct formatted file):
(defvar *linear*)
(defun read-linear-file (file-name)
"Reads the file with the given name
and converts it to a list"
(with-open-file (stream file-name)
(setf *linear* (read-line stream)))
(setf *linear* (substitute #\* #\' (substitute #\) #\] (substitute #\( #\[ *linear*))))
(setf *linear* (list *linear*))
(format t "~a~%" *linear*))
It doesn't seem to do exactly what I want. I need such a list:
(PP (Spec (right)) (P* (P across) (NP the bridge)))
but it gives me extra parantheses:
CL-USER> (read-linear-file "grammar-input.txt")
((PP (Spec (right)) (P* (P across) (NP the bridge))))
NIL
And if I want to see what *linear* is, that's what
I get:
CL-USER> *linear*
("(PP (Spec (right)) (P* (P across) (NP the bridge)))")
What is the right way to do it? How can I convert a
a string that looks like a list into a real list?
(I'd also like to hear other criticisms regarding other
parts of the code in terms of clarity, idiomaticity, compactness,
etc.)
Thanks in advance.
--
Emre Sevinc
eMBA Software Developer Actively engaged in:
http:www.bilgi.edu.tr http://ileriseviye.org
http://www.bilgi.edu.tr http://fazlamesai.net
Cognitive Science Student http://cazci.com
http://www.cogsci.boun.edu.tr
From: Peter Seibel
Subject: Re: Trying to convert a string into a list - code review
Date:
Message-ID: <m28y0mchgo.fsf@beagle.local>
Emre Sevinc <·····@bilgi.edu.tr> writes:
> To all Lisp meister who would be concerned,
>
> I have a small text file "grammar-input.txt" containing
> nothing but the line:
>
>
> [PP [Spec (right)] [P' [P across] [NP the bridge]]]
>
> I'm trying to read this file into a variable, convert
> the "["s, "]"s and "'"s into "(", ")"s, and "*"s and
> then build a list using the result. Here's my attempt
> (without any error handling and assuming a very smart
> user who created a correct formatted file):
>
> (defvar *linear*)
>
> (defun read-linear-file (file-name)
> "Reads the file with the given name
> and converts it to a list"
> (with-open-file (stream file-name)
> (setf *linear* (read-line stream)))
> (setf *linear* (substitute #\* #\' (substitute #\) #\] (substitute #\( #\[ *linear*))))
> (setf *linear* (list *linear*))
> (format t "~a~%" *linear*))
>
> It doesn't seem to do exactly what I want. I need such a list:
>
> (PP (Spec (right)) (P* (P across) (NP the bridge)))
>
> but it gives me extra parantheses:
>
> CL-USER> (read-linear-file "grammar-input.txt")
> ((PP (Spec (right)) (P* (P across) (NP the bridge))))
> NIL
>
> And if I want to see what *linear* is, that's what
> I get:
>
> CL-USER> *linear*
> ("(PP (Spec (right)) (P* (P across) (NP the bridge)))")
>
> What is the right way to do it? How can I convert a
> a string that looks like a list into a real list?
Well, there are a bunch of ways to slice his. If the difference
between the syntax you have and normal Lisp syntax really is just the
difference between [] and () and the fact that you want to translate '
to * then you could use something like the approach you tried. However
after you use character translations to turn the input text into legal
s-expressions then you need to use READ-FROM-STRING to use the Lisp
reader to turn the text into a list.
However, there are two more robust ways to do it. One is to modify the
Lisp readtable. That has the advantage of making it easy to handle
other kinds of lispy syntax--double quoted strings, numbers, etc. On
the other hand, it may be clearer to simply write a custom reader for
the syntax you actually care about. Here are a few functions that
define a simple reader for what I understand your syntax to be:
(defun read-thing (stream)
"This is the entry point"
(skip-whitespace stream)
(if (member (peek-char t stream t) '(#\[ #\())
(read-list stream)
(read-symbol stream)))
(defun read-list (stream)
(loop with close = (ecase (read-char stream t) (#\[ #\]) (#\( #\)))
for next = (peek-char nil stream t)
while (char/= next close) collect (read-thing stream)
do (skip-whitespace stream)
finally (read-char stream t)))
(defun read-symbol (stream)
(intern
(with-output-to-string (s)
(loop for next = (peek-char nil stream nil nil)
while (and next (symbol-char-p next)) do
(write-char (convert-symbol-char (read-char stream t)) s)))))
(defun symbol-char-p (char)
(not (member char '(#\Space #\Newline #\[ #\] #\( #\)))))
(defun convert-symbol-char (char)
(if (char= #\' char) #\* (char-upcase char)))
(defun skip-whitespace (stream)
(loop for char = (peek-char nil stream)
while (member char '(#\Space #\Newline)) do (read-char stream t)))
With those functions you can parse your example:
CL-USER> (with-input-from-string (in "[PP [Spec (right)] [P' [P across] [NP the bridge]]]]") (read-thing in))
(PP (SPEC (RIGHT)) (P* (P ACROSS) (NP THE BRIDGE)))
This also has the advantage of it handles input broken across lines.
-Peter
--
Peter Seibel * ·····@gigamonkeys.com
Gigamonkeys Consulting * http://www.gigamonkeys.com/
Practical Common Lisp * http://www.gigamonkeys.com/book/
From: Kent M Pitman
Subject: Re: Trying to convert a string into a list - code review
Date:
Message-ID: <uslyuf6g1.fsf@nhplace.com>
Peter Seibel <·····@gigamonkeys.com> writes:
> Emre Sevinc <·····@bilgi.edu.tr> writes:
>
> > To all Lisp meister who would be concerned,
> >
> > I have a small text file "grammar-input.txt" containing
> > nothing but the line:
> >
> >
> > [PP [Spec (right)] [P' [P across] [NP the bridge]]]
> >
> > I'm trying to read this file into a variable, convert
> > the "["s, "]"s and "'"s into "(", ")"s, and "*"s and
> > then build a list using the result. Here's my attempt
> > (without any error handling and assuming a very smart
> > user who created a correct formatted file):
> >
> > (defvar *linear*)
> >
> > (defun read-linear-file (file-name)
> > "Reads the file with the given name
> > and converts it to a list"
> > (with-open-file (stream file-name)
> > (setf *linear* (read-line stream)))
> > (setf *linear* (substitute #\* #\' (substitute #\) #\] (substitute #\( #\[ *linear*))))
> > (setf *linear* (list *linear*))
> > (format t "~a~%" *linear*))
> >
> > It doesn't seem to do exactly what I want. I need such a list:
> >
> > (PP (Spec (right)) (P* (P across) (NP the bridge)))
> >
> > but it gives me extra parantheses:
> >
> > CL-USER> (read-linear-file "grammar-input.txt")
> > ((PP (Spec (right)) (P* (P across) (NP the bridge))))
> > NIL
> >
> > And if I want to see what *linear* is, that's what
> > I get:
> >
> > CL-USER> *linear*
> > ("(PP (Spec (right)) (P* (P across) (NP the bridge)))")
> >
> > What is the right way to do it? How can I convert a
> > a string that looks like a list into a real list?
>
> Well, there are a bunch of ways to slice his. If the difference
> between the syntax you have and normal Lisp syntax really is just the
> difference between [] and () and the fact that you want to translate '
> to * then you could use something like the approach you tried. However
> after you use character translations to turn the input text into legal
> s-expressions then you need to use READ-FROM-STRING to use the Lisp
> reader to turn the text into a list.
>
> However, there are two more robust ways to do it. One is to modify the
> Lisp readtable. That has the advantage of making it easy to handle
> other kinds of lispy syntax--double quoted strings, numbers, etc. On
> the other hand, it may be clearer to simply write a custom reader for
> the syntax you actually care about. Here are a few functions that
> define a simple reader for what I understand your syntax to be:
>
> (defun read-thing (stream)
> "This is the entry point"
> (skip-whitespace stream)
> (if (member (peek-char t stream t) '(#\[ #\())
> (read-list stream)
> (read-symbol stream)))
>
> (defun read-list (stream)
> (loop with close = (ecase (read-char stream t) (#\[ #\]) (#\( #\)))
> for next = (peek-char nil stream t)
> while (char/= next close) collect (read-thing stream)
> do (skip-whitespace stream)
> finally (read-char stream t)))
>
> (defun read-symbol (stream)
> (intern
> (with-output-to-string (s)
> (loop for next = (peek-char nil stream nil nil)
> while (and next (symbol-char-p next)) do
> (write-char (convert-symbol-char (read-char stream t)) s)))))
>
> (defun symbol-char-p (char)
> (not (member char '(#\Space #\Newline #\[ #\] #\( #\)))))
>
> (defun convert-symbol-char (char)
> (if (char= #\' char) #\* (char-upcase char)))
>
> (defun skip-whitespace (stream)
> (loop for char = (peek-char nil stream)
> while (member char '(#\Space #\Newline)) do (read-char stream t)))
>
> With those functions you can parse your example:
>
> CL-USER> (with-input-from-string (in "[PP [Spec (right)] [P' [P across] [NP the bridge]]]]") (read-thing in))
> (PP (SPEC (RIGHT)) (P* (P ACROSS) (NP THE BRIDGE)))
>
>
> This also has the advantage of it handles input broken across lines.
Or you can just use the Lisp built-ins:
(defvar *parser-readtable* (copy-readtable))
(defun read-bracketed-list (stream char)
(read-delimited-list #\] stream))
(set-syntax-from-char #\' #\A *parser-readtable*)
(set-syntax-from-char #\] #\) *parser-readtable*)
(set-macro-character #\[ 'read-bracketed-list nil *parser-readtable*)
(defun parse-text (text)
(let ((*readtable* *parser-readtable*))
(with-input-from-string (s (substitute #\* #\' text))
(read s))))
(parse-text "[PP [Spec (right)] [P' [P across] [NP the bridge]]]")
=> (PP (SPEC (RIGHT)) (P* (P ACROSS) (NP THE BRIDGE)))
Or, if you add this in the syntax table setup code above:
(setf (readtable-case *parser-readtable*) :preserve)
you can get
(parse-text "[PP [Spec (right)] [P' [P across] [NP the bridge]]]")
=> (PP (|Spec| (|right|)) (P* (P |across|) (NP |the| |bridge|)))
the only disadvantage here is that you're allowing other Lisp syntax
to creep in. (Peter's answer gives you more tight control over the
allowed syntax, at the cost that you have to write all that detailed
parsing code.)
From: drewc
Subject: Re: Trying to convert a string into a list - code review
Date:
Message-ID: <JHkye.157931$El.33226@pd7tw1no>
> Emre Sevinc <·····@bilgi.edu.tr> writes:
[snip]
>>
>>What is the right way to do it? How can I convert a
>>a string that looks like a list into a real list?
To which Peter Seibel replied:
> Well, there are a bunch of ways to slice this.
This reminds me of Perl Golf (esp. the 'slice' part), and i was inspired
to write a perl-like monstrosity in CL to solve this.
First, using loop, which some say is as bad as perl :
CL-USER> (let ((_ "[PP [Spec (right)] [P' [P across] [NP the bridge]]]"))
(loop for $_ on '(#\( #\[ #\) #\] #\* #\') by #'cddr
do (nsubstitute (first $_) (second $_) _)
finally (return (read-from-string _))))
=> (PP (SPEC (RIGHT)) (P* (P ACROSS) (NP THE BRIDGE)))
Althoug personally i find the lispy recursive version just as horrific
as any perl i've seen :
CL-USER> (let ((~ "[PP [Spec (right)] [P' [P across] [NP the bridge]]]"))
(labels ((_ (@ $) (nsubstitute (car $)(cadr $) @)
(if $ (_ @ (cddr $))(read-from-string @))))
(_ ~ '(#\( #\[ #\) #\] #\* #\'))))
=> (PP (SPEC (RIGHT)) (P* (P ACROSS) (NP THE BRIDGE)))
> However, there are two more robust ways to do it.
We can be thankful for that! :)
--
Drew Crampsie
drewc at tech dot coop
"Never mind the bollocks -- here's the sexp's tools."
-- Karl A. Krueger on comp.lang.lisp
Emre Sevinc <·····@bilgi.edu.tr> writes:
> (defun read-linear-file (file-name)
> "Reads the file with the given name
> and converts it to a list"
> (with-open-file (stream file-name)
> (setf *linear* (read-line stream)))
> (setf *linear* (substitute #\* #\' (substitute #\) #\] (substitute #\( #\[ *linear*))))
> (setf *linear* (list *linear*))
> (format t "~a~%" *linear*))
>
> It doesn't seem to do exactly what I want. I need such a list:
>
> (PP (Spec (right)) (P* (P across) (NP the bridge)))
>
> but it gives me extra parantheses:
>
> CL-USER> (read-linear-file "grammar-input.txt")
> ((PP (Spec (right)) (P* (P across) (NP the bridge))))
> NIL
>
> And if I want to see what *linear* is, that's what
> I get:
>
> CL-USER> *linear*
> ("(PP (Spec (right)) (P* (P across) (NP the bridge)))")
>
> What is the right way to do it? How can I convert a
> a string that looks like a list into a real list?
1- Don't modify a global variable from a function such as read-linear-file.
2- If you don't want a list, why do you put the linear into a list with
(setf *linear* (list *linear*)) ?
3- Don't print a resut from a function when the specification doesn't say so.
4- LIST doesn't convert anything, it just collects its arguments into a list
and return it. You want to use READ-FROM-STRING.
(defun read-linear-file (file-name)
"Reads the first line in the file at path FILE-NAME and
parses it to a list of tokens and sublists."
(let ((line (with-open-file (stream file-name) (read-line stream))))
(dolist (substitution '(( #\* #\' )
( #\) #\] )
( #\( #\[ )))
(setf line (substitute (first substitution) (second substitution) line)))
(let ((*read-eval* nil)) (read-from-string line))))
(defparameter *linear* (read-linear-file "input"))
(format t "~S~%" *linear*)
> (I'd also like to hear other criticisms regarding other
> parts of the code in terms of clarity, idiomaticity, compactness,
> etc.)
But as Peter explained, it's much better to use reader macros to be
able to read free-form expressions instead of just one line. Perhaps
you'll want to change the *readtable* case too, since you write "PP"
and "Spec". With reader macros you'd not need to substitute * for ':
you'd get symbols such as |P'|, Which still print as P':
[96]> (format t "~A" '|P'|)
P'
NIL
[97]> (format t "~S" '|P'|)
|P'|
NIL
--
__Pascal Bourguignon__ http://www.informatimago.com/
Nobody can fix the economy. Nobody can be trusted with their finger
on the button. Nobody's perfect. VOTE FOR NOBODY.
From: Wade Humeniuk
Subject: Re: Trying to convert a string into a list - code review
Date:
Message-ID: <sFmye.80992$wr.69441@clgrps12>
Emre Sevinc wrote:
> To all Lisp meister who would be concerned,
>
> I have a small text file "grammar-input.txt" containing
> nothing but the line:
>
>
> [PP [Spec (right)] [P' [P across] [NP the bridge]]]
>
Here is a method using the PARSERGEN package in LW. The same
result can be obtained using CL-YACC. This is just a monkey-wrench,
take it with a grain of salt.
Wade
(in-package :cl-user)
(eval-when (:compile-toplevel :load-toplevel :execute)
(require "parsergen")
(use-package :parsergen))
(defparser [grammar-parser]
(([grammar] grammar))
((grammar \[ elements \]) $2)
((elements element) (list $1))
((elements element elements) (cons $1 $2))
((element symbol) $1)
((element \( elements \)) $2)
((element grammar) $1))
(defun [grammar-lexer] (stream)
(let ((c (read-char stream nil nil)))
(cond
((null c) (values nil nil))
((char= c #\[) (values '\[ c))
((char= c #\]) (values '\] c))
((char= c #\() (values '\( c))
((char= c #\)) (values '\) c)
((member c '(#\space #\tab #\newline) :test #'char=)
([grammar-lexer] stream))
((alpha-char-p c)
(let ((string (make-array 32 :element-type 'character
:fill-pointer 0
:adjustable t)))
(vector-push c string)
(loop for c = (read-char stream nil nil) do
(cond
((null c) (error "No ] termination"))
((member c '(#\space #\tab #\newline #\[ #\] #\( #\) ) :test #'char=)
(unread-char c stream)
(return-from [grammar-lexer] (values 'symbol (intern string))))
((char= c #\') (vector-push-extend #\* string))
((alphanumericp c)
(vector-push-extend c string))))))
(t (error "Invalid Character ~s" c)))))
(defun [parse-grammar-string] (string)
(with-input-from-string (stream string)
([grammar-parser] (lambda () ([grammar-lexer] stream)))))
CL-USER 7 > ([parse-grammar-string] "[PP [Spec (right)] [P' [P across] [NP the bridge]]]")
(PP (|Spec| (|right|)) (P* (P |across|) (NP |the| |bridge|)))
NIL
CL-USER 8 >
From: Emre Sevinc
Subject: Re: Trying to convert a string into a list - code review
Date:
Message-ID: <87k6k4x69b.fsf@ileriseviye.org>
Emre Sevinc <·····@bilgi.edu.tr> writes:
> To all Lisp meister who would be concerned,
>
> I have a small text file "grammar-input.txt" containing
> nothing but the line:
>
>
> [PP [Spec (right)] [P' [P across] [NP the bridge]]]
>
> I'm trying to read this file into a variable, convert
> the "["s, "]"s and "'"s into "(", ")"s, and "*"s and
> then build a list using the result. Here's my attempt
> (without any error handling and assuming a very smart
> user who created a correct formatted file):
>
> (defvar *linear*)
Thanks a lot to everybody who suggested solutions
for my problem.
Now I'm one step closer to that simple linguistic
utility of mine! :)
Happy hacking,
--
Emre Sevinc
eMBA Software Developer Actively engaged in:
http:www.bilgi.edu.tr http://ileriseviye.org
http://www.bilgi.edu.tr http://fazlamesai.net
Cognitive Science Student http://cazci.com
http://www.cogsci.boun.edu.tr