I'm trying to do something like the following.
(defun read-some-data (filename)
"Reads a file containitg data in the format
number-of-lines
item-0-1 item-0-2 item-0-3
etc.
..."
(with-open-file (stream filename :direction :input)
(let* ((number-of-items (read stream))
(items (make-array number-of-items)))
(dotimes (i number-of-items)
(setf (aref items i) (read-line stream)))
items)))
My complete lack of experience with read makes me feel lost.
Firstly, I would like the elements of the array to be arrays
themselves. Should I operate on the data once I have read them in (if
so, how ?) or should I do something clever with the readtable . . . or
something completely different ?
Besides, I can't help having the feeling that there is something very
naive about the way I have written the function.
I'd appreciate constructive input.
Thanks,
Jacek.
"Jacek Generowicz" <···@ecs.soton.ac.uk> wrote in message
···················@scumbag.ecs.soton.ac.uk...
> I'm trying to do something like the following.
>
> (defun read-some-data (filename)
> "Reads a file containitg data in the format
> number-of-lines
> item-0-1 item-0-2 item-0-3
> etc.
> ..."
> (with-open-file (stream filename :direction :input)
> (let* ((number-of-items (read stream))
> (items (make-array number-of-items)))
> (dotimes (i number-of-items)
> (setf (aref items i) (read-line stream)))
> items)))
>
> My complete lack of experience with read makes me feel lost.
>
> Firstly, I would like the elements of the array to be arrays
> themselves. Should I operate on the data once I have read them in (if
> so, how ?) or should I do something clever with the readtable . . . or
> something completely different ?
>
> Besides, I can't help having the feeling that there is something very
> naive about the way I have written the function.
>
This should provide a reasonable starting point. It doesn't put things into
arrays, but that should be a simple change; I used a string input instead of
a file, again simple change. It does illustrate a few of the basic things I
have learned about reading input streams, there are probably a few quirks of
my own in there.
CL-USER 57 > (setf test-data
"item-0-0 item-0-1 item-0-2
item-1-0 item-1-1 item-1-2
item-2-0 item-2-1 item-2-2")
"item-0-0 item-0-1 item-0-2
item-1-0 item-1-1 item-1-2
item-2-0 item-2-1 item-2-2"
CL-USER 58 > (let ((input (make-string-input-stream test-data)))
(with-standard-io-syntax
(let ((*read-eval* nil))
(loop for char = (peek-char nil input nil :eof)
until (eq :eof char)
do (when (char= #\Newline char) (read-char input))
collect (loop for char = (peek-char nil input nil
:eof)
until (or (eq char :eof) (char=
#\newline char))
collect (read-preserving-whitespace
input))))))
((ITEM-0-0 ITEM-0-1 ITEM-0-2) (ITEM-1-0 ITEM-1-1 ITEM-1-2) (ITEM-2-0
ITEM-2-1 ITEM-2-2))
Please note that read behaves best only when your data file was meant for it
(or you are very lucky!)
Hope that helps!
Coby
--
(remove #\space "coby . beck @ opentechgroup . com")
On 31 Jul 2001, Jacek Generowicz wrote:
> I'm trying to do something like the following.
>
> (defun read-some-data (filename)
> "Reads a file containitg data in the format
> number-of-lines
> item-0-1 item-0-2 item-0-3
> etc.
> ..."
> (with-open-file (stream filename :direction :input)
> (let* ((number-of-items (read stream))
> (items (make-array number-of-items)))
> (dotimes (i number-of-items)
> (setf (aref items i) (read-line stream)))
> items)))
>
> My complete lack of experience with read makes me feel lost.
Don't feel bad, you're sort of on the right track...
>
> Firstly, I would like the elements of the array to be arrays
> themselves. Should I operate on the data once I have read them in (if
> so, how ?) or should I do something clever with the readtable . . . or
> something completely different ?
Unless you have total control over the input syntax and can make it be
very Lispy (if not in fact Lisp), I'd avoid the readtable. The input
format is pretty simple and it's not clear where you could hook it into
the read table. I suppose you could play games with turning #\Newline
into a reader macro, but... ughh.
The first thing I'd do is get rid of the number-of-items at the beginning
of the file; it's redundant and doesn't play nicely with the Unix pipeline
approach (if anyone here cares :). It may provide a modicum of efficiency
savings, but worrying about that is not time well spent at this stage.
So, my strategy would be:
* Figure out how to read a line of data elements as a list. We can use
repeated calls to READ for that.
* Figure out how to turn a list into a vector. COERCE comes in handy.
A common idiom for collecting a list of things in order is to cons up the
list in reverse order and then reverse it; we'll do that.
Voila:
(defun read-some-data-from-stream (stream)
(let ((file-results '()))
(do ((line (read-line stream nil stream)
(read-line stream nil stream)))
((eq line stream)
(coerce (nreverse file-results) 'vector))
(with-input-from-string (s line)
(let ((line-results '()))
(do ((element (read s nil s) (read s nil s)))
((eq element s)
(push (coerce (nreverse line-results) 'vector)
file-results))
(push element line-results)))))))
(defun read-some-data (filename &key (ignore-initial-line t))
(with-open-file (stream filename :direction :input)
(when ignore-initial-line
(read-line stream))
(read-some-data-from-stream stream)))
Now, I hate DO with a passion; I tend to use the LOOP macro instead. LOOP
also has a nice facility for constructing lists in order. So I'd really
write READ-SOME-DATA-FROM-STREAM like this:
(defun read-some-data-from-stream (stream)
(loop for line = (read-line stream nil stream)
until (eq line stream)
collect (with-input-from-string (s line)
(loop for element = (read s nil s)
until (eq element s)
collect element into line-results
finally (return (coerce line-results
'vector))))
into file-results
finally (return (coerce file-results 'vector))))
Thanks to kmp for the (read s nil s) idiom.
>
> Besides, I can't help having the feeling that there is something very
> naive about the way I have written the function.
Naive maybe, but it's a large language.
Tim
In article <············@216.39.145.192>,
Tim Moore <·····@herschel.bricoworks.com> wrote:
>The first thing I'd do is get rid of the number-of-items at the beginning
>of the file; it's redundant and doesn't play nicely with the Unix pipeline
>approach (if anyone here cares :). It may provide a modicum of efficiency
>savings, but worrying about that is not time well spent at this stage.
His file format looks suspiciously like the format used by many of the
programming contests I've seen, such as the Computing Olympiad. The
contestants are generally not allowed to change the specification of the
problem, they have to work with the file format that they're given.
--
Barry Margolin, ······@genuity.net
Genuity, Woburn, MA
*** DON'T SEND TECHNICAL QUESTIONS DIRECTLY TO ME, post them to newsgroups.
Please DON'T copy followups to me -- I'll assume it wasn't posted to the group.
Barry Margolin <······@genuity.net> writes:
> His file format looks suspiciously like the format used by many of the
> programming contests I've seen,
It also looks suspiciously like the type of format that tends to get
spat out by mesh generators. They all differ slightly, but tend to
have this sort of theme in common.
> such as the Computing Olympiad. The contestants are generally not
> allowed to change the specification of the problem, they have to
> work with the file format that they're given.
I'm half tempted to pre-process the files into something that the lisp
reader could read in immediately, after slight adulteration of the
readtable.
Jacek
Jacek Generowicz <···@ecs.soton.ac.uk> writes:
> Barry Margolin <······@genuity.net> writes:
>> His file format looks suspiciously like the format used by many of
>> the programming contests I've seen,
That _is_ distinct from homework :-), and it would be an interesting
thing to get people used to some "idioms" useful to attacking these
sorts of contests.
After all, wouldn't it be a matter for some excitement in the Lisp
community if the next ACM Scholatic Programming Contest allowed Common
Lisp as one of the acceptable languages? :-)
> It also looks suspiciously like the type of format that tends to get
> spat out by mesh generators. They all differ slightly, but tend to
> have this sort of theme in common.
>> such as the Computing Olympiad. The contestants are generally not
>> allowed to change the specification of the problem, they have to
>> work with the file format that they're given.
> I'm half tempted to pre-process the files into something that the
> lisp reader could read in immediately, after slight adulteration of
> the readtable.
Streaming the files through SOME-PREPROCESSOR-FUNCTION that could then
pass the data through an "adulterated" readtable would seem Fair
Game, and I'd be quite interested in seeing examples of such a technique
in order to 'abuse' it to other purposes.
--
(concatenate 'string "aa454" ·@freenet.carleton.ca")
http://vip.hyperusa.com/~cbbrowne/
Rules of the Evil Overlord #50. "My main computers will have their own
special operating system that will be completely incompatible with
standard IBM and Macintosh powerbooks."
<http://www.eviloverlord.com/>
········@hex.net writes:
>
> After all, wouldn't it be a matter for some excitement in the Lisp
> community if the next ACM Scholatic Programming Contest allowed Common
> Lisp as one of the acceptable languages? :-)
As the grad student member of an ACM team a few years ago I tried to
convince them to allow it. It was unfortunate that I was arguing my
case through a faculty advisor who wasn't particularly interested in
the point. Instead they told us we could C, C++, or Java. We practiced
using Java and when the big day arrived the machines had MS J++
installed instead of real Java. We ended up writing in C++ and,
needless to say, we did poorly.
If only I'd had ACL, or even Clisp. <*sigh*>
Thom
--
Thom Goodsell ···@cra.com
Scientist (617) 491-3474 x574
Charles River Analytics http://www.cra.com/
"Thom Goodsell" <···@cra.com> wrote in message
······················@shalott.cra.com...
> ········@hex.net writes:
> >
> > After all, wouldn't it be a matter for some excitement in the Lisp
> > community if the next ACM Scholatic Programming Contest allowed Common
> > Lisp as one of the acceptable languages? :-)
>
> As the grad student member of an ACM team a few years ago I tried to
> convince them to allow it. It was unfortunate that I was arguing my
> case through a faculty advisor who wasn't particularly interested in
> the point. Instead they told us we could C, C++, or Java. We practiced
> using Java and when the big day arrived the machines had MS J++
> installed instead of real Java. We ended up writing in C++ and,
> needless to say, we did poorly.
>
> If only I'd had ACL, or even Clisp. <*sigh*>
>
Allowing C++ and Java to compete with C in a competition that relies
on parsing and proper output is bad enough, using lisp is tantamount
to cheating. (Translation: They wouldn't go for it here either)
Ick, or some other disgusting sound, as if the J word wasn't bad
enough, J++?? (Ok, that's a flame, I just feel that even the idea of
a sanitary version of C++ is somehow twisted and I feel they may have
carried OOP a little too far)
--------
Geoff
To heck with the standard disclaimer: This is strictly someone else's
opinion, I will deny all knowledge of this conversation unless you're
willing to send me a decent amount of money to rat on them.
Tim Moore <·····@herschel.bricoworks.com> writes:
> The first thing I'd do is get rid of the number-of-items at the beginning
> of the file; it's redundant
I'm afraid that it isn't; I was economical with the truth (TM). The
format is really
number-of-items-in-first-block
a b c
d e f
...
number-of-items-in-second-block
A B C
D E F
...
> * Figure out how to turn a list into a vector. COERCE comes in handy.
Just the pointer I needed.
> A common idiom for collecting a list of things in order is to cons up the
> list in reverse order and then reverse it; we'll do that.
I kinda like Paul Foley's with-collector macro:
(defmacro with-collector (name &body body)
(let ((head (gensym))
(tail (gensym)))
`(let ((,head nil)
(,tail nil))
(flet ((,name (value &aux (new (list value)))
(if ,tail
(setf (cdr ,tail) new)
(setf ,head new))
(setf ,tail new)
value))
,@body
,head))))
[ very slightly edited ]
> Voila:
> (defun read-some-data-from-stream (stream)
> (let ((file-results '()))
> (do ((line (read-line stream nil stream)
> (read-line stream nil stream)))
> ((eq line stream)
> (coerce (nreverse file-results) 'vector))
> (with-input-from-string (s line)
> (let ((line-results '()))
> (do ((element (read s nil s) (read s nil s)))
> ((eq element s)
> (push (coerce (nreverse line-results) 'vector)
> file-results))
> (push element line-results)))))))
>
> (defun read-some-data (filename &key (ignore-initial-line t))
> (with-open-file (stream filename :direction :input)
> (when ignore-initial-line
> (read-line stream))
> (read-some-data-from-stream stream)))
>
> Now, I hate DO with a passion; I tend to use the LOOP macro instead. LOOP
> also has a nice facility for constructing lists in order. So I'd really
> write READ-SOME-DATA-FROM-STREAM like this:
>
> (defun read-some-data-from-stream (stream)
> (loop for line = (read-line stream nil stream)
> until (eq line stream)
> collect (with-input-from-string (s line)
> (loop for element = (read s nil s)
> until (eq element s)
> collect element into line-results
> finally (return (coerce line-results
> 'vector))))
> into file-results
> finally (return (coerce file-results 'vector))))
I guess I'm a little disappointed when I compare this to, say
def read_some_data_from_file( filename ):
file = open( filename, 'r' )
number_of_items = int( file.readline() )
items = [None] * number_of_items
for i in range( number_of_items ):
items[ i ] = map( float, file.readline().split() )
file.close()
[ it's Python, in case anyone is wondering ]
If could ignore the number of items then it would be
def read_some_data_from_file( filename ):
file = open( filename, 'r' )
map( lambda line:map(float,line.split()), file.readlines() )
file.close()
OK, I'm cheating here by knowing that the items are homogeneous,
without that it would get messier.
> Thanks to kmp for the (read s nil s) idiom.
I must have missed the point of it. What do I gain over using
(read s nil 'eof) ? I seem to lose clarity of intent.
Jacek
On 2 Aug 2001, Jacek Generowicz wrote:
> Tim Moore <·····@herschel.bricoworks.com> writes:
> > A common idiom for collecting a list of things in order is to cons up the
> > list in reverse order and then reverse it; we'll do that.
>
> I kinda like Paul Foley's with-collector macro:
> [ very slightly edited ]
>
Yeah, I like with-collector style macros too; I shied away from using it
because every software package seems to have its own variant and it's not
in the standard language. One of these days I'll adopt one of them or
roll my own and use it more widely in my own code.
> > (defun read-some-data-from-stream (stream)
> > (loop for line = (read-line stream nil stream)
> > until (eq line stream)
> > collect (with-input-from-string (s line)
> > (loop for element = (read s nil s)
> > until (eq element s)
> > collect element into line-results
> > finally (return (coerce line-results
> > 'vector))))
> > into file-results
> > finally (return (coerce file-results 'vector))))
>
> I guess I'm a little disappointed when I compare this to, say
...
> [ it's Python, in case anyone is wondering ]
>
> If could ignore the number of items then it would be
>
> def read_some_data_from_file( filename ):
> file = open( filename, 'r' )
> map( lambda line:map(float,line.split()), file.readlines() )
> file.close()
>
> OK, I'm cheating here by knowing that the items are homogeneous,
> without that it would get messier.
It's true that reading unstructured data from files is not a strong point
of ANSI Common Lisp. On the other hand, Python and its evil twin Perl are
good at that and have to be because that's the lingua franca of the Unix
world in which those languages evolved. Also, those lanaguages don't
really have list and array types but instead have an all-purpose hybrid
that shares the best and worst features of both; one would hope that the
Python idioms to deal with them would be short and sweet.
You can, of course, approach the terseness of your Python example if
that's what floats your boat. You can use adjustable arrays and
vector-push-extend to accumulate values, though in practice Lisp
programmers don't because consing up list elements is much more
convenient. You can grab the PARTITION function at
http://www-jcsu.jesus.cam.ac.uk/~csr21/partition2.lisp and get the
functionality of Python's split (I think)... The MAP function maps over
arbitrary sequences. Given user-written functions SLURP-LINES-AS-VECTOR
and READ-FLOAT, we could write:
(defun read-some-data-from-file (path)
(with-open-file (stream filename :direction :input)
(map 'vector
#'(lambda (line)
(map 'vector #'(lambda (element) (read-float element))
(partition #\Space line)))
(slurp-lines-as-vector stream))))
Wheee. Anyway, the point of my original example code was to show off
basic I/O on unstructured data and coercion to arrays, not terseness and
unLispy constructs.
> > Thanks to kmp for the (read s nil s) idiom.
>
> I must have missed the point of it. What do I gain over using
> (read s nil 'eof) ? I seem to lose clarity of intent.
The reason for using (read s nil s) is that s can't possibly be returned
by READ, whereas other common eof-value choices like nil or 'eof might be,
depending on the data. It's not an issue in your application, but it
doesn't hurt to get in the habit of using the idiom.
Tim
Tim Moore <·····@herschel.bricoworks.com> writes:
> On 2 Aug 2001, Jacek Generowicz wrote:
> > Tim Moore <·····@herschel.bricoworks.com> writes:
>
> > > (defun read-some-data-from-stream (stream)
> > > (loop for line = (read-line stream nil stream)
> > > until (eq line stream)
> > > collect (with-input-from-string (s line)
> > > (loop for element = (read s nil s)
> > > until (eq element s)
> > > collect element into line-results
> > > finally (return (coerce line-results
> > > 'vector))))
> > > into file-results
> > > finally (return (coerce file-results 'vector))))
> >
> > I guess I'm a little disappointed when I compare this to, say
> ...
> > [ it's Python, in case anyone is wondering ]
> >
> > If could ignore the number of items then it would be
> >
> > def read_some_data_from_file( filename ):
> > file = open( filename, 'r' )
> > map( lambda line:map(float,line.split()), file.readlines() )
> > file.close()
> >
> > OK, I'm cheating here by knowing that the items are homogeneous,
> > without that it would get messier.
>
> It's true that reading unstructured data from files is not a strong point
> of ANSI Common Lisp. On the other hand, Python and its evil twin Perl are
> good at that and have to be because that's the lingua franca of the Unix
> world in which those languages evolved. Also, those lanaguages don't
> really have list and array types but instead have an all-purpose hybrid
> that shares the best and worst features of both;
Yup, and not being able to separate the worst from the best can get
frustrating at times.
> You can, of course, approach the terseness of your Python example if
> that's what floats your boat.
Well, when I'm trying to rapidly prototype (TM) the implementation of
an idea, its a bit annoying that I find myself having to faff around
quite so much in what is just the prelude . . . but I guess it will
become more natural with experience.
> You can use adjustable arrays and
> vector-push-extend to accumulate values, though in practice Lisp
> programmers don't because consing up list elements is much more
> convenient. You can grab the PARTITION function at
> http://www-jcsu.jesus.cam.ac.uk/~csr21/partition2.lisp and get the
Yes, looking into this is somewhere on my to-do list. Thanks for
reminding me.
> functionality of Python's split (I think)... The MAP function maps over
> arbitrary sequences.
That reminds me, is there an equivalent of mapc which works on
vectors ?
> Given user-written functions SLURP-LINES-AS-VECTOR
> and READ-FLOAT, we could write:
>
> (defun read-some-data-from-file (path)
> (with-open-file (stream filename :direction :input)
> (map 'vector
> #'(lambda (line)
> (map 'vector #'(lambda (element) (read-float element))
> (partition #\Space line)))
> (slurp-lines-as-vector stream))))
>
> Wheee. Anyway, the point of my original example code was to show off
> basic I/O on unstructured data and coercion to arrays, not terseness and
> unLispy constructs.
Of course. It was appreciated.
> > > Thanks to kmp for the (read s nil s) idiom.
> >
> > I must have missed the point of it. What do I gain over using
> > (read s nil 'eof) ? I seem to lose clarity of intent.
>
> The reason for using (read s nil s) is that s can't possibly be returned
> by READ, whereas other common eof-value choices like nil or 'eof might be,
> depending on the data.
A good point.
Thanks,
Jacek
Hannu Koivisto <·····@iki.fi> writes:
> Jacek Generowicz <···@ecs.soton.ac.uk> writes:
>
> | That reminds me, is there an equivalent of mapc which works on
> | vectors ?
>
> (map nil ...) ?
Thanks.
Given that this works with lists too, what is the point of having a
separate mapc ? . . . and why does the same argument not count for a
separate mapv (or whatever it would be called) ?
Jacek
Jacek Generowicz <···@ecs.soton.ac.uk> writes:
> Hannu Koivisto <·····@iki.fi> writes:
>
> > Jacek Generowicz <···@ecs.soton.ac.uk> writes:
> >
> > | That reminds me, is there an equivalent of mapc which works on
> > | vectors ?
> >
> > (map nil ...) ?
>
> Thanks.
>
> Given that this works with lists too, what is the point of having a
> separate mapc ?
Because (a) mapc is part of a well-known set of traditional operators,
(b) map nil is less syntactically convenient, and (c) map nil is potentially
slower and probably won't be open-coded because it usually won't know until
runtime if it got an argument of one or the other type.
> . . . and why does the same argument not count for a
> separate mapv (or whatever it would be called) ?
Well, map is more general. It allows you to do coercions, too.
(map 'string '(#\A #\B)) => "AB"
Presumably, if there were a MAPV it would always return a value (and so
be more of the equivalent of mapcar than mapc) and that value would always
be of the input type. But also, there is the issue of making the
container size be right. Consider
(mapv #'string "AB")
This wouldn't work because although you can do (string #\A) (string #\B)
you'd need a container of type VECTOR, not of type STRING, to receive
the output. MAPV wouldn't know in advance what kind of container to make.
In the case of the list operations, they all work on things of type T
because there are no "specialized lists" like there are "specialized arrays"
in CL.
Tim Moore <·····@herschel.bricoworks.com> writes:
> On 2 Aug 2001, Jacek Generowicz wrote:
> > Tim Moore <·····@herschel.bricoworks.com> writes:
>
> > > A common idiom for collecting a list of things in order is to cons up the
> > > list in reverse order and then reverse it; we'll do that.
> >
> > I kinda like Paul Foley's with-collector macro:
> > [ very slightly edited ]
> >
> Yeah, I like with-collector style macros too; I shied away from using it
> because every software package seems to have its own variant and it's not
> in the standard language. One of these days I'll adopt one of them or
> roll my own and use it more widely in my own code.
[ ... ]
> You can grab the PARTITION function at
> http://www-jcsu.jesus.cam.ac.uk/~csr21/partition2.lisp and get the
> functionality of Python's split (I think)...
[ ... ]
OK, firstly: the canonical place for finding the partition function is
from a cCLan place, simply as then you are
(a) guaranteed to get the latest version, and
(b) get the specification thrown in for free;
a URL that currently works is
<URL:http://www-jcsu.jesus.cam.ac.uk/ftp/pub/debian/lisp/source/cl-partition_20010702.tar.gz>
(that's a debian source package, but it contains the relevant files
that you can install on your system); this however does not contain
Pierre Mai's suggested change to make the default mode of operation
information-preserving.
Secondly, in the light of the community-nature of the way partition
was written, both in its original form and this year's refinements,
perhaps you'd like to think about the Platonic ideal WITH-COLLECTOR?
Cheers,
Christophe
--
Jesus College, Cambridge, CB5 8BL +44 1223 510 299
http://www-jcsu.jesus.cam.ac.uk/~csr21/ (defun pling-dollar
(str schar arg) (first (last +))) (make-dispatch-macro-character #\! t)
(set-dispatch-macro-character #\! #\$ #'pling-dollar)
From: Tim Moore
Subject: Platonic ideal of with-collector (was Re: Read idiom)
Date:
Message-ID: <9kemr5$v5$0@216.39.145.192>
On 3 Aug 2001, Christophe Rhodes wrote:
> Tim Moore <·····@herschel.bricoworks.com> writes:
> Secondly, in the light of the community-nature of the way partition
> was written, both in its original form and this year's refinements,
> perhaps you'd like to think about the Platonic ideal WITH-COLLECTOR?
Interesting... On the one hand, it hardly seems worthwhile as simple
versions are easy to write. On the other hand, that's true of a lot of
stuff in CL.
The collection macros I've seen all seem to be a variation on the
following (this is CMUCL's version):
(collect ((foo)
(bar))
(dolist (v honig)
(if (interestingp v)
(foo v)
(bar v)))
(values (foo) (bar)))
That is, "collecting" a value into a collection looks like a function
call. This version defines multiple collections and gives access to the
value of the collection through a function "call" with no arguments;
simpler versions only define one collection and return it as the value of
the form. CMUCL's collect is more general in that a function (or macro)
can be used as a collecting function instead of the default tail
collecting behavior.
Usually the collector "functions" are defined as macros for efficiency.
I've always thought that this was too bad in that it would be useful to
pass around and return the collectors as functions.
I've never seen a setf based collector macro that works like:
(with-collections ((foo)
(bar))
(dolist (v honig)
(if (interestingp v)
(setf foo v)
(setf bar v)))
(values foo bar))
Too confusing, perhaps?
And then there are series and generators...
Tim
Tim Moore <·····@herschel.bricoworks.com> writes:
| arbitrary sequences. Given user-written functions SLURP-LINES-AS-VECTOR
| and READ-FLOAT, we could write:
|
| (defun read-some-data-from-file (path)
| (with-open-file (stream filename :direction :input)
| (map 'vector
| #'(lambda (line)
| (map 'vector #'(lambda (element) (read-float element))
| (partition #\Space line)))
| (slurp-lines-as-vector stream))))
Or with Series (not tested):
(defun read-some-data-from-file (path)
(collect 'vector (#M(lambda (line)
(map 'vector #'parse-float (partition #\Space line)))
(scan-file path #'read-line))))
which is three lines just like the Python version, if that really
matters to someone.
(I'm using name PARSE-FLOAT to be consistent with PARSE-INTEGER and
because PARSE-FLOAT implementation can be found in the CMU AI
repository.)
--
Hannu
Tim Moore <·····@herschel.bricoworks.com> wrote:
> > > Thanks to kmp for the (read s nil s) idiom.
> >
> > I must have missed the point of it. What do I gain over using
> > (read s nil 'eof) ? I seem to lose clarity of intent.
>
> The reason for using (read s nil s) is that s can't possibly be returned
> by READ, whereas other common eof-value choices like nil or 'eof might be,
> depending on the data. It's not an issue in your application, but it
> doesn't hurt to get in the habit of using the idiom.
But as the OP said, this looses "clarity of intent". Sure, it's easily
explained and learned...
A rule-of-thumb of mine (that I've just made up ;->) is 'avoid idioms
where possible'. By this I mean not that idioms are a *bad* thing, but
that I prefer the explicit over the implied. Consequently I've been
experimenting recently with the following macro:
(defmacro with-end-of-file-predicate ((input-stream
name-for-eof-indicator
name-for-predicate)
&body forms)
"Establishes a named, local EOF indicator value and predicate function
for the given input stream.
The indicator value should be used as the EOF-VALUE argument to READ,
READ-LINE, etc. calls applied to that stream.
The predicate function should be called with two arguments: the stream
and the value returned from the call to READ, READ-LINE, etc.
The call to the predicate function should be the first test performed on
that return value.
The predicate function will return true if the end-of-file has been
reached, otherwise false.
This facility is intend to provide an expressive means for reliable
detection of end-of-file situations."
(with-gensyms (in)
`(let* ((,in ,input-stream)
(,name-for-eof-indicator ,in))
(flet ((,name-for-predicate (stream read-result)
(unless (eq stream ,in)
(error "End-of-file predicate, ~S, called on incorrect
stream, ~S" ',name-for-predicate stream))
(eql read-result ,name-for-eof-indicator)))
,@forms))))
Which can be used, for example, thusly:
(defun read-cooked-line (input-stream eof-value recursive-read?)
"As READ-LINE but the returned line has trailing white-space (as
defined by RFC 1521 section 5.1) removed"
(with-end-of-file-predicate (input-stream eof-indicator end-of-file?)
(multiple-value-bind (raw-line missing-newline?)
(read-line input-stream nil eof-indicator
recursive-read?)
(if (end-of-file? input-stream raw-line)
(values eof-value nil)
(values (string-right-trim '#(#\Space #\Tab) raw-line)
missing-newline?)))))
(defun read-MIME-quoted-printable-line (&optional
(input-stream *standard-input*)
(eof-error? t)
eof-value
recursive?)
"As READ-LINE but the returned line will have been decoded as MIME
Quoted-Printable Content-Transfer-Encoded data (as defined by RFC 1521
section 5.1)"
(with-end-of-file-predicate (input-stream eof-indicator end-of-file?)
(multiple-value-bind (line missing-newline?)
(read-unfolded-line input-stream eof-indicator
recursive?)
(if (end-of-file? input-stream line)
(if eof-error?
(error "Unexpected end-of-file encountered")
(values eof-value nil))
(values (unquote-line line) missing-newline?)))))
[Apologies for the wrapping errors I introduced above.]
The WITH-END-OF-FILE-PREDICATE macro has certainly scratched my
immediate itch, but I'm not convinced it's a sane invention. The idiom
mentioned by Kent is certainly terser.
Any comments?
--
Duncan Harvey
"Smiling and waving. Before letting himself fall."
-- Anja Garbarek, The Diver
On Sun, 5 Aug 2001, Duncan Harvey wrote:
> Tim Moore <·····@herschel.bricoworks.com> wrote:
> >
> > The reason for using (read s nil s) is that s can't possibly be returned
> > by READ, whereas other common eof-value choices like nil or 'eof might be,
> > depending on the data. It's not an issue in your application, but it
> > doesn't hurt to get in the habit of using the idiom.
>
> But as the OP said, this looses "clarity of intent". Sure, it's easily
> explained and learned...
>
> A rule-of-thumb of mine (that I've just made up ;->) is 'avoid idioms
> where possible'. By this I mean not that idioms are a *bad* thing, but
> that I prefer the explicit over the implied. Consequently I've been
> experimenting recently with the following macro:
As long as you remember that you can't gain fluency in a language unless
you learn and use the idioms...
> (defmacro with-end-of-file-predicate ((input-stream
> name-for-eof-indicator
> name-for-predicate)
> &body forms)
> "Establishes a named, local EOF indicator value and predicate function
> for the given input stream.
> The indicator value should be used as the EOF-VALUE argument to READ,
> READ-LINE, etc. calls applied to that stream.
> The predicate function should be called with two arguments: the stream
> and the value returned from the call to READ, READ-LINE, etc.
> The call to the predicate function should be the first test performed on
> that return value.
> The predicate function will return true if the end-of-file has been
> reached, otherwise false.
> This facility is intend to provide an expressive means for reliable
> detection of end-of-file situations."
> (with-gensyms (in)
> `(let* ((,in ,input-stream)
> (,name-for-eof-indicator ,in))
> (flet ((,name-for-predicate (stream read-result)
> (unless (eq stream ,in)
> (error "End-of-file predicate, ~S, called on incorrect
> stream, ~S" ',name-for-predicate stream))
> (eql read-result ,name-for-eof-indicator)))
> ,@forms))))
>
> Which can be used, for example, thusly:
>
> (defun read-cooked-line (input-stream eof-value recursive-read?)
> "As READ-LINE but the returned line has trailing white-space (as
> defined by RFC 1521 section 5.1) removed"
> (with-end-of-file-predicate (input-stream eof-indicator end-of-file?)
> (multiple-value-bind (raw-line missing-newline?)
> (read-line input-stream nil eof-indicator
> recursive-read?)
> (if (end-of-file? input-stream raw-line)
> (values eof-value nil)
> (values (string-right-trim '#(#\Space #\Tab) raw-line)
> missing-newline?)))))
>
This strikes me as a bit awkward and verbose (but then I'm a fan of (read
s nil s) :) If I became convinced that (read s nil s) was too confusing,
I'd probably start defining functions that do the error checking e.g.:
(defun read-line-eof (stream &optional (recursive-read-p nil))
"Returns line, missing-newline, eofp"
(multiple-value-bind (line missing-newline)
(read-line stream nil stream recursive-read-p)
(values line missing-newline (eq line stream))))
> The WITH-END-OF-FILE-PREDICATE macro has certainly scratched my
> immediate itch, but I'm not convinced it's a sane invention. The idiom
> mentioned by Kent is certainly terser.
>
> Any comments?
If you're going to introduce new syntax to deal with eof, don't forget
that there's a perfectly good condition that's signaled by read et al. on
end-of-file. Using that your read-cooked-line would become:
(defun read-cooked-line (input-stream eof-value recursive-read?)
(handler-case
(multiple-value-bind (raw-line missing-newline?)
(read-line input-stream t nil recursive-read?)
(values (string-right-trim '#(#\Space #\Tab) raw-line)
missing-newline?)
(end-of-file () (values eof-value nil))))
Then again, if you're using conditions you might not even deal with eof in
read-cooked-line; just let it's caller deal with the condition.
Tim
Tim Moore <·····@herschel.bricoworks.com> writes:
> On Sun, 5 Aug 2001, Duncan Harvey wrote:
>
> > (defun read-cooked-line (input-stream eof-value recursive-read?)
> > "As READ-LINE but the returned line has trailing white-space (as
> > defined by RFC 1521 section 5.1) removed"
> > (with-end-of-file-predicate (input-stream eof-indicator end-of-file?)
> > (multiple-value-bind (raw-line missing-newline?)
> > (read-line input-stream nil eof-indicator
> > recursive-read?)
> > (if (end-of-file? input-stream raw-line)
> > (values eof-value nil)
> > (values (string-right-trim '#(#\Space #\Tab) raw-line)
> > missing-newline?)))))
> >
>
> This strikes me as a bit awkward and verbose (but then I'm a fan of (read
> s nil s) :) If I became convinced that (read s nil s) was too confusing,
> I'd probably start defining functions that do the error checking e.g.:
>
> (defun read-line-eof (stream &optional (recursive-read-p nil))
> "Returns line, missing-newline, eofp"
> (multiple-value-bind (line missing-newline)
> (read-line stream nil stream recursive-read-p)
> (values line missing-newline (eq line stream))))
>
> > The WITH-END-OF-FILE-PREDICATE macro has certainly scratched my
> > immediate itch, but I'm not convinced it's a sane invention. The idiom
> > mentioned by Kent is certainly terser.
I don't think terse should always win out, but this one is pretty verbose.
I guess it's a personal style choice. It's not terrible.
Then again, I don't see a problem with the idiomatic thing either.
Idiomatic uses that mean different than they look, as in
(subst nil nil x)
in Maclisp to mean
(tree-copy x)
[which it doesn't mean in CL because we changed the semantics of SUBST to
do copy-only-changed-parts, so no copy occurs], are a problem. But something
like (read stream nil stream) may look funny but says precisely what it
means and is easily recognizable by a skilled programmer. Idioms have that
virtue--they are easily recognized by their common and standard use. It
rlies on nothing odd about the semantics of READ, it's just funny because
of the apparent pun in meaning between "use me as a stream" and "use me
as a thing that won't occur on the stream", but it's not a bad pun because
there is a causal reason why it should be true that a stream will not
generally contain itself.
> > Any comments?
(Oops, I jumped the gun on comments. :-)
I'd rather (read-line stream nil nil) since READ-LINE can't return NIL
itself. Similarly, I use (read-char stream nil nil). This problem mostly
doesn't even come up except in READ itself. As such, I'd agree with Tim's
solution here below, but I'd only do it for READ itself, not the others.
> If you're going to introduce new syntax to deal with eof, don't forget
> that there's a perfectly good condition that's signaled by read et al. on
> end-of-file. Using that your read-cooked-line would become:
> (defun read-cooked-line (input-stream eof-value recursive-read?)
> (handler-case
> (multiple-value-bind (raw-line missing-newline?)
> (read-line input-stream t nil recursive-read?)
> (values (string-right-trim '#(#\Space #\Tab) raw-line)
> missing-newline?)
> (end-of-file () (values eof-value nil))))
Jacek Generowicz <···@ecs.soton.ac.uk> writes:
> Tim Moore <·····@herschel.bricoworks.com> writes:
>
> I guess I'm a little disappointed when I compare this to, say
>
> def read_some_data_from_file( filename ):
> file = open( filename, 'r' )
> number_of_items = int( file.readline() )
> items = [None] * number_of_items
> for i in range( number_of_items ):
> items[ i ] = map( float, file.readline().split() )
> file.close()
>
> [ it's Python, in case anyone is wondering ]
>
Somewhat OT, but this would be clearer and possibly better as:
# begin excerpt
def read_some_data_from_file(filename):
file = open(filename, "r")
number_of_items = int(file.readline())
items = []
for i in range(number_of_items):
items.append(map(float,
string.split(file.readline(), None, number_of_items)))
# end excerpt
Then you can use the more common idiom of appending to an array, and
make sure that there are only number_of_items splits done. (Assuming
you imported `string'; I haven't yet upgraded to Python 2.0 and thus
my string objects don't have methods.)
Slightly more on-topic, Python is a very well-designed language, IMHO;
but there is no reason why you couldn't write a Python compatibility
package for CL; it's just that CL is not typically used for this sort
of scripting. (Python, while capable of scripting, is also better than
C or even C++, IMHO, for most projects; but Lisp is better in several
ways, it's just not ideal for scripting without some kind of extra
package.)
> If could ignore the number of items then it would be
>
> def read_some_data_from_file( filename ):
> file = open( filename, 'r' )
> map( lambda line:map(float,line.split()), file.readlines() )
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
And here we see some Lisp concepts being used in Python. Python, to
me, seems to have many similarities to Lisp, especially simpler Lisps
such as Emacs Lisp.
> file.close()
>
> OK, I'm cheating here by knowing that the items are homogeneous,
> without that it would get messier.
>
> > Thanks to kmp for the (read s nil s) idiom.
>
> I must have missed the point of it. What do I gain over using
> (read s nil 'eof) ? I seem to lose clarity of intent.
>
> Jacek
--
BPT <···@tunes.org> /"\ ASCII Ribbon Campaign
backronym for Linux: \ / No HTML or RTF in mail
Linux Is Not Unix X No MS-Word in mail
Meme plague ;) ---------> / \ Respect Open Standards
BPT <···@tunes.org> writes:
> Jacek Generowicz <···@ecs.soton.ac.uk> writes:
>
> > Tim Moore <·····@herschel.bricoworks.com> writes:
> >
> > I guess I'm a little disappointed when I compare this to, say
> >
> > def read_some_data_from_file( filename ):
> > file = open( filename, 'r' )
> > number_of_items = int( file.readline() )
> > items = [None] * number_of_items
> > for i in range( number_of_items ):
> > items[ i ] = map( float, file.readline().split() )
> > file.close()
> >
> > [ it's Python, in case anyone is wondering ]
> >
> Somewhat OT, but this would be clearer and possibly better as:
>
> # begin excerpt
> def read_some_data_from_file(filename):
> file = open(filename, "r")
> number_of_items = int(file.readline())
> items = []
> for i in range(number_of_items):
> items.append(map(float,
> string.split(file.readline(), None, number_of_items)))
> # end excerpt
(defun read-some-data-from-file (filename)
(with-open-file (file filename)
(let ((n-items (read file)))
(loop for i from 0 below n-items
collect (float (read file))))))
So, here we are: making similar assumptions to those evident in the
tllotbafir (nee` Python) program, you have a CL program that does the
same stuff plus ensuring that the file gets closed in case of error.
Moreover the CL program works for a file such as
data1.dat
line 0: 10 1 2 3 4 5 6 7 8 10 9
or as a file like
data2.dat
line 0: 10
line 1:
line 2:
.
.
.
line N: 1 2 3 4 5 6 7 8 10 9
Just to clarify...
* (read-some-data-from-file "data1.dat")
(1.0 2.0 3.0 4.0 5.0 6.0 7.0 8.0 10.0 9.0)
* (read-some-data-from-file "data2.dat")
(1.0 2.0 3.0 4.0 5.0 6.0 7.0 8.0 10.0 9.0)
*
So, in the simple case, CL is as concise (the CL program is shorter)
and elegant as you can get (plus, you get the program to do the right
thing wrt errors).
I rest my case.
Cheers
PS. Actually I can be even more nagging. The tllotbafir (nee` Python)
program as it appears in the posting is incorrect wrt
cut-n-paste. It will NOT work if I pasted it into an Emacs buffer
and tried to run it (can you see why?). The CL program WILL work.
--
Marco Antoniotti ========================================================
NYU Courant Bioinformatics Group tel. +1 - 212 - 998 3488
719 Broadway 12th Floor fax +1 - 212 - 995 4122
New York, NY 10003, USA http://bioinformatics.cat.nyu.edu
"Hello New York! We'll do what we can!"
Bill Murray in `Ghostbusters'.
Marco Antoniotti wrote:
[someone else posted, approvingly:]
> > # begin excerpt
> > def read_some_data_from_file(filename):
> > file = open(filename, "r")
> > number_of_items = int(file.readline())
> > items = []
> > for i in range(number_of_items):
> > items.append(map(float,
> > string.split(file.readline(), None, number_of_items)))
> > # end excerpt
>
> (defun read-some-data-from-file (filename)
> (with-open-file (file filename)
> (let ((n-items (read file)))
> (loop for i from 0 below n-items
> collect (float (read file))))))
Unfortunately this doesn't do the same thing. The Python code
reads a number N, then reads N lines and turns each into a
vector. The Lisp code reads a number N, then reads N numbers.
(defun read-some-data-from-file (filename)
(with-open-file (file filename)
(let ((n-items (read file)))
(loop repeat n-items collect
(read-from-string
(concatenate 'string "#(" (read-line file) ")"))))))
will sort-of do, but terrible things can happen with malformed
input data. (With the Python function, the worst that can happen
is that an exception is thrown.) So maybe we'd better make it
(defun read-some-data-from-file (filename)
(with-standard-io-syntax
(let ((*read-eval* nil))
(with-open-file (file filename)
(loop repeat (read file) collect
(read-from-string
(concatenate 'string "#(" (read-line file) ")")))))))
Marco also observed that his code closes the file if
execution gets interrupted. So does the Python function;
Python GCs file objects, and because it collects garbage
(primarily) by refcounting it will always close the file
at the first opportunity.
(Actually, I'm not sure whether my code needs an extra
(READ-LINE) in it after the first read, and I'm too lazy
to check right now.)
The two functions still aren't quite equivalent. The Python
function will also barf if any of the lines contains more
than N numbers, but it will be happy if some contain fewer.
This is almost certainly not the Right Thing; making each
version check that it's been handed exactly N numbers on
each line is going to be roughly equally easy for the two
languages.
If you find yourself wanting to do much of this sort of thing
in CL, write some functions so that you can say
(defun read-some-data-from-file (filename)
(with-open-file (file filename)
(let ((n (parse-integer (read-line file))))
(loop repeat n collect
(map 'vector #'parse-float
(split-string (read-line file)
:whitespace
:exactly n))))))
The only things you'll need are PARSE-FLOAT (which already
exists) and SPLIT-STRING (various things similar to which
already exist).
One final dissimilarity remains: Python's primary sequence
type is more like an adjustable vector than like a list.
The code above produces a list of vectors. This is a
difference of style, and you shouldn't try to "fix" it
until it's known that one style is inferior for a particular
application.
--
Gareth McCaughan ················@pobox.com
.sig under construc
················@pobox.com (Gareth McCaughan) writes:
> Marco Antoniotti wrote:
>
> [someone else posted, approvingly:]
> > > # begin excerpt
> > > def read_some_data_from_file(filename):
> > > file = open(filename, "r")
> > > number_of_items = int(file.readline())
> > > items = []
> > > for i in range(number_of_items):
> > > items.append(map(float,
> > > string.split(file.readline(), None, number_of_items)))
> > > # end excerpt
> >
> > (defun read-some-data-from-file (filename)
> > (with-open-file (file filename)
> > (let ((n-items (read file)))
> > (loop for i from 0 below n-items
> > collect (float (read file))))))
>
> Unfortunately this doesn't do the same thing. The Python code
> reads a number N, then reads N lines and turns each into a
> vector. The Lisp code reads a number N, then reads N numbers.
Oops you are right. Ok. I made an ass of myself :{
Cheers
--
Marco Antoniotti ========================================================
NYU Courant Bioinformatics Group tel. +1 - 212 - 998 3488
719 Broadway 12th Floor fax +1 - 212 - 995 4122
New York, NY 10003, USA http://bioinformatics.cat.nyu.edu
"Hello New York! We'll do what we can!"
Bill Murray in `Ghostbusters'.