From: Kent M Pitman
Subject: Re: Iterating through a file.
Date: 
Message-ID: <sfwae4vf0bd.fsf@world.std.com>
"Christopher J. Vogt" <····@computer.org> writes:

> (with-open-file (fh (pathname "test.lisp") :direction :input)
>   (loop for line = (read-line fh nil nil)
> 	while line do
> 	(write-string line)))

And, of course, there's no reason  to do (pathname "test.lisp") since
with-open-file (and open, which underlies it) will coerce a string to 
a pathname anyway.

Also, don't use a variable like "fh" which probably stands for
"file-handle" since it's an abbreviation, and an obscure one, and an
incorrect one.  Lisp has no "handles".  It has "streams".  Handles are
some sort of low-level operating system thing with probably a
different set of operations on them than streams.  Call it a stream,
and spell it out.

(with-open-file (stream "test.lisp" :direction :input)
  (loop for line = (read-line stream nil nil)
	while line 
        do (write-string line)))

- - - - - Other notes for greg (the original poster) follow - - - - - -

Note also that this example uses a designated input stream but the default
output stream.  It's worth noting also that you often want to specify both
streams explicitly, as in:

;;; Example uses explicit streams in both directions.
(defun copy-stream (instream outstream)
  (loop for line = (read-line instream nil nil)
	while line
	do (write-line line outstream)))

(with-open-file (stream "test.lisp" :direction :input)
  (copy-stream stream *standard-output*))

You could, though it would be more obscure, go the other way and just
redirect *standard-input*, as in:

;;; Example uses default streams in both directions.
(with-open-file (*standard-input* "test.lisp" :direction :input)
  (loop for line = (read-line nil nil nil) ;reads from *standard-input*
	while line
	do (write-line line))) ;writes to *standard-output*

From: Kent M Pitman
Subject: Re: Iterating through a file.
Date: 
Message-ID: <sfwd79q6hdl.fsf@world.std.com>
Kent M Pitman <······@world.std.com> writes:

> (with-open-file (stream "test.lisp" :direction :input)
>   (loop for line = (read-line stream nil nil)
> 	while line 
>         do (write-string line)))

Dunno why I didn't catch this before, but it should be write-line,
not write-string.

read-line reads to a newline and discards it.
If you want a proper copy,  you need to put back the discarded newline.
write-line will do this. otherwise, use write-string followed by terpri,
but no need since write-line does that paired operation for you.
From: Greg Menke
Subject: Re: Iterating through a file.
Date: 
Message-ID: <m3u232hcrt.fsf@europa.mindspring.com>
> > (with-open-file (stream "test.lisp" :direction :input)
> >   (loop for line = (read-line stream nil nil)
> > 	while line 
> >         do (write-string line)))
> 
> Dunno why I didn't catch this before, but it should be write-line,
> not write-string.
> 
> read-line reads to a newline and discards it.
> If you want a proper copy,  you need to put back the discarded newline.
> write-line will do this. otherwise, use write-string followed by terpri,
> but no need since write-line does that paired operation for you.

I cut the above code out of the program & removed extra stuff from the
line processing- leaving the write-string, which is actually suited at
the moment (I'm fiddling around generating a dynamic html page under
cl-http).  wrt the 'fh', thanks, its an old C habit of mine that keeps
slipping in...

Thanks,

Gregm
From: Marco Antoniotti
Subject: Re: Iterating through a file.
Date: 
Message-ID: <y6ck83ymru6.fsf@octagon.mrl.nyu.edu>
Greg Menke <··········@mindspring.com> writes:

> > > (with-open-file (stream "test.lisp" :direction :input)
> > >   (loop for line = (read-line stream nil nil)
> > > 	while line 
> > >         do (write-string line)))
> > 
> > Dunno why I didn't catch this before, but it should be write-line,
> > not write-string.
> > 
> > read-line reads to a newline and discards it.
> > If you want a proper copy,  you need to put back the discarded newline.
> > write-line will do this. otherwise, use write-string followed by terpri,
> > but no need since write-line does that paired operation for you.
> 
> I cut the above code out of the program & removed extra stuff from the
> line processing- leaving the write-string, which is actually suited at
> the moment (I'm fiddling around generating a dynamic html page under
> cl-http).  wrt the 'fh', thanks, its an old C habit of mine that keeps
> slipping in...

Now that you have gotten it right, you might as well getting it fast :)

Using READ-LINE will cons up a string each time it is called.  Most of
the time this is what you want, however, you should consider
READ-SEQUENCE with a buffer string as argument to speed up things.

Cheers


-- 
Marco Antoniotti ========================================================
NYU Courant Bioinformatics Group	tel. +1 - 212 - 998 3488
719 Broadway 12th Floor                 fax  +1 - 212 - 995 4122
New York, NY 10003, USA			http://bioinformatics.cat.nyu.edu
	       "Hello New York! We'll do what we can!"
			Bill Murray in `Ghostbusters'.
From: Barry Margolin
Subject: Re: Iterating through a file.
Date: 
Message-ID: <m4eI6.7$Ip3.294@burlma1-snr2>
In article <···············@octagon.mrl.nyu.edu>,
Marco Antoniotti  <·······@cs.nyu.edu> wrote:
>Using READ-LINE will cons up a string each time it is called.  Most of
>the time this is what you want, however, you should consider
>READ-SEQUENCE with a buffer string as argument to speed up things.

Ephemeral garbage collectors make it relatively cheap to cons such
short-lived objects.  Don't waste your brain cells worry about this; the
cost of the file I/O almost certainly outweighs this by orders of
magnitude.

-- 
Barry Margolin, ······@genuity.net
Genuity, Burlington, MA
*** DON'T SEND TECHNICAL QUESTIONS DIRECTLY TO ME, post them to newsgroups.
Please DON'T copy followups to me -- I'll assume it wasn't posted to the group.
From: Marco Antoniotti
Subject: Re: Iterating through a file.
Date: 
Message-ID: <y6c66fimoru.fsf@octagon.mrl.nyu.edu>
Barry Margolin <······@genuity.net> writes:

> In article <···············@octagon.mrl.nyu.edu>,
> Marco Antoniotti  <·······@cs.nyu.edu> wrote:
> >Using READ-LINE will cons up a string each time it is called.  Most of
> >the time this is what you want, however, you should consider
> >READ-SEQUENCE with a buffer string as argument to speed up things.
> 
> Ephemeral garbage collectors make it relatively cheap to cons such
> short-lived objects.  Don't waste your brain cells worry about this; the
> cost of the file I/O almost certainly outweighs this by orders of
> magnitude.

I guess you are right.  Maybe I am just fighting with very large files
I should be reading in in a different way.  Oh well.

Cheers

-- 
Marco Antoniotti ========================================================
NYU Courant Bioinformatics Group	tel. +1 - 212 - 998 3488
719 Broadway 12th Floor                 fax  +1 - 212 - 995 4122
New York, NY 10003, USA			http://bioinformatics.cat.nyu.edu
	       "Hello New York! We'll do what we can!"
			Bill Murray in `Ghostbusters'.
From: Rahul Jain
Subject: Re: Iterating through a file.
Date: 
Message-ID: <9csrlm$fuq$1@joe.rice.edu>
In article <···············@octagon.mrl.nyu.edu> on Thu, 03 May 2001
10:32:37 -0500, "Marco Antoniotti" <·······@cs.nyu.edu> wrote:

> I guess you are right.  Maybe I am just fighting with very large files I
> should be reading in in a different way.  Oh well.
> 

If you know that you're dealing with only Unix (I guess you'd have to
restrict yourself to one implementation, too), you could mmap the file
and then treat it as a huge array. At least I know that's how people do
it in C. Not sure how well/easily it would work in CL.

-- 
-> -/-                       - Rahul Jain -                       -\- <-
-> -\- http://linux.rice.edu/~rahul -=- ·················@usa.net -/- <-
-> -/- "I never could get the hang of Thursdays." - HHGTTG by DNA -\- <-
|--|--------|--------------|----|-------------|------|---------|-----|-|
   Version 11.423.999.220020101.23.50110101.042
   (c)1996-2000, All rights reserved. Disclaimer available upon request.
From: Marco Antoniotti
Subject: Re: Iterating through a file.
Date: 
Message-ID: <y6c7kzxyx7z.fsf@octagon.mrl.nyu.edu>
"Rahul Jain" <·····@rice.edu> writes:

> In article <···············@octagon.mrl.nyu.edu> on Thu, 03 May 2001
> 10:32:37 -0500, "Marco Antoniotti" <·······@cs.nyu.edu> wrote:
> 
> > I guess you are right.  Maybe I am just fighting with very large files I
> > should be reading in in a different way.  Oh well.
> > 
> 
> If you know that you're dealing with only Unix (I guess you'd have to
> restrict yourself to one implementation, too), you could mmap the file
> and then treat it as a huge array. At least I know that's how people do
> it in C. Not sure how well/easily it would work in CL.

It works decently through a bit of FFI magic.  However, I am not
dealing only with UNIX.

Since I have to deal with arrays anyway, I have thought about
something like

	(let* ((buffer (make-array (min (file-length stream)
                                       array-dimension-limit)
                                  :element-type 'character
                                  :initial-element #\Null))
               (bytes-read (read-sequence buffer (file-length stream))))
           (delete-eol buffer))

Haven't done it yet though.            

Cheers

-- 
Marco Antoniotti ========================================================
NYU Courant Bioinformatics Group	tel. +1 - 212 - 998 3488
719 Broadway 12th Floor                 fax  +1 - 212 - 995 4122
New York, NY 10003, USA			http://bioinformatics.cat.nyu.edu
	       "Hello New York! We'll do what we can!"
			Bill Murray in `Ghostbusters'.