From: ········@gmail.com
Subject: force eof on an output stream
Date: 
Message-ID: <1151635259.568943.303670@y41g2000cwy.googlegroups.com>
someone asked if it's possible to remove the first 300 lines of a file
without ever having two copies of the file or loading the entire thing
into memory.

(with-open-file (in #p"/path/to/bigfile" :direction :input)
    (with-open-file (out #p"/path/to/bigfile"
                          :direction :output
                          :if-exists :overwrite)
         (dotimes (i 300) (read-line in))
         (handler-case
             (loop (princ (read-line in) out) (terpri out))
          (error ()))

would leave the last 300 lines duplicated at the end of the file... Is
it possible to "force" an eof to an output stream? I'm just a pig
farmer and don't know these kind of things...

Nick

From: Pascal Bourguignon
Subject: Re: force eof on an output stream
Date: 
Message-ID: <877j2zz5wq.fsf@thalassa.informatimago.com>
········@gmail.com writes:

> someone asked if it's possible to remove the first 300 lines of a file
> without ever having two copies of the file or loading the entire thing
> into memory.
>
> (with-open-file (in #p"/path/to/bigfile" :direction :input)
>     (with-open-file (out #p"/path/to/bigfile"
>                           :direction :output
>                           :if-exists :overwrite)
>          (dotimes (i 300) (read-line in))
>          (handler-case
>              (loop (princ (read-line in) out) (terpri out))
>           (error ()))
>
> would leave the last 300 lines duplicated at the end of the file... Is
> it possible to "force" an eof to an output stream? I'm just a pig
> farmer and don't know these kind of things...

The best you can do, portably, is:

(defun file-truncate-lines (file nlines)
  (let ((copy (merge-pathnames (make-pathname :type "TRU") file nil)))
    (with-open-file (in file)
      (with-open-file (out copy :direction :output
                           :if-does-not-exist :create
                           :if-exists :error)
        (loop :repeat nlines
           :for line = (read-line in nil nil)
           :while line
           :do (format out "~A~%" line))))
    (delete-file file)
    (rename-file copy file)))

Otherwise, you could try to use some implementation and OS specific
feature, like FFI to truncate(2) (which isn't even defined for normal
POSIX, you need the XSI extension).

-- 
__Pascal Bourguignon__                     http://www.informatimago.com/
Kitty like plastic.
Confuses for litter box.
Don't leave tarp around.
From: Rob Warnock
Subject: Re: force eof on an output stream
Date: 
Message-ID: <t9ednUTGks93AjnZnZ2dnUVZ_sidnZ2d@speakeasy.net>
Pascal Bourguignon  <···@informatimago.com> wrote:
+---------------
| ········@gmail.com writes:
| > someone asked if it's possible to remove the first 300 lines of a file
| > without ever having two copies of the file or loading the entire thing
| > into memory.
...
| The best you can do, portably, is:
| (defun file-truncate-lines (file nlines)
|   (let ((copy (merge-pathnames (make-pathname :type "TRU") file nil)))
|     (with-open-file (in file)
|       (with-open-file (out copy :direction :output
|                            :if-does-not-exist :create
|                            :if-exists :error)
|         (loop :repeat nlines
|            :for line = (read-line in nil nil)
|            :while line
|            :do (format out "~A~%" line))))
|     (delete-file file)
|     (rename-file copy file)))
+---------------

I think it was asking for "tail +300", not "head -300". That is,
the loop needs to look something like this:

          (loop for i from 0
                and line = (read-line in nil nil)
                while line
	    when (>= i nlines)
              do (format out "~a~%" line))

But READ-LINE conses, PEEK-CHAR & READ-CHAR not so much (at least,
not in implementations in which CHAR is an "immediate" type), so
when doing this sort of thing I tend to replace the ignored-value
READ-LINES as follows:

          (loop repeat nlines
		while (and (peek-char #\newline in nil nil)
	                   (read-char in)))
          (loop for line = (read-line in nil nil)
                while line do
	    (format out "~a~%" line))

If NLINES is 29000 on a 30000-line file, the latter version will be
a *lot* faster...  ;-}


-Rob

-----
Rob Warnock			<····@rpw3.org>
627 26th Avenue			<URL:http://rpw3.org/>
San Mateo, CA 94403		(650)572-2607