From: Zach Beane
Subject: Re: Is there an efficient way to get file length w/o reading file?
Date:
Message-ID: <m31x9el9a4.fsf@unnamed.xach.com>
Jeffrey Cunningham <····················@boeing.com> writes:
> I started to wonder if it was possible to do this in CL without
> having to actually read each file to determine it's length.
Have you considered FILE-LENGTH?
I've used something like the following:
(defun file-length-in-bytes (file)
(with-open-file (stream file :direction :input
:element-type '(unsigned-byte 8))
(file-length stream)))
Zach
From: Zach Beane
Subject: Re: Is there an efficient way to get file length w/o reading file?
Date:
Message-ID: <m3wtr6jpja.fsf@unnamed.xach.com>
Jeffrey Cunningham <····················@boeing.com> writes:
>> Have you considered FILE-LENGTH?
>>
>> I've used something like the following:
>>
>> (defun file-length-in-bytes (file)
>> (with-open-file (stream file :direction :input
>> :element-type '(unsigned-byte 8))
>> (file-length stream)))
>>
>
> Of course. That reads the entire file to determine its length.
No, it doesn't.
Zach
From: Zach Beane
Subject: Re: Is there an efficient way to get file length w/o reading file?
Date:
Message-ID: <m3oecijh43.fsf@unnamed.xach.com>
Jeffrey Cunningham <····················@boeing.com> writes:
> On Wed, 13 Apr 2005 18:11:05 -0400, Zach Beane wrote:
>
>> No, it doesn't.
>>
>>
> I just read a very long and contentious thread from this forum last year
> on a related topic wherein about the only thing anyone seemed to agree on
> was that (file-length stream) did read the file from one end to the other
> in order to determine it's length. I suppose I should write a little test
> case and find out, but that seems to be the consensus.
The behavior of (file-length stream) can vary by the stream's
element-type. Streams with an element-type of (unsigned-byte 8) can
have their file lengths determined very efficiently on Unix, and
indeed, implementations seem to take the efficient route for this.
There are no guarantees about how efficiently FILE-LENGTH will be
implemented. If you want to find out if your implementation is
efficient, you can try it an see. A newsgroup's consensus will not
change performance characteristics.
> The other point made repeatedly is that there is no POSIX API support
> (along with all kinds of reasons why it would be a bad idea).
There is a difference between no standard POSIX API and no POSIX API.
> I have SBCL, CLISP, and CMUCL installed on my machines. Currently, SBCL is
> connected to SLIME, but I switch from time to time to see what the
> differences are. Are there implementation differences in (file-length
> stream) between these LISP platforms that anyone knows about?
They all determine the length of streams of element-type
'(unsigned-byte 8) very efficiently.
Zach
On Wed, 13 Apr 2005 21:13:00 -0400, Zach Beane wrote:
eam) between these LISP platforms that anyone knows about?
>
> They all determine the length of streams of element-type '(unsigned-byte
> 8) very efficiently.
>
> Zach
It would appear - experimentally, at least - you are correct: the time it
takes to sum up file sizes to 2,367,479,940 bytes in the test case I ran
is virtually identical to the time it takes du -s to figure it out.
Curiously, it doesn't seem to matter (at least with SBCL) whether I
specify the element type or not.
I love this. I'm learning something new every day. Thanks.
-jeff cunningham
Jeffrey Cunningham <····················@boeing.com> writes:
> On Wed, 13 Apr 2005 16:19:15 -0400, Zach Beane wrote:
> > Have you considered FILE-LENGTH?
> >
> > I've used something like the following:
> >
> > (defun file-length-in-bytes (file)
> > (with-open-file (stream file :direction :input
> > :element-type '(unsigned-byte 8))
> > (file-length stream)))
> >
>
>
> Of course. That reads the entire file to determine its length. That's what
> I don't want to do. It would take forever for a large number of large
> files. What I want is something analogous to the fifth column of the
> output of 'ls -l'.
An intelligent implementation could avoid reading the file.
But otherwise, the answer is that it's not possible portably.
Drop down to the POSIX API.
--
__Pascal Bourguignon__ http://www.informatimago.com/
Grace personified,
I leap into the window.
I meant to do that.
On Thu, 14 Apr 2005 00:03:19 +0200, Pascal Bourguignon wrote:
> Jeffrey Cunningham <····················@boeing.com> writes:
>
> An intelligent implementation could avoid reading the file.
> But otherwise, the answer is that it's not possible portably.
> Drop down to the POSIX API.
How does one do that? Is there an example somewhere?
--jeff cunningham
Jeffrey Cunningham <·······@cunningham.net> writes:
> On Thu, 14 Apr 2005 00:03:19 +0200, Pascal Bourguignon wrote:
>
> > Jeffrey Cunningham <····················@boeing.com> writes:
>
> >
> > An intelligent implementation could avoid reading the file.
> > But otherwise, the answer is that it's not possible portably.
> > Drop down to the POSIX API.
>
> How does one do that? Is there an example somewhere?
(defun file-length-in-bytes (file)
;; Note the "EXT" implementation specific package.
(with-open-stream (in (ext:run-program "/bin/ls"
:arguments (list "-l" (namestring file))
:output :stream))
(read in)(read in)(read in)(read in)(read in)))
Alternatively, one could use FFI:
(defun file-length-in-bytes (file)
(multiple-value-bind (res stat) (linux:stat (namestring file))
(linux:|stat-st_size| stat)))
--
__Pascal Bourguignon__ http://www.informatimago.com/
In deep sleep hear sound,
Cat vomit hairball somewhere.
Will find in morning.
On Thu, 14 Apr 2005 05:35:05 +0200, Pascal Bourguignon wrote:
> Jeffrey Cunningham <·······@cunningham.net> writes:
>
>> On Thu, 14 Apr 2005 00:03:19 +0200, Pascal Bourguignon wrote:
>>
>> > Jeffrey Cunningham <····················@boeing.com> writes:
>>
>> >
>> > An intelligent implementation could avoid reading the file.
>> > But otherwise, the answer is that it's not possible portably.
>> > Drop down to the POSIX API.
>>
>> How does one do that? Is there an example somewhere?
>
> (defun file-length-in-bytes (file)
> ;; Note the "EXT" implementation specific package.
> (with-open-stream (in (ext:run-program "/bin/ls"
> :arguments (list "-l" (namestring file))
> :output :stream))
> (read in)(read in)(read in)(read in)(read in)))
>
> Alternatively, one could use FFI:
>
> (defun file-length-in-bytes (file)
> (multiple-value-bind (res stat) (linux:stat (namestring file))
> (linux:|stat-st_size| stat)))
Much obliged. I appreciate your taking the time to help.
-jeff cunningham