I am a newbie in Common Lisp.
After browing the reference manual, I
cannot figure out how to do binary io
in common lisp.
For eample, I have a file with 100 4 byte integers,
is it possible to read in these integers as an array without
using external C functions.
Thanks in advance.
Su, Yuan-Liang
Institue for Information Industry
"Su, Yuan-Liang" <···@iii.org.tw> writes:
> I am a newbie in Common Lisp.
>
> After browing the reference manual, I
> cannot figure out how to do binary io
> in common lisp.
>
> For eample, I have a file with 100 4 byte integers,
> is it possible to read in these integers as an array without
> using external C functions.
>
> Thanks in advance.
>
> Su, Yuan-Liang
>
> Institue for Information Industry
In a perfect CL world you would basically do
(defun read-4-bytes-stream (filename)
(let ((buffer (make-array *some-buffer-size*
:element-type '(unsigned-byte 4)
:initial-element 0)))
(with-open-file (f :direction :input
:if-does-not-exist :error
:element-type '(unsigned-byte 4))
(read-sequence f buffer))))
Of course, looping till the end of file and error checking are missing
from this function. But the skeleton is there.
Cheers
--
Marco Antoniotti ===========================================
PARADES, Via San Pantaleo 66, I-00186 Rome, ITALY
tel. +39 - (0)6 - 68 10 03 17, fax. +39 - (0)6 - 68 80 79 26
http://www.parades.rm.cnr.it
Marco Antoniotti <·······@copernico.parades.rm.cnr.it> writes:
> "Su, Yuan-Liang" <···@iii.org.tw> writes:
>
> > For eample, I have a file with 100 4 byte integers,
> (let ((buffer (make-array *some-buffer-size*
> :element-type '(unsigned-byte 4)
Uh, I think you misread him, Marco. I did the same on first read.
4-byte, not 4-bit byte. To do 4-byte, you'll want to use
(unsigned-byte 8) and to make the array 4 times as long, and then
you'll want to make a post-pass that seams them up into a shorter
array of (unsigned-byte 32)... or (signed-byte 32), as the case may be.
Kent M Pitman <······@world.std.com> writes:
> Marco Antoniotti <·······@copernico.parades.rm.cnr.it> writes:
>
> > "Su, Yuan-Liang" <···@iii.org.tw> writes:
> >
> > > For eample, I have a file with 100 4 byte integers,
>
> > (let ((buffer (make-array *some-buffer-size*
> > :element-type '(unsigned-byte 4)
>
> Uh, I think you misread him, Marco. I did the same on first read.
> 4-byte, not 4-bit byte. To do 4-byte, you'll want to use
> (unsigned-byte 8) and to make the array 4 times as long, and then
> you'll want to make a post-pass that seams them up into a shorter
> array of (unsigned-byte 32)... or (signed-byte 32), as the case may be.
>
Yep. I did misread him. However, I believe that the definition of
STREAM-ELEMENT-TYPE requires that the open stream yields elements of -
at least - the specified element type.
I.e.
(with-open-file (f "some-binary-file.bin"
:direction :input
:element-type '(unsigned-byte 32))
(stream-element-type f))
should return a type specifier as wide as or wider that (UNSIGNED-BYTE 32).
I'd expect then READ-SEQUENCE of an appropriate matching sequence
element type would behave consequentely. CMUCL seems to behave accordingly.
Cheers
--
Marco Antoniotti ===========================================
PARADES, Via San Pantaleo 66, I-00186 Rome, ITALY
tel. +39 - (0)6 - 68 10 03 17, fax. +39 - (0)6 - 68 80 79 26
http://www.parades.rm.cnr.it
Marco Antoniotti <·······@copernico.parades.rm.cnr.it> writes:
> Yep. I did misread him. However, I believe that the definition of
> STREAM-ELEMENT-TYPE requires that the open stream yields elements of -
> at least - the specified element type.
>
> (with-open-file (f "some-binary-file.bin"
> :direction :input
> :element-type '(unsigned-byte 32))
> (stream-element-type f))
>
> should return a type specifier as wide as or wider that (UNSIGNED-BYTE 32).
>
> I'd expect then READ-SEQUENCE of an appropriate matching sequence
> element type would behave consequentely. CMUCL seems to behave accordingly.
I think the spec is fuzzier than you're giving it credit for.
If you open in (UNSIGNED-BYTE n), I think it's expecting to read only
something written in that format unless it's one of the standard sizes.
I remember seeing an implementation that implemented (UNSIGNED-BYTE 1)
by storing whole words of 1's or 0's on write, and undid the effect on
read. We didn't fix that problem in the general case, e.g., unsigned
byte 7 might still be written as 8-bit words with a high 0 in each word,
but we did require certain sizes to be really present as expected.
I don't recall what they are, but I think they're enumerated.
Now you are armed with information to appreciate why they're enumerated.
(Maybe 32 is in the list--I don't recall. I only remember 8 is. And
probably 1.)
From: Howard R. Stearns
Subject: Re: Binary file io in standard common lisp
Date:
Message-ID: <36DD4F3A.4B108F5A@elwood.com>
Kent, I think I must be misunderstanding you, because I do not think it
is the case that ANSI-CL enumerates any particular element-types:
OPEN:
"With regard to the :element-type option, if a type is requested that is
not supported by the file system, a substitution of
types such as that which goes on in upgrading is permissible. As a
minimum requirement, it should be the case that
opening an output stream to a file in a given element type and later
opening an input stream to the same file in the same
element type should work compatibly."
STREAM-ELEMENT-TYPE:
"Examples:
;; Note that the stream must accomodate at least the specified type,
;; but might accomodate other types. Further note that even if it does
;; accomodate exactly the specified type, the type might be specified in
;; any of several ways.
(with-open-file (s "test" :element-type '(integer 0 1)
:if-exists :error
:direction :output)
(stream-element-type s))
=> INTEGER
OR=> (UNSIGNED-BYTE 16)
OR=> (UNSIGNED-BYTE 8)
OR=> BIT
OR=> (UNSIGNED-BYTE 1)
OR=> (INTEGER 0 1)
OR=> (INTEGER 0 (2))"
My understanding is that the implementation can use whatever "word size"
it wants, but of course, the marketplace might expect particular word
sizes to be reasonably implemented -- i.e., compatible with some
standard definition on the platform.
Just to confuse the issue, I further believe that external-format should
be used to define such issues as byte-order and padding. For example,
if someone wanted 7 bit bytes to not be padded with one zero bit, they
would need the implementation to supply some combination of element-type
and external-format that did this. Here's the def of external-format
from OPEN:
"The external-format is meaningful for any kind of file stream whose
element type is a subtype of character. This
option is ignored for streams for which it is not meaningful;
however, implementations may define other element types
for which it is meaningful. The consequences are unspecified if a
character is written that cannot be represented by
the given external file format."
I would like to share ideas with anyone who wants to work on a protocol
that extends CLOS-STREAMS (as in Gray and CLIM) to provide both
CLOS-PATHNAMES and CLOS-FILE-SYSTEM-OPERATIONS. This would allow users
to create their own combinations of element-type and external-format,
make use of implementation-provided buffering utilitites, and have
everything work with straight ANSI CL application code.
Kent M Pitman wrote:
>
> Marco Antoniotti <·······@copernico.parades.rm.cnr.it> writes:
>
> > Yep. I did misread him. However, I believe that the definition of
> > STREAM-ELEMENT-TYPE requires that the open stream yields elements of -
> > at least - the specified element type.
> >
> > (with-open-file (f "some-binary-file.bin"
> > :direction :input
> > :element-type '(unsigned-byte 32))
> > (stream-element-type f))
> >
> > should return a type specifier as wide as or wider that (UNSIGNED-BYTE 32).
> >
> > I'd expect then READ-SEQUENCE of an appropriate matching sequence
> > element type would behave consequentely. CMUCL seems to behave accordingly.
>
> I think the spec is fuzzier than you're giving it credit for.
> If you open in (UNSIGNED-BYTE n), I think it's expecting to read only
> something written in that format unless it's one of the standard sizes.
>
> I remember seeing an implementation that implemented (UNSIGNED-BYTE 1)
> by storing whole words of 1's or 0's on write, and undid the effect on
> read. We didn't fix that problem in the general case, e.g., unsigned
> byte 7 might still be written as 8-bit words with a high 0 in each word,
> but we did require certain sizes to be really present as expected.
> I don't recall what they are, but I think they're enumerated.
> Now you are armed with information to appreciate why they're enumerated.
> (Maybe 32 is in the list--I don't recall. I only remember 8 is. And
> probably 1.)
"Howard R. Stearns" <······@elwood.com> writes:
> Kent, I think I must be misunderstanding you, because I do not think it
> is the case that ANSI-CL enumerates any particular element-types:
I haven't done the book research so I'll defer to the specific data
you dredged up. I mentioned I was working only from memory and that's
not always perfect.
Incidentally, I do agree with you that external-format should help to
resolve some of these issues, but since there is no external format
other than :default, that's kind of a weak solution. A lot of the
game is in naming those external formats and figuring out how the
format interacts with the element type. (For example, an element type
might actually vary the file format, or the file format might describe
a set of appropriate element types. Those are incompatible theories,
but nothing about the spec decides between them).
> My understanding is that the implementation can use whatever "word size"
> it wants, but of course, the marketplace might expect particular word
> sizes to be reasonably implemented -- i.e., compatible with some
> standard definition on the platform.
Right. I got off on the tangent of this word size bit because I
originally had misread this request as requesting 4-bit bytes. I
later repaired my understanding but didn't "deactivate" the
corresponding concern about byte sizes. Maybe my ill-placed focus on
this irrelevancy is what is confusing you.
> I would like to share ideas with anyone who wants to work on a protocol
> that extends CLOS-STREAMS (as in Gray and CLIM) to provide both
> CLOS-PATHNAMES and CLOS-FILE-SYSTEM-OPERATIONS. This would allow users
> to create their own combinations of element-type and external-format,
> make use of implementation-provided buffering utilitites, and have
> everything work with straight ANSI CL application code.
This sounds like a productive thing to do, especially if you don't try
too hard to worry about how to connect it up to any organization's formal
work and focus instead first on getting something useful. :-)
From: Lieven Marchand
Subject: Re: Binary file io in standard common lisp
Date:
Message-ID: <7bmgs7$fa0$2@nickel.uunet.be>
"Howard R. Stearns" <······@elwood.com> writes:
> I would like to share ideas with anyone who wants to work on a protocol
> that extends CLOS-STREAMS (as in Gray and CLIM) to provide both
> CLOS-PATHNAMES and CLOS-FILE-SYSTEM-OPERATIONS.
I recently searched the net for the Gray stream proposal but came up
empty. Is it still available somewhere?
--
Lieven Marchand <···@bewoner.dma.be>
------------------------------------------------------------------------------
It was six months of C++ that made me determined to find something else to do
with my life. #:Erik Naggum
From: Will Fitzgerald
Subject: Re: Binary file io in standard common lisp
Date:
Message-ID: <7bokdg$ick@news.net-link.net>
Could someone enlighten me as to why these streams are "Gray"? And what are
"green" streams?
>In article <············@nickel.uunet.be>,
> Lieven Marchand <···@bewoner.dma.be> writes:
>> "Howard R. Stearns" <······@elwood.com> writes:
>>
>>> I would like to share ideas with anyone who wants to work on a protocol
>>> that extends CLOS-STREAMS (as in Gray and CLIM) to provide both
>>> CLOS-PATHNAMES and CLOS-FILE-SYSTEM-OPERATIONS.
>>
>> I recently searched the net for the Gray stream proposal but came up
>> empty. Is it still available somewhere?
>>
>
> I don't know where the original is located but the CLIM version can be
found
>at http://www.mikemac.com/mikemac/clim/page348.html
>
>
> Mike McDonald
> ·······@mikemac.com
From: Howard R. Stearns
Subject: Re: Binary file io in standard common lisp
Date:
Message-ID: <36E00E70.B7702F99@elwood.com>
Will Fitzgerald wrote:
> Could someone enlighten me as to why these streams are "Gray"? And what are
> "green" streams?
David Gray of TI made a proposal to X3J13 while the commitee was
preparing the ANSI standard for CL. The proposal was called
STREAM-DEFINITION-BY-USER, and is commonly called CLOS-STREAMS. The
basic idea is that streams are implemented by some documented CLOS
classes, and the CL stream functions are implemented using some
documented generic functions which can be specialized by the user on
user-defined stream classes. While I don't know the details on why
(Barmar? Kent? Steve H.?), my understanding is that it was not yet
stable/complete enough to be included in the standard.
(For what its worth, I agree on the grounds that it isn't really
complete enough to do the things it's meant to, and that solving those
problems might require changes. In short, it's bad to standardize too
early. Remember Kent's admonishment that standardization should not be
about design, but simply codifying existing practice. Anyway, my desire
to extend Gray's work is what started this thread, for anyone coming in
late.)
Some form of CLOS streams is in most ANSI CL implementations, and is
often documented by the vendor as an extension. Collecting some of the
various references in one place, we have:
- proposal from the archives
ftp://parcftp.xerox.com/pub/cl/cleanup/mail/stream-definition-by-user.mail,
as pointed out by Arthur
- CLIM manual, http://www.mikemac.com/mikemac/clim/page348.html, as
pointed out by Mike. (See also
http://www.elwood.com/alu/table/references.htm#clim for references on
CLIM generally.)
- Eclipse documentation, with some slight additions and qualifications,
http://www.elwoodcorp.com/eclipse/extensions.htm#streams
Some common concerns about the existing CLOS-streams implementations are
that they don't make buffering and external-formatting protocols
available to the user, even though each implementation must surely have
them internally. Also, while CLOS-streams does integrate user defined
and user created streams with the CL streams operations, it does not
integrate them with the file-system operations, or with pretty-printing
streams. I have done a lot of this for Eclipse, but not documented it
yet for users because I'm not sure its ready. In general, though, the
big ideas are:
- extend clos-streams with more clases for buffering-streams, and
streams built on them such as string-streams and file-streams.
- give access to generic-functions for manipulating buffers
- provide a similar set of protocols for pathnames and pathname
operations, so that users can define their own pathname classes which
can participate in both pathname and file system generic functions
- give access to generic-functions for file system operations
- make sure OPEN can make use of all this
- integrate clos-streams with "standard" pretty-printing
implementations (such as Roger Waters' XP and its antecedents), so that
the benefits of all this aren't lost as soon as soon as someone sets
*print-pretty* to t.
- although I haven't done any work on it yet, it would be a shame to do
all the above and still not have a documented user-extendable protocol
for sockets and TCP/IP.
I have no idea what "green streams" are.
From: Will Fitzgerald
Subject: Re: Binary file io in standard common lisp
Date:
Message-ID: <7bpfae$i1q@news.net-link.net>
Howard R. Stearns wrote in message <·················@elwood.com>...
>Will Fitzgerald wrote:
>> Could someone enlighten me as to why these streams are "Gray"? And what
are
>> "green" streams?
>
> [Answers gray streams question]
Thanks, Howard.
>
>I have no idea what "green streams" are.
That's because I meant "green threads," apparently a multithreading concept.
Sorry.
From: Lieven Marchand
Subject: Re: Binary file io in standard common lisp
Date:
Message-ID: <7brdoi$s8c$1@xenon.inbe.net>
Arthur Lemmens <·······@simplex.nl> writes:
> I had some trouble finding this proposal myself a while ago.
> Searching DejaNews for "Gray streams" helped.
> It's at
>
> ftp://parcftp.xerox.com/pub/cl/cleanup/mail/stream-definition-by-user.mail
Thanks. I knew about that repository but I hadn't thought to look for it
in the cleanups.
--
Lieven Marchand <···@bewoner.dma.be>
------------------------------------------------------------------------------
It was six months of C++ that made me determined to find something else to do
with my life. #:Erik Naggum
"Su, Yuan-Liang" <···@iii.org.tw> writes:
> I am a newbie in Common Lisp.
Welcome.
> After browing the reference manual, I cannot figure out how to do
> binary io in common lisp.
>
> For eample, I have a file with 100 4 byte integers, is it possible
> to read in these integers as an array without using external C
> functions.
Common Lisp doesn't guarantee that its I/O will do the thing you need
for 4-byte data, in the sense that it can write and later read
fractional and multi-byte data but it may not do so in a format you
expect. Ask your vendor what its conventions are and what can be relied
upon. However, I think 8-bit byte data is pretty standard and if you
want to read this you can read it as individual bytes and then reconstruct
it into integers by using appropriate calls to dpb. I've done
this successfully myself many times.
See the doc in the Common Lisp HyperSpec for read-sequence and dpb:
http://www.harlequin.com/education/books/HyperSpec/Body/fun_read-sequence.html
http://www.harlequin.com/education/books/HyperSpec/Body/fun_dpb.html