I am looking for ways to read and write blocks of data fast in Scheme
or Common Lisp, but I can only find per-character IO-routines or, at
best, per-line. what I would like to do is to read entire files using
read() or equivalent -- or perhaps something like mmap().
can anyone recommend something or would I have to hack this into my
Scheme/CL environment myself?
-Bj�rn
--
Bj�rn Borud <·····@guardian.no> | "The Net interprets censorship
<URL:http://www.pvv.unit.no/~borud/> | as damage and routes around it."
UNIX person, one of "them" | - John Gilmore
* Bjorn Borud
| I am looking for ways to read and write blocks of data fast in Scheme or
| Common Lisp, but I can only find per-character IO-routines or, at best,
| per-line. what I would like to do is to read entire files using read()
| or equivalent -- or perhaps something like mmap().
ANSI Common Lisp has `read-sequence' and `write-sequence'. I have answered
the exact same question from somebody else from GUARDIAN.NO in no.lisp, and
I don't feel like translating. however, here's the code I suggested:
(let ((block (make-array '(8192) :element-type 'character)))
(with-open-file (stream <file> :element-type 'character)
... (read-sequence block stream) ...))
in Franz, Inc's, Allegro Common Lisp 4.3 for Unix, this uses the `read'
system call directly from file descriptor to the block, at least as far as
I can determine, and it also returns the same value as the `read' system
call returned.
in Allegro, you can also give an array static allocation, so you are
guaranteed that it won't be moved during garbage collection
I have not seen anything that supports `mmap', and I can't offhand see how
I could do it work since `mmap' interacts with memory in weird ways and I
don't know how or if that would upset Allegro. it might be a trivial
question of using the Foreign Function Interface extension to Common Lisp.
I don't have time to investigate that right now.
#\Erik
--
1,3,7-trimethylxanthine -- a basic ingredient in quality software.
Bjorn Borud <·····@lucifer.guardian.no> writes:
>I am looking for ways to read and write blocks of data fast in Scheme
>or Common Lisp, but I can only find per-character IO-routines or, at
>best, per-line. what I would like to do is to read entire files using
>read() or equivalent -- or perhaps something like mmap().
>can anyone recommend something or would I have to hack this into my
>Scheme/CL environment myself?
The problem is not only getting block file operations, but also to
preallocate buffers of raw data that are reused on the next
operations, otherwise you won't get the performance you desire. This
is generally easy to do in Common Lisp and very difficult in Scheme.
ANSI CL's read-sequence and write-sequence do what you want, some
implementations already offer these.
The Lisp-FAQ refers to a fast I/O lisp file which lists low-level
block-access and memory allocation for raw data blocks for a number of
implementations. Let me know if you need it for an implementation you
don't have it for or can't locale the file.
Otherwise, functions like these are rather easy to write as foreign
functions, as long as the data type you need in Lisp has a fast
conversion to something you can access in C. In Scheme, you generally
convert Data in C anyway and will face real problems allocating
storage for the blocks, while most CL implementations do this from
Lisp and offer Lisp constructs to hold raw buffers in Lisp.
I plan to use mmap in CMUCL for strings, if you are interestedt,
please remind me in few weeks.
Last, you'll have to note that many Lisp data types are represented in
a form that might not be compatible with the format C uses. Especially
mmap()ing requires compatible data types (i.e. character strings in
CMUCL).
Martin
--
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
···············@wavehh.hanse.de http://cracauer.cons.org Fax.: +4940 5228536
"As far as I'm concerned, if something is so complicated that you can't ex-
plain it in 10 seconds, then it's probably not worth knowing anyway"- Calvin
In article <··············@lucifer.guardian.no> Bjorn Borud <·····@lucifer.guardian.no> writes:
;;; I am looking for ways to read and write blocks of data fast in Scheme
;;; or Common Lisp, but I can only find per-character IO-routines or, at
;;; best, per-line. what I would like to do is to read entire files using
;;; read() or equivalent -- or perhaps something like mmap().
;;;
;;; can anyone recommend something or would I have to hack this into my
;;; Scheme/CL environment myself?
http://www.cs.cmu.edu/afs/cs/project/ai-repository/ai/lang/lisp/code/io/fast_io/0.html
--
Happy Happy Joy Joy.
--
Patric Jonsson,·······@nada.kth.se;"always mount a scratch monkey"-jargon-file.
Bjorn Borud <·····@lucifer.guardian.no> wrote:
+---------------
| I am looking for ways to read and write blocks of data fast in Scheme
| or Common Lisp, but I can only find per-character IO-routines or, at
| best, per-line. what I would like to do is to read entire files using
| read() or equivalent...
+---------------
Jaffer's "SCM" implementation of Scheme provides a non-standard Scheme
datatype called "uniform vector" (much like CL's specialized arrays),
allocated with:
(make-uniform-vector length prototype)
The type/value of "prototype" determines the specialization of the
vector. Such vectors may be shortened or lengthened with:
(vector-set-length! uve length)
The procedures:
(uniform-vector-read! uve)
(uniform-vector-read! uve port)
(uniform-vector-write uve)
(uniform-vector-write uve port)
read or write (uniform-vector-length uve) binary objects from/to the
(possibly default) input or output port, and return the actual number
of objects read/written [e.g., if you get EOF on input, the number
read may be less than (uniform-vector-length uve)]. When running on
Unix, these procedures use the underlying Unix read() and write() calls
directly. Code such as this:
(define BUFSIZ 16384)
(define buf (make-uniform-vector BUFSIZ #\a ))
(define (cat-all-bytes port)
(do ((n (uniform-vector-read! buf port)
(uniform-vector-read! buf port)))
((<= n 0))
(vector-set-length! buf n)
(uniform-vector-write buf)
(vector-set-length! buf BUFSIZ)))
will run at approximately disk speed (the same as when written in C).
-Rob
-----
Rob Warnock, 7L-551 ····@sgi.com
Silicon Graphics, Inc. http://reality.sgi.com/rpw3/
2011 N. Shoreline Blvd. Phone: 415-933-1673 FAX: 415-933-0979
Mountain View, CA 94043 PP-ASEL-IA
····@rigden.engr.sgi.com (Rob Warnock) writes:
>
> Bjorn Borud <·····@lucifer.guardian.no> wrote:
> +---------------
> | I am looking for ways to read and write blocks of data fast in Scheme
> | or Common Lisp, but I can only find per-character IO-routines or, at
> | best, per-line. what I would like to do is to read entire files using
> | read() or equivalent...
> +---------------
>
> Jaffer's "SCM" implementation of Scheme provides a non-standard Scheme
^^^^^^^^^^^^
> datatype called "uniform vector" (much like CL's specialized arrays),
^ insert "standard"
remove "specialized"
Sorry, I couldn't resist!
Cheers
--
Marco Antoniotti - Resistente Umano
===============================================================================
International Computer Science Institute | ·······@icsi.berkeley.edu
1947 Center STR, Suite 600 | tel. +1 (510) 643 9153
Berkeley, CA, 94704-1198, USA | +1 (510) 642 4274 x149
===============================================================================
...it is simplicity that is difficult to make.
...e` la semplicita` che e` difficile a farsi.
Bertholdt Brecht
Marco Antoniotti <·······@crawdad.icsi.berkeley.edu> writes:
>
> ····@rigden.engr.sgi.com (Rob Warnock) writes:
>
> >
> > Bjorn Borud <·····@lucifer.guardian.no> wrote:
> > +---------------
> > | I am looking for ways to read and write blocks of data fast in Scheme
> > | or Common Lisp, but I can only find per-character IO-routines or, at
> > | best, per-line. what I would like to do is to read entire files using
> > | read() or equivalent...
> > +---------------
> >
> > Jaffer's "SCM" implementation of Scheme provides a non-standard Scheme
> ^^^^^^^^^^^^
> > datatype called "uniform vector" (much like CL's specialized arrays),
> ^ insert "standard"
> remove "specialized"
>
> Sorry, I couldn't resist!
>
I thought CL allowed but did not require specialized arrays. That is,
an implementation was free to implement them as (array t).
Ray