hi,
I've been working with very large data files. Before doing anything,
I always have to load that data file, which is just a file of
s-expressions. Something like:
(setq *data* #(...very long array, many megs))
I was wondering if this is the best way to go about doing this. the
Lisp reader is not light-speed to say the least, so I was thinking
that instead of printing the object to the file, using the `print'
function, there might be a better way, or maybe a way to compile it,
so the Lisp reader can just shove it all directly into memory without
much effort.
thanks,
dave
I had a similar problem few months back. There does not seem
to be any good general solution to this problem except having
a good persistant object store such as AllegorStore, which is essentially
a database. But there are some tricks that people suggested
which migth work in your case. One such trick is dumping
the data in a binary form which can be loaded back into memory
much faster. There is this FASL-WRITE and FASL-WRITE fro Allegro CL.
Srini
David Bakhash (·····@bu.edu) wrote:
: hi,
: I've been working with very large data files. Before doing anything,
: I always have to load that data file, which is just a file of
: s-expressions. Something like:
: (setq *data* #(...very long array, many megs))
: I was wondering if this is the best way to go about doing this. the
: Lisp reader is not light-speed to say the least, so I was thinking
: that instead of printing the object to the file, using the `print'
: function, there might be a better way, or maybe a way to compile it,
: so the Lisp reader can just shove it all directly into memory without
: much effort.
: thanks,
: dave
--
--------------------
Srinivas Palthepu Email: ·····@cs.usask.ca
ARIES laboratory, Phones: (306) 966-8654 (lab)
Department of Computer Science, (306) 966-4759 (office)
University of Saskatchewan, (306) 373-6724 (home)
57 Campus Dr, Saskatoon, SK, S7N 5A9 Fax: (306) 966-4884
>: (setq *data* #(...very long array, many megs))
The soln might be system dependent, what implementation are you
using?
You might do an (apropos 'file) and see if anything looks promising.
To create a compiled file, append 3 files together, and do a
(compile-file ...) on the resulting file. The 3 files are:
1) a file containing a single line:
(setq *data*
2) your orig data file
3) a file containing a single line:
)
Also, if the the data doesn't change too often, you might
create a new lisp image that contains the data.
Foreign functions might be another area to investigate.
John
Depending on what the data looks like, it may be possible to compile
it. You could always try compiling the file with your data in it as a
first step to see if that improves things.
Also, unless the data changes a lot (in which case compiling wouldn't
really help much), you can also save runnable images in most lisp
systems that will allow you to restart the program with the data intact.
For example:
Allegro: (excl:dumplisp :name <name-for-image>)
MCL : (save-application <name-for-image>)
Lucid : (disksave <name-for-image>)
--
Thomas A. Russ, USC/Information Sciences Institute ···@isi.edu
Lisp compilers must know how to dump random data structure, so you can
do the following if you want to "checkpoint" a given variable: Create
a small file "checkpoint.lisp":
(setq *program-state* '#.*program-state*)
Compile it and everything that is referenced from this variable will
be reconstructed when you load the FASL file. Personally I find this
quite useful, so I write my programs in such a way that the program
state is actually represented by one or very few data structures that
I can make persistent using this idiom.
Obviously there are ways to defeat this, such as working with symbol
property lists a lot. Then your best bet may be dumping your entire
Lisp image to disk, which makes it hard to get rid of unwanted stuff.
--
Simon Leinen ·····@babar.switch.ch
SWITCH http://www.switch.ch/misc/leinen/
Who is General Failure & why's he reading my disk?
David Bakhash <·····@bu.edu> writes:
> hi,
>
> I've been working with very large data files. Before doing anything,
> I always have to load that data file, which is just a file of
> s-expressions. Something like:
>
> (setq *data* #(...very long array, many megs))
>
> I was wondering if this is the best way to go about doing this. the
> Lisp reader is not light-speed to say the least, so I was thinking
> that instead of printing the object to the file, using the `print'
> function, there might be a better way, or maybe a way to compile it,
> so the Lisp reader can just shove it all directly into memory without
> much effort.
>
This seems like a job for READ-SEQUENCE and WRITE-SEQUENCE. You
should change your setup and provide ways to read and write data saved
on "mass storage" :) After all, you must have generated the #(...very
long array, many megs) automagically. Having something like
(with-open-file (stuffed-file :direction :input)
(let* ((many-megs (compute-how-many-megs (file-length stuffed-file)))
(new-array (make-array many-megs #| You fill in the relevant
element-type and friends
|# ))
)
;; Also remember to check for errors! :)
(read-sequence new-array stuffed-file)
(setq *data* new-sequence))
should make your life easier.
--
Marco Antoniotti ===========================================
PARADES, Via San Pantaleo 66, I-00186 Rome, ITALY
tel. +39 - (0)6 - 68 80 79 23, fax. +39 - (0)6 - 68 80 79 26
http://www.parades.rm.cnr.it