kp gores <·····@sip.medizin.uni-ulm.de> writes:
> in my system (cmucl under sun solaris) i use a hash-table to store my
> dictionary.
> at startup, i fill the hash-table by "load"-ing the dictionary from a
> file. each line in the file looks like:
> (setf (gethash "year" dict) (make-instance 'dictentry...
>
> the file has about 13000 entries and loading/evaluating takes some time.
> memory requirements are high too.
>
> is there a better way to handle this problem?
There are a couple of other approaches you could persue. But given
that you already have taken the "Dictionary is just another Lisp file"
approach, there are optimizations to this approach that you could try
first:
- IF your dictionary changes infrequently (which I'd assume), but you
startup more often, you could either compile the dictionary file,
and load the compiled version, or load it once and dump an image
that already contains your dictionary.
- IF your dictionary changes more often, because of smallish local
changes (e.g. changed entries, etc.), you could still take the same
approach, but additionally loading small patch files after start-up
to incorporate changes. You might want to write an "un-dumper",
that takes the in-core dictionary, and reproduces your external file
format, to periodically incorporate the accumulated patch files into
the main dictionary file, so that you can start afresh from this new
base file.
If those approaches don't work for you, you can take the common
approaches of defining specialized file formats for your dictionary,
and specialized reading/writing routines. In this way you can gain
some speed over the general Lisp reader, and gain interoperability
with reader-challenged software, BUT you loose the flexibility of the
Lisp reader, too. It's also a bit more work.
Regs, Pierre.
--
Pierre Mai <····@acm.org> PGP and GPG keys at your nearest Keyserver
"One smaller motivation which, in part, stems from altruism is Microsoft-
bashing." [Microsoft memo, see http://www.opensource.org/halloween1.html]
····@acm.org (Pierre R. Mai) writes:
> kp gores <·····@sip.medizin.uni-ulm.de> writes:
>
> > the file has about 13000 entries and loading/evaluating takes some time.
> > memory requirements are high too.
> >
> > is there a better way to handle this problem?
>
> - IF your dictionary changes infrequently (which I'd assume), but you
> startup more often, you could either compile the dictionary file,
> and load the compiled version, or load it once and dump an image
> that already contains your dictionary.
>
If your dictionary is being used as a runtime constant (i.e. you load
it at the beginning and keep it around for a long time) you should try
to allocate it in PURE or CONSTANT space (whatever your lisp calls it).
If you have a generational garbage collector, you don't want the GC to
copy the whole thing several times before it decides that this structure
is long lived. If it really won't contain volatile pointers, you can
put it in a chunk of memory that the GC won't even *see*.
--
~jrm
* kp gores wrote:
> i dont know how to dump an image in cmucl
>> If your dictionary is being used as a runtime constant (i.e. you load
>> it at the beginning and keep it around for a long time) you should try
>> to allocate it in PURE or CONSTANT space (whatever your lisp calls it).
> and i donT know how to do this in cmucl. any hints where to read about
> that?
(save-lisp <core file> :purify t) will save an image and put any live
objects into space not looked at by the GC. I'm not sure how to make
them sharable (this may do that).
--tim
kp gores <·····@sip.medizin.uni-ulm.de> writes:
> In article <·············@alum.mit.edu>, Joe Marshall
> <·········@alum.mit.edu> wrote:
>
> > If your dictionary is being used as a runtime constant (i.e. you load
> > it at the beginning and keep it around for a long time) you should try
> > to allocate it in PURE or CONSTANT space (whatever your lisp calls it).
>
> and i donT know how to do this in cmucl. any hints where to read about
> that?
>
> > If you have a generational garbage collector, you don't want the GC to
> > copy the whole thing several times before it decides that this structure
> > is long lived. If it really won't contain volatile pointers, you can
> > put it in a chunk of memory that the GC won't even *see*.
>
> neither do i know about that :-(.
>
I've never used cmucl, so I don't know how to it either. But it is a
lot easier to figure out how to do it once you know what it is you
want to do.
--
~jrm