From: Bruce L. Lambert
Subject: Reading Large Data Files with Kyoto Common Lisp
Date: 
Message-ID: <4bsekt$3cq2@tigger.cc.uic.edu>
Happy Holidays Lispers:

I'm doing some document clustering experiments (information retrieval),
and I am trying to read a relatively large similarity matrix (3.5 Mbytes)
into a hash table so I can have fast access to the similarity values
during the clustering phase of the algorithm.

Unfortunaely, Austin Kyoto Common Lisp (AKCL) gives me an out of memory
error when I try to read in data structures larger than about 1.5 Mbytes.
It says to allocate more relocatable pages, but there's an upper limit on
the number of relocatable pages one can allocate.

I'm running AKCL version 1.615 on a NeXTstation, NeXTStep 3.2, with a
68040 chip and 20 Mbytes of RAM. (I know, it's a fairly primitive
platform).

Are these physical memory limitations or software limitations intrinsic
to AKCL? Any suggestions on how to handle this similarity matrix more
efficiently, in terms of both time and space. BTW, the values in the
similarity matrix are a combination of zeroes, ones, and real numbers.

There are about 1.6 million entries in the lower diagonal of the matrix
I'm currently struggling with.

Thanks.

-bruce

Bruce L. Lambert, Ph.D.
University of Illinois at Chicago
········@uic.edu
·····@ludwig.pmad.uic.edu
+1 (312) 996-2411