Hi,
I have this problem of loading a large file (160Mb) that contain
a huge S-expression into LISP system. I am running Allegro CL 4.3.1
on Ultra Sparc (Solaris). But when I tried to load this file, it tries
for 2 hrs before it gives up saying that it can not satisfy the
memory request.
I will explain the problem in detail.
I have a huge tree created by some other LISP program that is saved
into a file as a huge S-expression so that when it gets loaded into
another LISP system, the tree structured is re-created preserving
the links and some of the important attributes of each node.
But the problem is that when I tried to load, Allegor CL gives up
with a memory problem. I know this is the system problem not
ACL problem. I have 64Mb of RAM and a 1Gb of swap partition.
ACL spends half of the CPU time swaping and garbage collecting.
My questions are as follows:
1) Are there any ACL parameters/Solaris parameters that
I should change to handle large data?
2) Are there any better, more efficient ways of saving and re-creating
large set of objects. I know that the real solution is to go
for full-fledged OODMS. But that is not a solution for now.
I just need to handle with basic CL and file system.
3) Why does one CL was able to handle the large tree and able to save
into a file, but the same size tree can not be created(loaded)
into another LISP process. Both tehse LISP processes are
on the same machine and same LISP. (I am running each of these LISP
processes as exclusively so that the two are not running at the
same time). Are there any extra overheads inharent in constructing
objects from LISP forms from a file?
4) Dumping in a binary form is not a solution as the objects in the
source tree and the objects in the target tree are not exactly
instances of the same class. They differ in a minor ways. What I
was doing is changing the class-names and attributes names to
the target object system while saving into a file.
5) Does compilation of this hge file help? (Assming it can be compiled
without any problem, which, I suspect, might not be the case.
(have anybody tried to compile a LISP file of 160Mb size?).
Any help from LISP hackers is very much apreciated.
srini
--------------------
Srinivas Palthepu Email: ·····@cs.usask.ca
ARIES laboratory, Phones: (306) 966-8654 (lab)
Department of Computer Science, (306) 966-4759 (office)
University of Saskatchewan, (306) 373-6724 (home)
57 Campus Dr, Saskatoon, SK, S7N 5A9 Fax: (306) 966-4884
From: Laurent Arditi
Subject: Re: Help needed in handling large LISP file
Date:
Message-ID: <34EBEEA6.4F4B482E@tif.ti.com>
Are you sure your tree actually was not a graph?
Even if you didn't build it as a graph, you may have unintentional
structure
sharing. This is why, you were able to construct the tree and save it
but
produced a so huge file.
The solution is:
1) to really handle your structure as a graph and not as a tree. So you
will save
a graph which will be much smaller than the tree.
Or
2) to save the tree as you do now. But when you load it, do not use a
single (read).
Instead, use a deep-recursive function to load it node by node, and
creating a new node
only when it does not already exist.
P. Srinivas wrote:
>
> Hi,
>
> I have this problem of loading a large file (160Mb) that contain
> a huge S-expression into LISP system. I am running Allegro CL 4.3.1
> on Ultra Sparc (Solaris). But when I tried to load this file, it tries
> for 2 hrs before it gives up saying that it can not satisfy the
> memory request.
>
> I will explain the problem in detail.
>
> I have a huge tree created by some other LISP program that is saved
> into a file as a huge S-expression so that when it gets loaded into
> another LISP system, the tree structured is re-created preserving
> the links and some of the important attributes of each node.
> But the problem is that when I tried to load, Allegor CL gives up
> with a memory problem. I know this is the system problem not
> ACL problem. I have 64Mb of RAM and a 1Gb of swap partition.
> ACL spends half of the CPU time swaping and garbage collecting.
> ...
--
============================================================ |\/\/\/|
Laurent Arditi Texas Instruments France | |
Email: ·······@tif.ti.com MS. 21, BP 5 ( O O )
Tel: +33 (0)4 9322 2856 06271 Villeneuve-Loubet Cedex o |
From: Erik Naggum
Subject: Re: Help needed in handling large LISP file
Date:
Message-ID: <3096867616328513@naggum.no>
* Laurent Arditi -> P. Srinivas
| Are you sure your tree actually was not a graph? Even if you didn't
| build it as a graph, you may have unintentional structure sharing. This
| is why, you were able to construct the tree and save it but produced a so
| huge file.
|
| The solution is:
it doesn't appear that this is the problem, but if it is, I'd venture
that
(setq *print-circle* t)
should take care of these problems in the writing Lisp image. however,
it _might_ take a little longer to write the tree to disk now...
if Allegro CL complains about memory problems before it finishes loading,
it might be because it is trying too hard to keep objects in new-space.
the macro TENURING in the FRANZ (or EXCL) package will try to make most
allocations in old-space. if you know you will need a lot of old-space,
you can also pre-allocate that space with SYSTEM:RESIZE-AREAS, as in:
(system:resize-areas :old (* 160 1024 1024))
Allegro CL can also be built with a larger maximum heap estimate that
defaults to about 120M, but I don't know what this would do to help you.
there's also a LOAD-APPLICATION macro that might be useful. consult the
User Guide for explanations on how to use this and the other functions
mentioned. see also ROOM which should be helpful in reporting where
memory is used.
#:Erik
--
God grant me serenity to accept the code I cannot change,
courage to change the code I can, and wisdom to know the difference.