I've got a CLOS program which is supposed to manage a lot of data,
much more than there is actually avaible on my computer.
Until now, I asumed that writing the data of my program to
hard disk when the program finish would be enough.
But when the program grows and cross over the physical memory
avaiable on my system, the OS starts to swap memory to the hard
disk, until it exaust the entire system.
But there is also a big problem: my objects are interconected (even
they are recursively interconected). So my only solution is to store
all my data in a database.
Elsewhere, and this is the point, I don't want to rewrite my entire
program.
So, the question here is: is there any way to work with all my classes
using a database (not directly allocating memory), without changing
my source code?
I've been looking at
http://www.lispworks.com/documentation/HyperSpec/Body/c_object.htm
and thougth to start implementing some generic functions for my
problem, but I'm not sure I should do it... I don't even fully understand
what all of that generic functions do.
Thanks.
Javier wrote:
> I've got a CLOS program which is supposed to manage a lot of data,
> much more than there is actually avaible on my computer.
> Until now, I asumed that writing the data of my program to
> hard disk when the program finish would be enough.
> But when the program grows and cross over the physical memory
> avaiable on my system, the OS starts to swap memory to the hard
> disk, until it exaust the entire system.
> But there is also a big problem: my objects are interconected (even
> they are recursively interconected). So my only solution is to store
> all my data in a database.
>
> Elsewhere, and this is the point, I don't want to rewrite my entire
> program.
> So, the question here is: is there any way to work with all my classes
> using a database (not directly allocating memory), without changing
> my source code?
>
> I've been looking at
>
> http://www.lispworks.com/documentation/HyperSpec/Body/c_object.htm
>
> and thougth to start implementing some generic functions for my
> problem, but I'm not sure I should do it... I don't even fully understand
> what all of that generic functions do.
>
If your willing to pay for ACL, check out AllegroCache. It would
require inserting the code to open a database (open-file-database
"/path/to/a/directory/"), adding the class-option (:metaclass
persistent-class) to the definitions of the classes you want to be
persistent, and then deciding where and when you would like to (commit)
or (rollback) the db. If your objects are that interconnected, you
might not even need an indexing, you could just rely on virtual-pointer
chasing from a few key entry points...
http://www.franz.com/products/allegrocache/AllegroCache_White_Paper.pdf
http://www.franz.com/products/allegrocache/acachetutorial.pdf
hth
Nick
> Thanks.
········@gmail.com ha escrito:
> If your willing to pay for ACL, check out AllegroCache. It would
> require inserting the code to open a database (open-file-database
> "/path/to/a/directory/"), adding the class-option (:metaclass
> persistent-class) to the definitions of the classes you want to be
> persistent, and then deciding where and when you would like to (commit)
> or (rollback) the db. If your objects are that interconnected, you
> might not even need an indexing, you could just rely on virtual-pointer
> chasing from a few key entry points...
>
>
>
> http://www.franz.com/products/allegrocache/AllegroCache_White_Paper.pdf
> http://www.franz.com/products/allegrocache/acachetutorial.pdf
Thank you for the information, something like this is what I need.
Do you know of some open source library which does something similar?
Javier wrote:
> ········@gmail.com ha escrito:
>
> > If your willing to pay for ACL, check out AllegroCache. It would
> > require inserting the code to open a database (open-file-database
> > "/path/to/a/directory/"), adding the class-option (:metaclass
> > persistent-class) to the definitions of the classes you want to be
> > persistent, and then deciding where and when you would like to (commit)
> > or (rollback) the db. If your objects are that interconnected, you
> > might not even need an indexing, you could just rely on virtual-pointer
> > chasing from a few key entry points...
> >
> >
> >
> > http://www.franz.com/products/allegrocache/AllegroCache_White_Paper.pdf
> > http://www.franz.com/products/allegrocache/acachetutorial.pdf
>
> Thank you for the information, something like this is what I need.
> Do you know of some open source library which does something similar?
These articles compare two of popular serialization choices for cl:
http://xach.livejournal.com/76764.html
http://bc.tech.coop/blog/060604.html
I'm not sure what Rucksack's status is as a library. There's a CVS
repository, but it's not ASDF-installable. The Common Lisp Dictionairy
says it's maturity level is "experimental"...
take care
Nick
"Javier" <·······@gmail.com> schrieb im Newsbeitrag ·····························@l39g2000cwd.googlegroups.com...
> ········@gmail.com ha escrito:
> ...
>>
>> http://www.franz.com/products/allegrocache/AllegroCache_White_Paper.pdf
>> http://www.franz.com/products/allegrocache/acachetutorial.pdf
>
> Thank you for the information, something like this is what I need.
> Do you know of some open source library which does something similar?
>
I think the basic idea behind allegro-cache is the open source elephant which could be found at www.cliki.net.
Andreas
There's elephant which is a CLOS persistence lib on top of BDB (GPL-ed)
and it also has a very simple SQL (on top of CLSQL) backend.
There's cl-wdim (a web framework) which aims CLOS persistency on top of
SQL (one primary table per class) with an optimizing query compiler. It
supports boolean, string, integer, number, symbol, serialized, etc.
primitive types and classes and associations (with referential
integrity from both ends) as compound type.
Unfortunately we do not have any realeses yet, because the API is still
changing, but the darcs repos are available and there are quite a lot
of test cases.
levy
levy ha escrito:
> There's elephant which is a CLOS persistence lib on top of BDB (GPL-ed)
> and it also has a very simple SQL (on top of CLSQL) backend.
>
> There's cl-wdim (a web framework) which aims CLOS persistency on top of
> SQL (one primary table per class) with an optimizing query compiler. It
> supports boolean, string, integer, number, symbol, serialized, etc.
> primitive types and classes and associations (with referential
> integrity from both ends) as compound type.
>
> Unfortunately we do not have any realeses yet, because the API is still
> changing, but the darcs repos are available and there are quite a lot
> of test cases.
Thanks, I'm going to see both.
> Thank you for the information, something like this is what I need.
> Do you know of some open source library which does something similar?
I'm not completely sure if this is relevent, but we solved a similar
problem by using Berkeley DB (sleepycat, now oracle) and CL-Store. We
wrote a thin C veneer over BDB to store/retrieve the serialized objects
as a byte stream. It's fast enough such that our app meets our
soft-real-time requirements. CL-Store is especially cool because just
about any Lisp object may be serialized; thus, our complex CLOS
objects, which contain instances of other classes, hash tables, etc,
are handled seamlessly by CL-Store. Very nice...
Cheers, david
+ Javier <·······@gmail.com>:
| I've got a CLOS program which is supposed to manage a lot of data,
| much more than there is actually avaible on my computer.
| Until now, I asumed that writing the data of my program to
| hard disk when the program finish would be enough.
| But when the program grows and cross over the physical memory
| avaiable on my system, the OS starts to swap memory to the hard
| disk, until it exaust the entire system.
| But there is also a big problem: my objects are interconected (even
| they are recursively interconected). So my only solution is to store
| all my data in a database.
|
| Elsewhere, and this is the point, I don't want to rewrite my entire
| program.
| So, the question here is: is there any way to work with all my classes
| using a database (not directly allocating memory), without changing
| my source code?
Um, obviously not. The question is if you can get away with rewriting
only a very small portion of it, preferably just playing around with
class definitions and accessors.
I am not an expert at this, but let me try a few stabs in the dark
anyway. I am sure that more experienced folks will rip my suggestions
to pieces, which may well be educational.
You say your objects are interconnected, by which I suppose you mean
they reference each other directly. I think those references will
need to be replaced by some sort of indirect reference, say a key that
will let you get the referenced object's information out of a
database. If the reference resides in an object slot, and your
program never accesses the slots directly by using slot-value, you may
be in luck: Just rewrite the accessor to get the object by the key
that is now stored in the slot in place of the object itself. If the
object is not in-memory, the accessor will have to create it and
populate its slots from the database. You'll need a hash table or
something to keep track of those objects that are in-memory. The most
problematic part, I guess, is you need a strategy for removing rarely
used in-memory objects to make room for new ones. This seems like a
good use for weak hash tables if your Lisp has them. Or if it has
weak pointers, you could populate a hash table with weak pointers. If
it has neither, I don't know what you can do, but maybe someone else
has ideas.
--
* Harald Hanche-Olsen <URL:http://www.math.ntnu.no/~hanche/>
- It is undesirable to believe a proposition
when there is no ground whatsoever for supposing it is true.
-- Bertrand Russell
Harald Hanche-Olsen ha escrito:
> | So, the question here is: is there any way to work with all my classes
> | using a database (not directly allocating memory), without changing
> | my source code?
>
> Um, obviously not. The question is if you can get away with rewriting
> only a very small portion of it, preferably just playing around with
> class definitions and accessors.
Well, yes, I can modify the source code of my program, but I won't like
to completely change its style.
> You say your objects are interconnected, by which I suppose you mean
> they reference each other directly.
Yes, that is what I meant.
> I think those references will
> need to be replaced by some sort of indirect reference, say a key that
> will let you get the referenced object's information out of a
> database.
Yes, that is what I thought.
> If the reference resides in an object slot, and your
> program never accesses the slots directly by using slot-value, you may
> be in luck: Just rewrite the accessor to get the object by the key
> that is now stored in the slot in place of the object itself. If the
> object is not in-memory, the accessor will have to create it and
> populate its slots from the database. You'll need a hash table or
> something to keep track of those objects that are in-memory. The most
> problematic part, I guess, is you need a strategy for removing rarely
> used in-memory objects to make room for new ones. This seems like a
> good use for weak hash tables if your Lisp has them. Or if it has
> weak pointers, you could populate a hash table with weak pointers. If
> it has neither, I don't know what you can do, but maybe someone else
> has ideas.
What about writing my own version of slot-value, make-instance and
defmethod?
Thanks.
+ "Javier" <·······@gmail.com>:
| What about writing my own version of slot-value, make-instance and
| defmethod?
I suppose you could write some :around methods on slot-value. Writing
:after methods on make-instance is commonplace enough. But messing
around with defmethod? Hmm. That would be hackery above my present
level of proficiency, so I'll leave it to others to advice. Good luck.
--
* Harald Hanche-Olsen <URL:http://www.math.ntnu.no/~hanche/>
- It is undesirable to believe a proposition
when there is no ground whatsoever for supposing it is true.
-- Bertrand Russell
(message (Hello 'Harald)
(you :wrote :on '(Sat, 25 Nov 2006 22:08:20 +0100))
(
HHO> weak pointers, you could populate a hash table with weak pointers. If
HHO> it has neither, I don't know what you can do, but maybe someone else
HHO> has ideas.
i think it won't be hard to maintain a LRU list.
i.e. hash table points into a elements of doubly-linked list. if we touch
object, we move it to the head of list.
if we detect that we have too much objects, we trim the tail of list,
deleting both list and entries in ht.
the objects will be reclaimed on the next GC if they are not used in any
other places.
)
(With-best-regards '(Alex Mizrahi) :aka 'killer_storm)
"People who lust for the Feel of keys on their fingertips (c) Inity")
(message (Hello 'Javier)
(you :wrote :on '(Sat, 25 Nov 2006 20:28:44 +0100))
(
J> But there is also a big problem: my objects are interconected (even
J> they are recursively interconected). So my only solution is to store
J> all my data in a database.
you need some kind of object-relational mapping (ORM) in this case
J> Elsewhere, and this is the point, I don't want to rewrite my entire
J> program.
existing solutions will require you to change classes. you can either use
some kind of adapter, or write your own ORM will suit your purposes.
J> So, the question here is: is there any way to work with all my classes
J> using a database (not directly allocating memory), without changing
J> my source code?
as Harald have pointed, you can just change accessors in the simpliest case.
you can even make it transparent if you use MOP -- e.g. you just write
my-db-object as a superclass of you object, and it automatically creates
proper accessors wherever needed, according to slot types, for example.
J> But when the program grows and cross over the physical memory
J> avaiable on my system, the OS starts to swap memory to the hard
J> disk, until it exaust the entire system.
are you sure that it's not suitable? you can address upto ~4 GB data on
32-bit OS, and OS will use physical RAM as a cache for whole data set,
database application will do aprox. the same, so these approaches are not
very different.
with 64-bit processor and OS you can address almost unlimited mem (8 TB, for
example).
i think the only problem in this case will be GC -- Lisp implementations are
not optimized to do GC on on-disk data structures, so basically you'll need
to read whole gigabytes of data to do GC (maybe multiple times). if you can
kill GC, or you can live with long GC, all will be well.
)
(With-best-regards '(Alex Mizrahi) :aka 'killer_storm)
"People who lust for the Feel of keys on their fingertips (c) Inity")
Alex Mizrahi ha escrito:
> (message (Hello 'Javier)
> (you :wrote :on '(Sat, 25 Nov 2006 20:28:44 +0100))
> (
>
> J> But there is also a big problem: my objects are interconected (even
> J> they are recursively interconected). So my only solution is to store
> J> all my data in a database.
>
> you need some kind of object-relational mapping (ORM) in this case
>
> J> Elsewhere, and this is the point, I don't want to rewrite my entire
> J> program.
>
> existing solutions will require you to change classes. you can either use
> some kind of adapter, or write your own ORM will suit your purposes.
>
> J> So, the question here is: is there any way to work with all my classes
> J> using a database (not directly allocating memory), without changing
> J> my source code?
>
> as Harald have pointed, you can just change accessors in the simpliest case.
> you can even make it transparent if you use MOP -- e.g. you just write
> my-db-object as a superclass of you object, and it automatically creates
> proper accessors wherever needed, according to slot types, for example.
>
Thinking about it, I don't really need most of the sophisticated
features of CLOS. Just using defclass, defgeneric, slot-value and
defmethod is enough for my program to run.
It would not be very hard to reimplement them by myself, if I only do
so for those parts I really need.
The hard part would probably be defgeneric and defmethod. Do you think
I could reuse them?
But, do you think it would be better to use MOP? What generic functions
do you think I should implement for "my-db-object" if so?
> J> But when the program grows and cross over the physical memory
> J> avaiable on my system, the OS starts to swap memory to the hard
> J> disk, until it exaust the entire system.
>
> are you sure that it's not suitable?
At this stage, it is. What worries me is what would happen in the
future, when the program scales up to a point in which it needs several
gigabytes of memory to function. I've made some tests, and definitively
trusting on the virtual memory is not the solution: the entire system
slows down and the application itself becomes less responsive and
slower until it finally becomes unusable. I haven't made any test on
how it would became if using database operations instead, but I think
it would not be so bad.
Another issue I have not commented until now: I use the standard
format, read and eval functions to write up all my data to disk and
then restore it back. This is clearly not very fast and I think it
consumes much more memory too. If using some kind of database, I could
eliminate this part.
>you can address upto ~4 GB data on
> 32-bit OS, and OS will use physical RAM as a cache for whole data set,
> database application will do aprox. the same, so these approaches are not
> very different.
> with 64-bit processor and OS you can address almost unlimited mem (8 TB, for
> example).
>
> i think the only problem in this case will be GC -- Lisp implementations are
> not optimized to do GC on on-disk data structures, so basically you'll need
> to read whole gigabytes of data to do GC (maybe multiple times). if you can
> kill GC, or you can live with long GC, all will be well.
>
> )
> (With-best-regards '(Alex Mizrahi) :aka 'killer_storm)
> "People who lust for the Feel of keys on their fingertips (c) Inity")
Thanks.
(message (Hello 'Javier)
(you :wrote :on '(25 Nov 2006 17:50:41 -0800))
(
J> The hard part would probably be defgeneric and defmethod. Do you think
J> I could reuse them?
there's no point to change defgeneric
J> But, do you think it would be better to use MOP?
yes
J> What generic functions do you think I should implement for
J> "my-db-object" if so?
(honestly speaking i've done stuff like that long time ago, so i hope people
who know it better willl advise you better)
you can hook creation of new classes:
(defmethod initialize-instance :around ((class my-metaclass) &rest all-keys)
then you can extract all slot definitions:
(let ((slots (getf all-keys :direct-slots))
and install :around defmethods that will do the database stuff (at this
point i'm not sure, i've emited them with eval, maybe there's a better way.
hm, yes, i remember there was a MOP method that adds method to generic-fun).
i think you can find code like this in persistence libs and do it youself if
those persistence libs do not suit you.
)
(With-best-regards '(Alex Mizrahi) :aka 'killer_storm)
"People who lust for the Feel of keys on their fingertips (c) Inity")
Javier <·······@gmail.com> writes:
> Elsewhere, and this is the point, I don't want to rewrite my entire
> program.
> So, the question here is: is there any way to work with all my classes
> using a database (not directly allocating memory), without changing
> my source code?
Take a look at CL-SQL and see if that allows what you want with only
minimal rewriting of your code.
http://clsql.b9.com/
--
Thomas A. Russ, USC/Information Sciences Institute