Are there any tips or techniques out there for storing and accessing large
quantities of data (presumably from the hard disk) under Lisp? I've read
or had indications from several scattered articles that relational
databases aren't necessarily always the best way to go (the reasons for
this conclusion still remain relatively unclear to me).
To put forth a more concrete example, assume there are something on the
order of 10 million data objects that need to be stored and indexed - what
would be one of the most optimal ways of doing so from a Common Lisp
implementation?
Cory Spencer <········@interchange.ubc.ca> writes:
> Are there any tips or techniques out there for storing and accessing large
> quantities of data (presumably from the hard disk) under Lisp? I've read
> or had indications from several scattered articles that relational
> databases aren't necessarily always the best way to go (the reasons for
> this conclusion still remain relatively unclear to me).
Relational databases determine identity by the contents of a
record. Also, they don't have pointers; references to other records
must be done via a "serial number" field. However, implementations of
relational databases are very mature.
> To put forth a more concrete example, assume there are something on the
> order of 10 million data objects that need to be stored and indexed - what
> would be one of the most optimal ways of doing so from a Common Lisp
> implementation?
If the data fits the relational model, I strongly suggest using that,
since the interface to that (SQL) is standardized, and a number of
excellent, portable connectors between CL and RDBMSes exist (including
(Un)CommonSQL for CLOS-to-RDBMS).
Object databases are much newer, and I don't think there are any free,
ported ones available. The only free one I've seen is for MCL. Franz
offers AllegroStore, but that seems to be based on ref-counting, not
GC (it seems to use ObjectStore, which is designed for C or C++
applications).
--
-> -/ - Rahul Jain - \- <-
-> -\ http://linux.rice.edu/~rahul -=- ············@techie.com /- <-
-> -/ "Structure is nothing if it is all you got. Skeletons spook \- <-
-> -\ people if [they] try to walk around on their own. I really /- <-
-> -/ wonder why XML does not." -- Erik Naggum, comp.lang.lisp \- <-
|--|--------|--------------|----|-------------|------|---------|-----|-|
(c)1996-2002, All rights reserved. Disclaimer available upon request.
From: Arthur Lemmens
Subject: Re: Storing/Accessing large amounts of data
Date:
Message-ID: <3C9AF500.ADA63786@xs4all.nl>
Rahul Jain wrote:
> since the interface to that (SQL) is standardized, and a number of
> excellent, portable connectors between CL and RDBMSes exist (including
> (Un)CommonSQL for CLOS-to-RDBMS).
I've never heard about CLOS-to-RDBMS. Do you have a reference?
Thanks.
From: Arthur Lemmens
Subject: Re: Storing/Accessing large amounts of data
Date:
Message-ID: <3C9B14AF.3B7D823F@xs4all.nl>
I wrote:
> Rahul Jain wrote:
>
> > since the interface to that (SQL) is standardized, and a number of
> > excellent, portable connectors between CL and RDBMSes exist (including
> > (Un)CommonSQL for CLOS-to-RDBMS).
>
> I've never heard about CLOS-to-RDBMS. Do you have a reference?
On re-reading this, I see that I didn't read you correctly the first time.
I thought I read "CommonSQL or CLOS-to-RDBMS" but you wrote "for", not "or".
Sorry about the confusion.
Arthur Lemmens <········@xs4all.nl> writes:
> Rahul Jain wrote:
> > since the interface to that (SQL) is standardized, and a number of
> > excellent, portable connectors between CL and RDBMSes exist (including
> > (Un)CommonSQL for CLOS-to-RDBMS).
^^^ that's not a typo :)
> I've never heard about CLOS-to-RDBMS. Do you have a reference?
> Thanks.
I'm talking about the DEF-VIEW-CLASS macro, which is documented in the
Lispworks User Guide (?) in the CommonSQL section. It allows you to
define CLOS classes whose data comes from the RDBMS.
--
-> -/ - Rahul Jain - \- <-
-> -\ http://linux.rice.edu/~rahul -=- ············@techie.com /- <-
-> -/ "Structure is nothing if it is all you got. Skeletons spook \- <-
-> -\ people if [they] try to walk around on their own. I really /- <-
-> -/ wonder why XML does not." -- Erik Naggum, comp.lang.lisp \- <-
|--|--------|--------------|----|-------------|------|---------|-----|-|
(c)1996-2002, All rights reserved. Disclaimer available upon request.
10 million data elements may be a `modest' amount
(depending on the size of each).
At ITA Software they deal with hundreds of millions
of data elements. They just use a huge amount of RAM.
But the solution is going to be highly dependent on
what sort of data you have and how you intend to
manipulate it.
"Cory Spencer" <········@interchange.ubc.ca> wrote in message
·················@nntp.itservices.ubc.ca...
> Are there any tips or techniques out there for storing and accessing large
> quantities of data (presumably from the hard disk) under Lisp? I've read
> or had indications from several scattered articles that relational
> databases aren't necessarily always the best way to go (the reasons for
> this conclusion still remain relatively unclear to me).
>
> To put forth a more concrete example, assume there are something on the
> order of 10 million data objects that need to be stored and indexed - what
> would be one of the most optimal ways of doing so from a Common Lisp
> implementation?
>