Doing web applications in other languages, the conventional approach to
persistence is to bung everything into a relational database. I'm not
convinced that that is the right approach for what I want to do, because
databases are too inflexible; I find that an unreasonable proportion of
my present code base deals with data munging and type coercion, and the
problem of natural data - whether integer or string - over-flowing
limited storage formats is intractable with current generation
databases.
When I was last working with Lisp seriously, 'persistence' was a matter
of making periodic sysouts, and taking a sysout meant stop whatever
you're doing... If you had a serious crash between sysouts, you lost
everything you hadn't explicitly written to file.
So how does one deal with persistence in Lisp these days?
There's an abstract of an article on Lisp persistence here:
<URL:http://www.brics.dk/~hosc/vol10/1-jacobs-swanson.html>
but unfortunately the linked full text PDF doesn't appear to be available
this morning. I've also found a reference manual for a persistent CLOS
at <URL:http://www.hpl.hp.com/techreports/91/HPL-91-182.pdf>.
Are there other papers I should be reading/other software I should be
playing with?
--
·····@jasmine.org.uk (Simon Brooke) http://www.jasmine.org.uk/~simon/
Morning had broken, and there was nothing left for us to do
but pick up the pieces.
> So how does one deal with persistence in Lisp these days?
As already mentioned by Petter if you're looking for a commercial (thus
fully supported) solution, take a look at AllegroCache
(http://www.franz.com/products/allegrocache/).
It is still in its infancy, but it works and looks very promising. It's
definitely the most straightforward persistence solution I used so far.
Sascha
Simon Brooke wrote:
> When I was last working with Lisp seriously, 'persistence' was a matter
> of making periodic sysouts, and taking a sysout meant stop whatever
> you're doing... If you had a serious crash between sysouts, you lost
> everything you hadn't explicitly written to file.
>
> So how does one deal with persistence in Lisp these days?
Arthur Lemmens (mentioned later for the upcoming Rucksack) wrote a good
overview of the persistence libraries last year, for a talk he gave.
http://www.pentaside.org/paper/persistence-lemmens.txt
Now, there's probably a couple more, like BKNR.
Tayssir
I've seen a few references to Rucksack, but haven't been able to find
any details about how it handles persistence. I'm at a point in a
project where I need to make some decisions about how to handle
persistence and would like to know if Rucksack is something that might
fit my needs. Is any information available now?
From: Edi Weitz
Subject: Re: Approaches to persistance and journalling
Date:
Message-ID: <ur75aovz7.fsf@agharta.de>
On 10 Mar 2006 12:53:35 -0800, "Jon" <··········@gmail.com> wrote:
> I've seen a few references to Rucksack, but haven't been able to
> find any details about how it handles persistence. I'm at a point
> in a project where I need to make some decisions about how to handle
> persistence and would like to know if Rucksack is something that
> might fit my needs. Is any information available now?
I think you'll have to either wait for the ECLM or ask Arthur
directly.
Cheers,
Edi.
--
European Common Lisp Meeting 2006: <http://weitz.de/eclm2006/>
Real email: (replace (subseq ·········@agharta.de" 5) "edi")
Simon Brooke <·····@jasmine.org.uk> writes:
> So how does one deal with persistence in Lisp these days?
>
> There's an abstract of an article on Lisp persistence here:
> <URL:http://www.brics.dk/~hosc/vol10/1-jacobs-swanson.html>
> but unfortunately the linked full text PDF doesn't appear to be available
> this morning. I've also found a reference manual for a persistent CLOS
> at <URL:http://www.hpl.hp.com/techreports/91/HPL-91-182.pdf>.
>
> Are there other papers I should be reading/other software I should be
> playing with?
Well, if you have enough RAM to keep your data in, saving a core image
is fast enough you could do it quite often:
For example, for 99 MB
[1]> (defparameter *db* (loop repeat 1000 collect (make-string 100000)))
*DB*
[2]> (time (ext:saveinitmem "/tmp/100M.mem"))
Real time: 23.961992 sec.
Run time: 1.45 sec.
Space: 4968 Bytes
GC: 1, GC time: 0.52 sec.
103527224 ;
25881806
[3]>
-rw-r--r-- 1 pjb pjb 99M 2006-03-07 10:59 /tmp/100M.mem
If you don't want to bear the 23 second freeze, you can proceed as follow:
(defun snapshoot ()
(incf *db-version*)
(if (zerop (linux:fork))
(ext:saveinitmem (format nil "app+db-~A.mem" *db-version*))
(progn
(close-log)
(open-new-log (format nil "db-~A.log" *db-version*)))))
;; Schedule a call to snapshoot every 5 minutes.
;; And if you need to recover between two snapshoots:
(defun replay ()
;; *db-version* is saved in the core image.
(replay-old-log (format nil "db-~A.log" *db-version*))))
So you only need to reload a snapshoot and call replay to put it up to date.
--
__Pascal Bourguignon__ http://www.informatimago.com/
"This statement is false." In Lisp: (defun Q () (eq nil (Q)))
in message <··············@thalassa.informatimago.com>, Pascal
Bourguignon (·······@informatimago.com') wrote:
> Simon Brooke <·····@jasmine.org.uk> writes:
>> So how does one deal with persistence in Lisp these days?
>>
>> There's an abstract of an article on Lisp persistence here:
>> <URL:http://www.brics.dk/~hosc/vol10/1-jacobs-swanson.html>
>> but unfortunately the linked full text PDF doesn't appear to be
>> available this morning. I've also found a reference manual for a
>> persistent CLOS at
>> <URL:http://www.hpl.hp.com/techreports/91/HPL-91-182.pdf>.
>>
>> Are there other papers I should be reading/other software I should be
>> playing with?
>
>
> If you don't want to bear the 23 second freeze, you can proceed as
> follow:
>
> (defun snapshoot ()
> (incf *db-version*)
> (if (zerop (linux:fork))
> (ext:saveinitmem (format nil "app+db-~A.mem" *db-version*))
> (progn
> (close-log)
> (open-new-log (format nil "db-~A.log" *db-version*)))))
>
> ;; Schedule a call to snapshoot every 5 minutes.
> ;; And if you need to recover between two snapshoots:
>
> (defun replay ()
> ;; *db-version* is saved in the core image.
> (replay-old-log (format nil "db-~A.log" *db-version*))))
>
> So you only need to reload a snapshoot and call replay to put it up to
> date.
That sounds pretty much like what I want to do. Presumably (linux:fork)
is a call to the fork system call, and returns the child PID in the
parent environment and 0 in the child environment? Where do I find this
'linux' package? What Lisps support it?
--
·····@jasmine.org.uk (Simon Brooke) http://www.jasmine.org.uk/~simon/
;; how did we conclude that a fucking cartoon mouse is deserving
;; of 90+ years of protection, but a cure for cancer, only 14?
-- user 'Tackhead', in /. discussion of copyright law, 22/05/02
Simon Brooke <·····@jasmine.org.uk> writes:
>> If you don't want to bear the 23 second freeze, you can proceed as
>> follow:
>>
>> (defun snapshoot ()
>> (incf *db-version*)
>> (if (zerop (linux:fork))
>> (ext:saveinitmem (format nil "app+db-~A.mem" *db-version*))
>> (progn
>> (close-log)
>> (open-new-log (format nil "db-~A.log" *db-version*)))))
>>
>> ;; Schedule a call to snapshoot every 5 minutes.
>> ;; And if you need to recover between two snapshoots:
>>
>> (defun replay ()
>> ;; *db-version* is saved in the core image.
>> (replay-old-log (format nil "db-~A.log" *db-version*))))
>>
>> So you only need to reload a snapshoot and call replay to put it up to
>> date.
>
> That sounds pretty much like what I want to do. Presumably (linux:fork)
> is a call to the fork system call, and returns the child PID in the
> parent environment and 0 in the child environment?
Yes.
> Where do I find this 'linux' package? What Lisps support it?
Sorry, I should have inserted a #+clisp.
The LINUX package is found in clisp, but other unix or posix based implementations
have a similar package exporting fork and other unix amenities:
#+clisp (LINUX:|fork|)
#+cmu (UNIX:UNIX-FORK)
#+sbcl (SB-POSIX:FORK)
etc...
The function to save a core image is implementation dependant too.
--
__Pascal Bourguignon__ http://www.informatimago.com/
On Tue, 07 Mar 2006 14:29:52 +0000, Simon Brooke wrote:
>> So you only need to reload a snapshoot and call replay to put it up to
>> date.
>
> That sounds pretty much like what I want to do. Presumably (linux:fork)
> is a call to the fork system call, and returns the child PID in the
> parent environment and 0 in the child environment? Where do I find this
> 'linux' package? What Lisps support it?
That's a good one :-)
More serious: what sort of persistance do you need? Application persitance
a la "saved corefile" (much like persistance in a Smalltalk system) or
object persistance? AFAIK most of the solutions mentioned so far only
provide object persistance (don't actually know how AllegroCache handles
this).
Even a dumped core image will often _not_ do what you want: unlike in
the golden days of Lisp Machines there's usually a world outside the Lisp
image whose state can be rather tricky to capture (database connections,
open file handles, open network sockets etc.).
Cheers, Ralf Mattes
in message <······························@hobbes.mh-freiburg.de>, R.
Mattes (·····@hobbes.mh-freiburg.de') wrote:
> On Tue, 07 Mar 2006 14:29:52 +0000, Simon Brooke wrote:
>
>>> So you only need to reload a snapshoot and call replay to put it up
>>> to date.
>>
>> That sounds pretty much like what I want to do. Presumably
>> (linux:fork) is a call to the fork system call, and returns the child
>> PID in the parent environment and 0 in the child environment? Where do
>> I find this 'linux' package? What Lisps support it?
>
> That's a good one :-)
> More serious: what sort of persistance do you need? Application
> persitance
> a la "saved corefile" (much like persistance in a Smalltalk system) or
> object persistance? AFAIK most of the solutions mentioned so far only
> provide object persistance (don't actually know how AllegroCache
> handles this).
> Even a dumped core image will often _not_ do what you want: unlike in
> the golden days of Lisp Machines there's usually a world outside the
> Lisp image whose state can be rather tricky to capture (database
> connections, open file handles, open network sockets etc.).
I think the core image is what I am trying to capture. The idea is that
the 'database' is held in core, in Lisp structures, and that I/O is
fundamentally via HTTP, so that individual transactions are isolated.
So, so long as I can maintain, by serialising a journal, a log of all
actions which change the state of the core image between core dumps (and
providing that journal doesn't get hosed in the event of e.g. sudden
power loss) I think that periodic sysouts plus journaling is an answer
which will satisfy me. Furthermore, if the journal is at the level of
the text of HTTP POSTs and GETs it can be maintained by an Apache module
rather than by a layer of Lisp function wrappers.
--
·····@jasmine.org.uk (Simon Brooke) http://www.jasmine.org.uk/~simon/
;; Let's have a moment of silence for all those Americans who are stuck
;; in traffic on their way to the gym to ride the stationary bicycle.
;; Rep. Earl Blumenauer (Dem, OR)
"Simon Brooke" <·····@jasmine.org.uk> wrote
> Doing web applications in other languages, the conventional approach to
> persistence is to bung everything into a relational database. I'm not
> convinced that that is the right approach for what I want to do, because
> databases are too inflexible; I find that an unreasonable proportion of
> my present code base deals with data munging and type coercion, and the
> problem of natural data - whether integer or string - over-flowing
> limited storage formats is intractable with current generation
> databases.
>
> When I was last working with Lisp seriously, 'persistence' was a matter
> of making periodic sysouts, and taking a sysout meant stop whatever
> you're doing... If you had a serious crash between sysouts, you lost
> everything you hadn't explicitly written to file.
>
> So how does one deal with persistence in Lisp these days?
>
> There's an abstract of an article on Lisp persistence here:
> <URL:http://www.brics.dk/~hosc/vol10/1-jacobs-swanson.html>
> but unfortunately the linked full text PDF doesn't appear to be available
> this morning. I've also found a reference manual for a persistent CLOS
> at <URL:http://www.hpl.hp.com/techreports/91/HPL-91-182.pdf>.
>
> Are there other papers I should be reading/other software I should be
> playing with?
You should come to the next ECLM (http://weitz.de/eclm2006/ ). Arthur
Lemmens will present Rucksack: a flexible, lightweight, open source
persistence library.
Marc
Simon Brooke <·····@jasmine.org.uk> writes:
> So how does one deal with persistence in Lisp these days?
If you are looking for a commercial product theres AllegroCache:
http://www.franz.com/products/allegrocache/index.lhtml
Petter
--
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
A: Top-posting.
Q: What is the most annoying thing on usenet and in e-mail?
Simon Brooke <·····@jasmine.org.uk> writes:
> Doing web applications in other languages, the conventional approach to
> persistence is to bung everything into a relational database. I'm not
> convinced that that is the right approach for what I want to do, because
> databases are too inflexible; I find that an unreasonable proportion of
> my present code base deals with data munging and type coercion, and the
> problem of natural data - whether integer or string - over-flowing
> limited storage formats is intractable with current generation
> databases.
>
> When I was last working with Lisp seriously, 'persistence' was a matter
> of making periodic sysouts, and taking a sysout meant stop whatever
> you're doing... If you had a serious crash between sysouts, you lost
> everything you hadn't explicitly written to file.
>
> So how does one deal with persistence in Lisp these days?
>
> There's an abstract of an article on Lisp persistence here:
> <URL:http://www.brics.dk/~hosc/vol10/1-jacobs-swanson.html>
> but unfortunately the linked full text PDF doesn't appear to be available
> this morning. I've also found a reference manual for a persistent CLOS
> at <URL:http://www.hpl.hp.com/techreports/91/HPL-91-182.pdf>.
>
> Are there other papers I should be reading/other software I should be
> playing with?
Besides the other hints you might also take a look at
http://www.common-lisp.net/project/cl-store/
From the README:
> 0. About.
> CL-STORE is an portable serialization package which
> should give you the ability to store all common-lisp
> data types (well not all yet) into streams.
Regards,
Marco
Simon Brooke schrieb:
> Doing web applications in other languages, the conventional approach to
> persistence is to bung everything into a relational database. I'm not
> convinced that that is the right approach for what I want to do, because
> databases are too inflexible; I find that an unreasonable proportion of
> my present code base deals with data munging and type coercion, and the
> problem of natural data - whether integer or string - over-flowing
> limited storage formats is intractable with current generation
> databases.
>
> When I was last working with Lisp seriously, 'persistence' was a matter
> of making periodic sysouts, and taking a sysout meant stop whatever
> you're doing... If you had a serious crash between sysouts, you lost
> everything you hadn't explicitly written to file.
I have't tried the following, but it sounds like something:
http://common-lisp.net/project/cl-prevalence/
Those with interest for some real exotic stuff, look at AP5:
http://ap5.com/
AP5 is Lisp extended with transactional data types...
On Wed, 08 Mar 2006 15:17:03 -0800, joswig wrote:
>
> Simon Brooke schrieb:
>
>> Doing web applications in other languages, the conventional approach to
>> persistence is to bung everything into a relational database. I'm not
>> convinced that that is the right approach for what I want to do, because
>> databases are too inflexible; I find that an unreasonable proportion of
>> my present code base deals with data munging and type coercion, and the
>> problem of natural data - whether integer or string - over-flowing
>> limited storage formats is intractable with current generation
>> databases.
>>
>> When I was last working with Lisp seriously, 'persistence' was a matter
>> of making periodic sysouts, and taking a sysout meant stop whatever
>> you're doing... If you had a serious crash between sysouts, you lost
>> everything you hadn't explicitly written to file.
>
> I have't tried the following, but it sounds like something:
> http://common-lisp.net/project/cl-prevalence/
>
> Those with interest for some real exotic stuff, look at AP5:
> http://ap5.com/
> AP5 is Lisp extended with transactional data types...
That looks like an interesting project. Tanks for the link. Do you
have any idea about the license for that code? I couldn't find
anything on the web page.
Cheers, Ralf Mattes
On 2006-03-10 09:11:18 -0500, "R. Mattes" <····@hobbes.mh-freiburg.de> said:
> That looks like an interesting project. Tanks for the link. Do you
> have any idea about the license for that code? I couldn't find
> anything on the web page.
Not Rainer here, but it's LLGPL. This is from the .asd file:
;;;; The CL-PREVALENCE ASDF system definition
;;;;
;;;; Copyright (C) 2003, 2004 Sven Van Caekenberghe, Beta Nine BVBA.
;;;;
;;;; You are granted the rights to distribute and use this software
;;;; as governed by the terms of the Lisp Lesser General Public License
;;;; (http://opensource.franz.com/preamble.html), also known as the LLGPL.
Simon Brooke <·····@jasmine.org.uk> writes:
[...]
> So how does one deal with persistence in Lisp these days?
[...]
No one mentioned Elephant: http://common-lisp.net/project/elephant/
With a Sleepycat/Berkeley DB back-end it might do what you want. I
haven't used it myself, but it looks promising (and it's an active
project).
--
John Landahl <····@landahl.org>
http://landahl.org/john