On Dec 4, 12:56 pm, gavino <·········@gmail.com> wrote:
> paul graham said he did not use a relational db for viaweb
>
> can one do webapps with lisp and avoid use of a relational db?
Serious relational databases (not all are) go to very considerable
lengths to ensure data integrity, ACID properties, allow online
backup, replicate transactions to your DR site, scale to large systems
or to clusters of small (or large systems). If you don't need those
properties or are willing to implement the subset you do need (which,
for some or most of them, is pretty hard), then you probably don't
need the expense or overhead of a relational database.
From: Daniel Weinreb
Subject: Re: are relational databases not needed when you use lisp? paul graham did not
Date:
Message-ID: <AUn5j.11439$Lg.7307@trndny09>
Tim Bradshaw wrote:
> On Dec 4, 12:56 pm, gavino <·········@gmail.com> wrote:
>> paul graham said he did not use a relational db for viaweb
>>
>> can one do webapps with lisp and avoid use of a relational db?
>
> Serious relational databases (not all are) go to very considerable
> lengths to ensure data integrity, ACID properties, allow online
> backup, replicate transactions to your DR site, scale to large systems
> or to clusters of small (or large systems). If you don't need those
> properties or are willing to implement the subset you do need (which,
> for some or most of them, is pretty hard), then you probably don't
> need the expense or overhead of a relational database.
Yup, those are the issues. I wrote one of the first, perhaps
the first, object-oriented database for Lisp (Statice at
Symbolics), and spent the next fourteen years at a startup
I co-founded (Object Design) doing object-oriented database
systems. We never did a Lisp version -- not enough market
for it. We did C++, and then when Java came along, Java.
We did data integrity and ACID properties just fine.
Online backup was not so good, replication to the DR site
was not provided. The scaling issue is sort of hard to
explain (i.e. it depends what you mean). We did not
have what Oracle calls "Oracle RAC", the redundant
highly-available architecture. We did not have
smarts on the server side, no stored procedures, no
views. Oracle has third-party vendors with some
really spiffy tools; we were too small to attract
such vendors. So it's all a complicated question.
The issues for Oracle are different than the
issues for other relational systems like MySQL,
of course. (Yes, I know that these systems are not
literally "relational" in all theoretical aspects.)
The ITA reservation system is using Oracle RAC. We
have our own Lisp object-to-relational mapping
subsystem. It's not easy to make one, particularly
if you want both flexibility, long-term caching,
and other high-performance features. It takes a lot
of performance work to get it fast.
Had we written in Java, it's likely that we would have
used Hibernate, and had less work to do ourselves.
I don't currently know of an existing available
open-source object-to-relational system for Common
Lisp, at least not one that would meet our needs.
On Dec 4, 6:19 pm, Daniel Weinreb <····@alum.mit.edu> wrote:
> Tim Bradshaw wrote:
> > On Dec 4, 12:56 pm, gavino <·········@gmail.com> wrote:
> >> paul graham said he did not use a relational db for viaweb
>
> >> can one do webapps with lisp and avoid use of a relational db?
>
> > Serious relational databases (not all are) go to very considerable
> > lengths to ensure data integrity, ACID properties, allow online
> > backup, replicate transactions to your DR site, scale to large systems
> > or to clusters of small (or large systems). If you don't need those
> > properties or are willing to implement the subset you do need (which,
> > for some or most of them, is pretty hard), then you probably don't
> > need the expense or overhead of a relational database.
>
> Yup, those are the issues. I wrote one of the first, perhaps
> the first, object-oriented database for Lisp (Statice at
> Symbolics), and spent the next fourteen years at a startup
> I co-founded (Object Design) doing object-oriented database
> systems. We never did a Lisp version -- not enough market
> for it. We did C++, and then when Java came along, Java.
> We did data integrity and ACID properties just fine.
> Online backup was not so good, replication to the DR site
> was not provided. The scaling issue is sort of hard to
> explain (i.e. it depends what you mean). We did not
> have what Oracle calls "Oracle RAC", the redundant
> highly-available architecture. We did not have
> smarts on the server side, no stored procedures, no
> views. Oracle has third-party vendors with some
> really spiffy tools; we were too small to attract
> such vendors. So it's all a complicated question.
> The issues for Oracle are different than the
> issues for other relational systems like MySQL,
> of course. (Yes, I know that these systems are not
> literally "relational" in all theoretical aspects.)
>
> The ITA reservation system is using Oracle RAC. We
> have our own Lisp object-to-relational mapping
> subsystem. It's not easy to make one, particularly
> if you want both flexibility, long-term caching,
> and other high-performance features. It takes a lot
> of performance work to get it fast.
>
> Had we written in Java, it's likely that we would have
> used Hibernate, and had less work to do ourselves.
> I don't currently know of an existing available
> open-source object-to-relational system for Common
> Lisp, at least not one that would meet our needs.
at what pont does the power of the programming language become not
important if you are using a relational db? I read some great stuff
by P Greenspun about how tcl combines with aolserver and postgresql
are a killer combination.....Does this mean that the programming
language is not where the gain are to be made?
On Dec 5, 2:19 am, Daniel Weinreb <····@alum.mit.edu> wrote:
> The scaling issue is sort of hard to
> explain (i.e. it depends what you mean).
I think I specifically meant scaling well to large shared-memory
systems. I don't know how well Oracle &co do on very large systems -
where I am now I know we use it on systems with 30-100 cores and
perhaps larger - I don't know how big our largest domains are. This
means not being able to use the number of processors well, but
nowadays also understanding the memory characteristics well I think.
> We did not
> have what Oracle calls "Oracle RAC", the redundant
> highly-available architecture.
Also this.
--tim
From: Daniel Weinreb
Subject: Re: are relational databases not needed when you use lisp? paul graham did not
Date:
Message-ID: <Fmw5j.2331$UG1.1923@trnddc01>
Tim Bradshaw wrote:
> On Dec 5, 2:19 am, Daniel Weinreb <····@alum.mit.edu> wrote:
>> The scaling issue is sort of hard to
>> explain (i.e. it depends what you mean).
>
> I think I specifically meant scaling well to large shared-memory
> systems. I don't know how well Oracle &co do on very large systems -
> where I am now I know we use it on systems with 30-100 cores and
> perhaps larger - I don't know how big our largest domains are. This
> means not being able to use the number of processors well, but
> nowadays also understanding the memory characteristics well I think.
An ObjectStore server is multi-threaded and so presumably
can handle multiple cores, although I don't know how much
load they have been subjected to and whether internal
lock contention starts to come into play. You'd have to
ask the people who are there now (www.objectstore.com).
As for size, you can make pretty big ObjectStore databases.
And then you can make a lot of them and have pointers
between them, and those can be on different servers.
Again, you'd have to ask them for details of how it
works currently; I've been out of that game for years now.
>
>> We did not
>> have what Oracle calls "Oracle RAC", the redundant
>> highly-available architecture.
>
> Also this.
>
> --tim
From: Ken Tilton
Subject: Re: are relational databases not needed when you use lisp? paul graham did not
Date:
Message-ID: <PCq5j.1341$g26.511@newsfe08.lga>
Daniel Weinreb wrote:
> Tim Bradshaw wrote:
>
>> On Dec 4, 12:56 pm, gavino <·········@gmail.com> wrote:
>>
>>> paul graham said he did not use a relational db for viaweb
>>>
>>> can one do webapps with lisp and avoid use of a relational db?
>>
>>
>> Serious relational databases (not all are) go to very considerable
>> lengths to ensure data integrity, ACID properties, allow online
>> backup, replicate transactions to your DR site, scale to large systems
>> or to clusters of small (or large systems). If you don't need those
>> properties or are willing to implement the subset you do need (which,
>> for some or most of them, is pretty hard), then you probably don't
>> need the expense or overhead of a relational database.
>
>
> Yup, those are the issues. I wrote one of the first, perhaps
> the first, object-oriented database for Lisp (Statice at
> Symbolics), and spent the next fourteen years at a startup
> I co-founded (Object Design) doing object-oriented database
> systems. We never did a Lisp version -- not enough market
> for it.
Ah, so that is how AllegroStore ended up atop ObjectStore. A closet
Lispnik at ODI! :)
> We did C++, and then when Java came along, Java.
> We did data integrity and ACID properties just fine.
> Online backup was not so good, replication to the DR site
> was not provided. The scaling issue is sort of hard to
> explain (i.e. it depends what you mean). We did not
> have what Oracle calls "Oracle RAC", the redundant
> highly-available architecture. We did not have
> smarts on the server side, no stored procedures, no
> views. Oracle has third-party vendors with some
> really spiffy tools; we were too small to attract
> such vendors. So it's all a complicated question.
> The issues for Oracle are different than the
> issues for other relational systems like MySQL,
> of course. (Yes, I know that these systems are not
> literally "relational" in all theoretical aspects.)
>
> The ITA reservation system is using Oracle RAC. We
> have our own Lisp object-to-relational mapping
> subsystem. It's not easy to make one, particularly
> if you want both flexibility, long-term caching,
> and other high-performance features. It takes a lot
> of performance work to get it fast.
>
> Had we written in Java, it's likely that we would have
> used Hibernate, and had less work to do ourselves.
> I don't currently know of an existing available
> open-source object-to-relational system for Common
> Lisp, at least not one that would meet our needs.
I find myself fascinated by RDF triple stores, and have my eye on
Redland as an open persistent storage solution for Lisp apps. yeah, to
hell with persistent CLOS, gavino talked me out of OO. :)
kzo
--
http://www.theoryyalgebra.com/
"In the morning, hear the Way;
in the evening, die content!"
-- Confucius
From: Daniel Weinreb
Subject: Re: are relational databases not needed when you use lisp? paul graham did not
Date:
Message-ID: <ekw5j.2330$UG1.1175@trnddc01>
Ken Tilton wrote:
>
>
>
> Ah, so that is how AllegroStore ended up atop ObjectStore. A closet
> Lispnik at ODI! :)
Actually, no, I didn't have anything to do with it. In fact,
as far as I know, Franz went ahead and invented and implemented
it without particularly interacting with anyone at ODI, although
maybe they talked to some of my co-workers and I didn't hear about it.
From: Duane Rettig
Subject: Re: are relational databases not needed when you use lisp? paul graham did not
Date:
Message-ID: <o07ijtru8o.fsf@gemini.franz.com>
Daniel Weinreb <···@alum.mit.edu> writes:
> Ken Tilton wrote:
>> Ah, so that is how AllegroStore ended up atop ObjectStore. A closet
>> Lispnik at ODI! :)
>
> Actually, no, I didn't have anything to do with it. In fact,
> as far as I know, Franz went ahead and invented and implemented
> it without particularly interacting with anyone at ODI, although
> maybe they talked to some of my co-workers and I didn't hear about it.
We did negotiate with ODI for cooperation in handling signals. Both
OS and Allegro CL uses SIGSEGV, as it happens, and since signals are a
global resource, it makes for poor multiple-usage. ODI didn't want to
give away their secrets, but we knew that instantiation of objects was
done by explicitly forcing a SEGV and handling it, so we negotiated
with them an interface whereby they would write their SEGV handler to
save off any current handlers and invoke them if/when it determined
that the SEGV wasn't an object instantiation. Thus, OS could filter
out the SEGVs it knew about, and Allegro CL could handle the rest.
--
Duane Rettig ·····@franz.com Franz Inc. http://www.franz.com/
555 12th St., Suite 1450 http://www.555citycenter.com/
Oakland, Ca. 94607 Phone: (510) 452-2000; Fax: (510) 452-0182
From: Maciej Katafiasz
Subject: Re: are relational databases not needed when you use lisp? paul graham did not
Date:
Message-ID: <fj6t0u$24t$4@news.net.uni-c.dk>
Den Wed, 05 Dec 2007 08:17:43 -0800 skrev Duane Rettig:
> [...] but we knew that instantiation of objects was done by explicitly
> forcing a SEGV and handling it [...]
Now, it's hard to reach an informed conclusion from such a short passage,
but instantiation through SEGV sounds so outlandish that I just have to
ask for details, if there are any you can share without breaching any
contracts.
Cheers,
Maciej
From: Duane Rettig
Subject: Re: are relational databases not needed when you use lisp? paul graham did not
Date:
Message-ID: <o0wsrs214c.fsf@gemini.franz.com>
Maciej Katafiasz <········@gmail.com> writes:
> Den Wed, 05 Dec 2007 08:17:43 -0800 skrev Duane Rettig:
>
>> [...] but we knew that instantiation of objects was done by explicitly
>> forcing a SEGV and handling it [...]
>
> Now, it's hard to reach an informed conclusion from such a short passage,
> but instantiation through SEGV sounds so outlandish that I just have to
> ask for details, if there are any you can share without breaching any
> contracts.
Why outlandish?
I can't share facts with you because I don't know many, other than
what my fellow developer told me. I can only guess, though, at both
reasons for ODI's design and also reasons for your horror. I assume
that you can address the latter, and perhaps Daniel Weinreb would
consider sharing some of the former. But to take some guesses, I
presume that you are either reacting against the very thought of using
traditionally error-based programming to handle normal programming
situations, or against the long cycle times that would be present in
handling user level traps in a unix environment. Recall, though, that
not too many years ago 2 Mb was considered to be a huge program, and a
file-based database application that could avoid structural program
change from the first instantiation of an object to the second was
likely to be very efficient for those objects already in memory.
As for the timing issue, when you are dealing with a single hit of
several hundred to a thousand or two cycles for a signal handler to be
dispatched, that cycle count gets swamped by the time it takes to go
out to the disk to get the data, especially if it is randomly
addressed.
As for the possibility that your reaction is to the very thought of
using signals to effect programming - if so, I've seen this kind of
reaction before - my theory is that such aversion is based on the
tendency of error and exception systems to be afterthoughts, and thus
either not complete or not well-tested. I've even had a conversation
with a person in comp.arch who insisted that I could never get
recoverable semantics for stack-overflows. And yet, on operating
systems that support soft overflows, we are indeed able to fully
recover to the point where programmers can continue and complete their
projects (rather than having to consider the environment to be corrupt
after a stack overflow).
It is likely that your guess as to how things were set up in OS are
correct - it seemed like a straightforward situation, at least from my
persepective of supporting the fellow-developer's low-level
requirements when he did his build of AllegroStore. Perhaps you can
explain why your reaction is so strong...
--
Duane Rettig ·····@franz.com Franz Inc. http://www.franz.com/
555 12th St., Suite 1450 http://www.555citycenter.com/
Oakland, Ca. 94607 Phone: (510) 452-2000; Fax: (510) 452-0182
From: Maciej Katafiasz
Subject: Re: are relational databases not needed when you use lisp? paul graham did not
Date:
Message-ID: <fj8f0o$8h2$1@news.net.uni-c.dk>
Den Wed, 05 Dec 2007 15:05:55 -0800 skrev Duane Rettig:
>> Now, it's hard to reach an informed conclusion from such a short
>> passage, but instantiation through SEGV sounds so outlandish that I
>> just have to ask for details, if there are any you can share without
>> breaching any contracts.
>
> Why outlandish?
Perhaps outlandish was a wrong word, but it certainly seemed like an
unusual take on object instantiation. And my reaction was a combination
of several factors, see below.
> I can't share facts with you because I don't know many, other than what
> my fellow developer told me. I can only guess, though, at both reasons
> for ODI's design and also reasons for your horror. I assume that you
> can address the latter, and perhaps Daniel Weinreb would consider
> sharing some of the former. But to take some guesses, I presume that
> you are either reacting against the very thought of using traditionally
> error-based programming to handle normal programming situations, or
> against the long cycle times that would be present in handling user
> level traps in a unix environment. Recall, though, that not too many
> years ago 2 Mb was considered to be a huge program, and a file-based
> database application that could avoid structural program change from the
> first instantiation of an object to the second was likely to be very
> efficient for those objects already in memory.
I guess I was thinking about a different meaning of "object
instantiation". Page fault-like handling is indeed less surprising.
However, my main objection was that signals are so brittle and offer so
little in the way of a protocol (there's no way to attach any data to a
signal, which makes them inherently a global resource, which is very
yucky). I appreciate the ability to present a uniform interface to the
outside code, I'm unsure, however, how signals help compared to ordinary
function calls. Unless you posses the ability to do true page fault
handling (ie. can rewrite memory references), it seems to me like just a
very indirect way of doing funcalls.
> As for the timing issue,
> when you are dealing with a single hit of several hundred to a thousand
> or two cycles for a signal handler to be dispatched, that cycle count
> gets swamped by the time it takes to go out to the disk to get the data,
> especially if it is randomly addressed.
Good point.
> As for the possibility that your reaction is to the very thought of
> using signals to effect programming - if so, I've seen this kind of
> reaction before - my theory is that such aversion is based on the
> tendency of error and exception systems to be afterthoughts, and thus
> either not complete or not well-tested.
I agree that robust error handling can substantially change the notion
of what is a recoverable situation, however, Unix signals aren't exactly
a paragon of expressible, well-integrated error handling mechanism
design :).
Cheers,
Maciej
From: Duane Rettig
Subject: Re: are relational databases not needed when you use lisp? paul graham did not
Date:
Message-ID: <o04pevepu6.fsf@gemini.franz.com>
Maciej Katafiasz <········@gmail.com> writes:
> Den Wed, 05 Dec 2007 15:05:55 -0800 skrev Duane Rettig:
>
>>> Now, it's hard to reach an informed conclusion from such a short
>>> passage, but instantiation through SEGV sounds so outlandish that I
>>> just have to ask for details, if there are any you can share without
>>> breaching any contracts.
>>
>> Why outlandish?
>
> Perhaps outlandish was a wrong word, but it certainly seemed like an
> unusual take on object instantiation. And my reaction was a combination
> of several factors, see below.
> I guess I was thinking about a different meaning of "object
> instantiation". Page fault-like handling is indeed less surprising.
> However, my main objection was that signals are so brittle and offer so
> little in the way of a protocol
Agreed so far
> (there's no way to attach any data to a signal,
Ah, but that's the beauty of thinking inside the box [most people
think that thinkiong outside the box is the epitmy of creativity, but
consciously thinking of creative solutions with a tight set of
constraints is one of the most rewarding of programming experiences].
You'd be surprised what kind of data you can attach to a signal. The
handler itself gives you a whole context, which includes the state of
the registers at the time of the signal. And one of these registers
is the program counter, which, if you are willing to do a bit of
instruction decoding, gives you a wealth of information about what the
code was trying to do at the time. We use this extensively in Allegro
CL, for things like taking the car of a non-listp object (this is
especially beautiful on a machine which traps on mis-aligned accesses -
the CAR and CDR operations are exactly one instruction long, and yet
the handler is able to figure out what the object was that you were
trying to access).
which makes them inherently a global resource,
Also agreed,
> which is very yucky).
This I disagree with, on two counts:
1. It is only disgusting/repugnant/disagreeable if you are not
willing to work within the constraints that are set by their
operation.
2. Any yuckiness should not stem from their being global resources.
How do you deal with the possibility of multiple definitions of the
same symbol in CL? You inject rules: naming conventions, protocols
for introducing packages, rules about the usage of public packages
(and most implementations provide package locks to support such
rules). Have you ever described the fact that you could overwrite
your function definition as "yucky"? I doubt it.
I appreciate the ability to present a uniform interface to the
> outside code, I'm unsure, however, how signals help compared to ordinary
> function calls.
As is the case when deciding whether to use function calls or macros,
using signals is a choice that must be made carefully and with
purpose.
> Unless you posses the ability to do true page fault
> handling (ie. can rewrite memory references), it seems to me like just a
> very indirect way of doing funcalls.
Is this a Turing argument? Of course it can be done differently; the
main point of difference is in the efficiency of the design along the
axes of optimization you consider important.
>> As for the possibility that your reaction is to the very thought of
>> using signals to effect programming - if so, I've seen this kind of
>> reaction before - my theory is that such aversion is based on the
>> tendency of error and exception systems to be afterthoughts, and thus
>> either not complete or not well-tested.
>
> I agree that robust error handling can substantially change the notion
> of what is a recoverable situation, however, Unix signals aren't exactly
> a paragon of expressible, well-integrated error handling mechanism
> design :).
Agreed here, also. Our biggest problems with signals is that they are
all different. They even manage sometimes to change in subtle ways
from one operating system version to the next. However, similar
things could be said about hardware or operating systems themselves.
I have no problem dealing with them on this basis, when categorizing
signals as a part of the uniqueness that comes with a particular
architecture/operating-system pair.
--
Duane Rettig ·····@franz.com Franz Inc. http://www.franz.com/
555 12th St., Suite 1450 http://www.555citycenter.com/
Oakland, Ca. 94607 Phone: (510) 452-2000; Fax: (510) 452-0182
From: Maciej Katafiasz
Subject: Re: are relational databases not needed when you use lisp? paul graham did not
Date:
Message-ID: <fj9i31$fi8$1@news.net.uni-c.dk>
Den Thu, 06 Dec 2007 08:43:29 -0800 skrev Duane Rettig:
>> (there's no way to attach any data to a signal,
>
> Ah, but that's the beauty of thinking inside the box [most people think
> that thinkiong outside the box is the epitmy of creativity, but
> consciously thinking of creative solutions with a tight set of
> constraints is one of the most rewarding of programming experiences].
> You'd be surprised what kind of data you can attach to a signal. The
> handler itself gives you a whole context, which includes the state of
> the registers at the time of the signal. And one of these registers is
> the program counter, which, if you are willing to do a bit of
> instruction decoding, gives you a wealth of information about what the
> code was trying to do at the time. We use this extensively in Allegro
> CL, for things like taking the car of a non-listp object (this is
> especially beautiful on a machine which traps on mis-aligned accesses -
> the CAR and CDR operations are exactly one instruction long, and yet the
> handler is able to figure out what the object was that you were trying
> to access).
Ah, interesting. I never really tried pushing the boundaries of what you
can do with signals, so I wasn't aware you can localise the signalling
site so precisely. This is indeed a very nifty hack.
> which makes them inherently a global resource,
>
> Also agreed,
>
>> which is very yucky).
>
> This I disagree with, on two counts:
>
> 1. It is only disgusting/repugnant/disagreeable if you are not
> willing to work within the constraints that are set by their operation.
>
> 2. Any yuckiness should not stem from their being global resources.
> How do you deal with the possibility of multiple definitions of the same
> symbol in CL? You inject rules: naming conventions, protocols for
> introducing packages, rules about the usage of public packages (and most
> implementations provide package locks to support such rules). Have you
> ever described the fact that you could overwrite your function
> definition as "yucky"? I doubt it.
It's only yucky when you aren't able to effectively locate the
originating site, which, as you explain, is perfectly doable, and thus
reverts signal handling back to non-yucky granularity.
> I appreciate the ability to present a uniform interface to the
>> outside code, I'm unsure, however, how signals help compared to
>> ordinary function calls.
>
> As is the case when deciding whether to use function calls or macros,
> using signals is a choice that must be made carefully and with purpose.
Yes. But the real (ie. the one I had in mind when asking this) answer to
this question you already gave above :). I'll skip the rest of no-longer-
valid points.
>> I agree that robust error handling can substantially change the notion
>> of what is a recoverable situation, however, Unix signals aren't
>> exactly a paragon of expressible, well-integrated error handling
>> mechanism design :).
Of course, I meant *expressive* here.
> Agreed here, also. Our biggest problems with signals is that they are
> all different. They even manage sometimes to change in subtle ways from
> one operating system version to the next. However, similar things could
> be said about hardware or operating systems themselves. I have no
> problem dealing with them on this basis, when categorizing signals as a
> part of the uniqueness that comes with a particular
> architecture/operating-system pair.
That is also one of the reasons I don't jump to signals as the first
solution of a problem, they have almost no defined semantics. About the
only stable thing you can say about them is that they exist, and that you
can't handle KILL and STOP/CONT. Everything else, from their meaning, to
semantics of signalling yourself, to semantics of what happens in multi-
threaded scenarios, is pretty much a big cloud of unspecifiedness.
Cheers,
Maciej
From: Daniel Weinreb
Subject: Re: are relational databases not needed when you use lisp? paul graham did not
Date:
Message-ID: <eF16j.9731$6k1.6792@trndny02>
Maciej Katafiasz wrote:
> That is also one of the reasons I don't jump to signals as the first
> solution of a problem, they have almost no defined semantics. About the
> only stable thing you can say about them is that they exist, and that you
> can't handle KILL and STOP/CONT. Everything else, from their meaning, to
> semantics of signalling yourself, to semantics of what happens in multi-
> threaded scenarios, is pretty much a big cloud of unspecifiedness.
Not SIGSEGV. A confusing things about Unix/Posix/Linux is that
there are two completely different things, both modelled as
"signals". There are asynchronous ones (like killing a process)
and there are synchronous ones like SIGSEGV. The semantics of
SIGSEGV is 100% completely clear.
Daniel Weinreb <···@alum.mit.edu> wrote:
+---------------
| Maciej Katafiasz wrote:
| > That is also one of the reasons I don't jump to signals as the first
| > solution of a problem, they have almost no defined semantics. About the
| > only stable thing you can say about them is that they exist, and that you
| > can't handle KILL and STOP/CONT. Everything else, from their meaning, to
| > semantics of signalling yourself, to semantics of what happens in multi-
| > threaded scenarios, is pretty much a big cloud of unspecifiedness.
|
| Not SIGSEGV. A confusing things about Unix/Posix/Linux is that
| there are two completely different things, both modelled as
| "signals". There are asynchronous ones (like killing a process)
| and there are synchronous ones like SIGSEGV. The semantics of
| SIGSEGV is 100% completely clear.
+---------------
And indeed, a number of generational garbage collectors[1] use the
VM system itself to implement a hardware write barrier to enable the
noting/recording of the "remembered sets", that is, which objects
in older generations point to objects in newer generations, and thus
which older generation pointers must be added to the root set when
collecting newer generations. There are several variants on this
theme[2], and there are arguments about whether this or that workload
would benefit from using a software write barrier [perhaps plus
"card marking"] instead[3], but the "paging hardware write barrier"
approach is certainly reliable [where it can be used at all].
-Rob
[1] CMUCL's (on x86) is one, but there are many others.
[2] In all of them, at the completion of a GC all of the write
enable bits are turned off (or write protection bits are
turned on) on all of the pages in the GC-managed heap
[*except* for the nursery, of course!]. In some versions,
for any store into a write-protected heap page the signal
handler [usually part of the GC] will simply turn write-enable
back on for that page and return from the signal, which will
automatically retry the store, which will now quietly succeed.
At the next GC, any heap pages in generations older that the
one(s) being collection which have their write enable bits on
have thus been stored into since the last GC, and all of the
objects on all such pages are scanned to see if they need to
be added to the remembered set. Overall, this approach can be
quite cheap if there are *lots* of stores but to relatively few
older generation pages.
Another approach is, when a write-protection SIGSEGV occurs, to
precisely locate the object being stored into and record either
the whole object (if it's small) or *only* the slot being written
into (if the object's really large) in the remembered set, then
(a) temporarily turn on write-enable on the page, (b) emulate
[that is, perform] the store in the signal handler, (c) "push
forward" the interrupted PC past the store, (d) turn write-enable
back off on that page, and (e) return from the signal handler
to the new PC just past the store. Overall, this approach may
be cheaper than that previous one if there are relatively few
total stores but to widely-scattered older generation pages.
[3] This is particularly true if (a) the cost of a write-protection
trap and the associated SIGSEGV is *very* high on that platform;
(b) type-propagation can be used to suppress performing the write
barrier when non-pointer data is being stored; (c) pointer stores
are fairly infrequent, and (d) pointer stores tend to be clustered
around regions considerably smaller than a page size [in which
case that region/cluster size is probably a good choice for the
"card" size in a card-marking scheme].
-----
Rob Warnock <····@rpw3.org>
627 26th Avenue <URL:http://rpw3.org/>
San Mateo, CA 94403 (650)572-2607
From: Daniel Weinreb
Subject: Re: are relational databases not needed when you use lisp? paul graham did not
Date:
Message-ID: <Uec6j.13960$xB.13166@trndny06>
Rob Warnock wrote:
> Daniel Weinreb <···@alum.mit.edu> wrote:
> +---------------
> | Maciej Katafiasz wrote:
> | > That is also one of the reasons I don't jump to signals as the first
> | > solution of a problem, they have almost no defined semantics. About the
> | > only stable thing you can say about them is that they exist, and that you
> | > can't handle KILL and STOP/CONT. Everything else, from their meaning, to
> | > semantics of signalling yourself, to semantics of what happens in multi-
> | > threaded scenarios, is pretty much a big cloud of unspecifiedness.
> |
> | Not SIGSEGV. A confusing things about Unix/Posix/Linux is that
> | there are two completely different things, both modelled as
> | "signals". There are asynchronous ones (like killing a process)
> | and there are synchronous ones like SIGSEGV. The semantics of
> | SIGSEGV is 100% completely clear.
> +---------------
>
> And indeed, a number of generational garbage collectors[1] use the
> VM system itself to implement a hardware write barrier to enable the
> noting/recording of the "remembered sets", that is, which objects
> in older generations point to objects in newer generations, and thus
> which older generation pointers must be added to the root set when
> collecting newer generations. There are several variants on this
> theme[2], and there are arguments about whether this or that workload
> would benefit from using a software write barrier [perhaps plus
> "card marking"] instead[3], but the "paging hardware write barrier"
> approach is certainly reliable [where it can be used at all].
>
>
> -Rob
>
> [1] CMUCL's (on x86) is one, but there are many others.
>
> [2] In all of them, at the completion of a GC all of the write
> enable bits are turned off (or write protection bits are
> turned on) on all of the pages in the GC-managed heap
> [*except* for the nursery, of course!]. In some versions,
> for any store into a write-protected heap page the signal
> handler [usually part of the GC] will simply turn write-enable
> back on for that page and return from the signal, which will
> automatically retry the store, which will now quietly succeed.
> At the next GC, any heap pages in generations older that the
> one(s) being collection which have their write enable bits on
> have thus been stored into since the last GC, and all of the
> objects on all such pages are scanned to see if they need to
> be added to the remembered set. Overall, this approach can be
> quite cheap if there are *lots* of stores but to relatively few
> older generation pages.
>
> Another approach is, when a write-protection SIGSEGV occurs, to
> precisely locate the object being stored into and record either
> the whole object (if it's small) or *only* the slot being written
> into (if the object's really large) in the remembered set, then
> (a) temporarily turn on write-enable on the page, (b) emulate
> [that is, perform] the store in the signal handler, (c) "push
> forward" the interrupted PC past the store, (d) turn write-enable
> back off on that page, and (e) return from the signal handler
> to the new PC just past the store. Overall, this approach may
> be cheaper than that previous one if there are relatively few
> total stores but to widely-scattered older generation pages.
>
> [3] This is particularly true if (a) the cost of a write-protection
> trap and the associated SIGSEGV is *very* high on that platform;
> (b) type-propagation can be used to suppress performing the write
> barrier when non-pointer data is being stored; (c) pointer stores
> are fairly infrequent, and (d) pointer stores tend to be clustered
> around regions considerably smaller than a page size [in which
> case that region/cluster size is probably a good choice for the
> "card" size in a card-marking scheme].
>
> -----
> Rob Warnock <····@rpw3.org>
> 627 26th Avenue <URL:http://rpw3.org/>
> San Mateo, CA 94403 (650)572-2607
That's a great explanation of how to use SIGSEGV as a hardware
write barrier in a GC.
In the case of ObjectStore, we had to do it in "hardware" (SIGSEGV)
rather than software, because of the fundamental claim that
we were making, that once you had touched all the pages
that you were going to use during a transaction, your
program would run at absolutely full C++ speed. ObjectStore
(for C++) was, at its heart, a largely-transparent way to
have persistent storage for C++ objects. It was a selling
point that you could use ObjectStore by doing only a small
amount of rewriting of your existing C++ code. I demonstrated
this for our first semi-public demo of the project, by taking
an existing graphics package from Berkeley (I can't remember
its name any more), and converting it, mainly by changing
all the "new"'s to "new (db)" and putting in transaction
boundaries. It only took me a few hours and it worked fine;
this was source code that I had never seen before.
Actually I don't think too many of customers back-converted
their own C++ code, but I do think they converted pre-existing
libraries. If your library didn't do any "new"'s (which some
math libraries, etc, do), you could often just link in the
binaries of the library direction without even recompiling
and it would work just fine on persistent memory.
(I should not talk in the past tense. It's still maintained
and you can buy it from ObjectStore (Progress Software).)
From: Daniel Weinreb
Subject: Re: are relational databases not needed when you use lisp? paul graham did not
Date:
Message-ID: <0y16j.9730$6k1.1699@trndny02>
Maciej Katafiasz wrote:
> Den Wed, 05 Dec 2007 15:05:55 -0800 skrev Duane Rettig:
>
>>> Now, it's hard to reach an informed conclusion from such a short
>>> passage, but instantiation through SEGV sounds so outlandish that I
>>> just have to ask for details, if there are any you can share without
>>> breaching any contracts.
>> Why outlandish?
>
> Perhaps outlandish was a wrong word, but it certainly seemed like an
> unusual take on object instantiation. And my reaction was a combination
> of several factors, see below.
It's not actually object instantiation. Object instantiation
was done by an overloaded "operator new", where the parameter
said what database or cluster to allocate the object in.
This might or might not do a SIGSEGV depending only on whether
it happened to touch a page that was not already writable.
>
>> I can't share facts with you because I don't know many, other than what
>> my fellow developer told me. I can only guess, though, at both reasons
>> for ODI's design and also reasons for your horror. I assume that you
>> can address the latter, and perhaps Daniel Weinreb would consider
>> sharing some of the former. But to take some guesses, I presume that
>> you are either reacting against the very thought of using traditionally
>> error-based programming to handle normal programming situations, or
>> against the long cycle times that would be present in handling user
>> level traps in a unix environment. Recall, though, that not too many
>> years ago 2 Mb was considered to be a huge program, and a file-based
>> database application that could avoid structural program change from the
>> first instantiation of an object to the second was likely to be very
>> efficient for those objects already in memory.
>
> I guess I was thinking about a different meaning of "object
> instantiation". Page fault-like handling is indeed less surprising.
> However, my main objection was that signals are so brittle and offer so
> little in the way of a protocol (there's no way to attach any data to a
> signal, which makes them inherently a global resource, which is very
> yucky). I appreciate the ability to present a uniform interface to the
> outside code, I'm unsure, however, how signals help compared to ordinary
> function calls. Unless you posses the ability to do true page fault
> handling (ie. can rewrite memory references), it seems to me like just a
> very indirect way of doing funcalls.
The SIGSEGV was a lot like a page fault. It told us that the
program was trying to read or write a certain page. We didn't
need to attach any data. The fact of the SIGSEGV was all
we needed.
>
>> As for the timing issue,
>> when you are dealing with a single hit of several hundred to a thousand
>> or two cycles for a signal handler to be dispatched, that cycle count
>> gets swamped by the time it takes to go out to the disk to get the data,
>> especially if it is randomly addressed.
>
> Good point.
Yes, although often the data would already be in the
local cache so that you didn't pay that price of going out
to the server.
>
>> As for the possibility that your reaction is to the very thought of
>> using signals to effect programming - if so, I've seen this kind of
>> reaction before - my theory is that such aversion is based on the
>> tendency of error and exception systems to be afterthoughts, and thus
>> either not complete or not well-tested.
>
> I agree that robust error handling can substantially change the notion
> of what is a recoverable situation, however, Unix signals aren't exactly
> a paragon of expressible, well-integrated error handling mechanism
> design :).
>
> Cheers,
> Maciej
Duane Rettig wrote:
> Maciej Katafiasz <········@gmail.com> writes:
>
>> Den Wed, 05 Dec 2007 08:17:43 -0800 skrev Duane Rettig:
>>
>>> [...] but we knew that instantiation of objects was done by explicitly
>>> forcing a SEGV and handling it [...]
>>
>> Now, it's hard to reach an informed conclusion from such a short passage,
>> but instantiation through SEGV sounds so outlandish that I just have to
>> ask for details, if there are any you can share without breaching any
>> contracts.
>
> Why outlandish?
At least for me it sounds more complicated than necessary. I assume the
part of the database which handles the SEGV is part of the application
process, so you don't mean sending a SEGV to another process, but to the
same process, e.g. by accessing invalid memory. Then the SEGV handler has
to determine what it means. What is the advantage compared to a simple
function call?
--
Frank Buss, ··@frank-buss.de
http://www.frank-buss.de, http://www.it4-systems.de
From: Duane Rettig
Subject: Re: are relational databases not needed when you use lisp? paul graham did not
Date:
Message-ID: <o0sl2g164v.fsf@gemini.franz.com>
Frank Buss <··@frank-buss.de> writes:
> Duane Rettig wrote:
>
>> Maciej Katafiasz <········@gmail.com> writes:
>>
>>> Den Wed, 05 Dec 2007 08:17:43 -0800 skrev Duane Rettig:
>>>
>>>> [...] but we knew that instantiation of objects was done by explicitly
>>>> forcing a SEGV and handling it [...]
>>>
>>> Now, it's hard to reach an informed conclusion from such a short passage,
>>> but instantiation through SEGV sounds so outlandish that I just have to
>>> ask for details, if there are any you can share without breaching any
>>> contracts.
>>
>> Why outlandish?
>
> At least for me it sounds more complicated than necessary. I assume the
> part of the database which handles the SEGV is part of the application
> process, so you don't mean sending a SEGV to another process, but to the
> same process, e.g. by accessing invalid memory. Then the SEGV handler has
> to determine what it means. What is the advantage compared to a simple
> function call?
Space, as I understand it, both code and data. The code accesses the
"bogus" (i.e. unloaded) datum as if it were a real object and gets the
segv, and the segv handler loads it in and ensures that it is in the
right register for the code to get at. The next time it is accessed,
the same code (i.e. without having to take a different route) accesses
the loaded data in the same simple way (i.e. without having to be
concerned about the case where the datum is unloaded) and gets the
datum because it is now loaded.
It's actually simpler than usual - there are fewer decision paths in
the runtime code - all of the decisions and bending over backward is
done in the trap handler, which can arrange for just about any context
to be presented back to the user code.
I must repeat: I don't know the ObjectStore code personally, but it
was described to me, and all of my encounters with it tend to be
consistent with my own experience in programming-by-trap. We actually
do a lot of it in Allegro CL itself, although we don't use traps for
functionality that is likely to occur very often. Instead we trap on
things like unbound-variable errors for symbol-value accesses, so that
you have restarts that include setting the symbol's value and/or
continuing with the value at the point of the trap. A programmatic
interface to such global variable accesses would be much more bulky,
since the locus of code would have to temporarily jump out of the code
to an error handler, and then find its way back into the same code
path. With a trap, the handler has the complete state of the running
lisp and can come back to that point after handling the error. We use
the same technique on our instruction-level breakpoint stepper, which
I describe (along with some of the concepts of user trap handling) in
ftp://ftp.franz.com/pub/duane/break.ps
--
Duane Rettig ·····@franz.com Franz Inc. http://www.franz.com/
555 12th St., Suite 1450 http://www.555citycenter.com/
Oakland, Ca. 94607 Phone: (510) 452-2000; Fax: (510) 452-0182
From: Daniel Weinreb
Subject: Re: are relational databases not needed when you use lisp? paul graham did not
Date:
Message-ID: <48S5j.29224$AX6.27308@trnddc07>
Duane Rettig wrote:
> Frank Buss <··@frank-buss.de> writes:
>
>> Duane Rettig wrote:
>>
>>> Maciej Katafiasz <········@gmail.com> writes:
>>>
>>>> Den Wed, 05 Dec 2007 08:17:43 -0800 skrev Duane Rettig:
>>>>
>>>>> [...] but we knew that instantiation of objects was done by explicitly
>>>>> forcing a SEGV and handling it [...]
>>>> Now, it's hard to reach an informed conclusion from such a short passage,
>>>> but instantiation through SEGV sounds so outlandish that I just have to
>>>> ask for details, if there are any you can share without breaching any
>>>> contracts.
>>> Why outlandish?
>> At least for me it sounds more complicated than necessary. I assume the
>> part of the database which handles the SEGV is part of the application
>> process, so you don't mean sending a SEGV to another process, but to the
>> same process, e.g. by accessing invalid memory. Then the SEGV handler has
>> to determine what it means. What is the advantage compared to a simple
>> function call?
>
> Space, as I understand it, both code and data. The code accesses the
> "bogus" (i.e. unloaded) datum as if it were a real object and gets the
> segv, and the segv handler loads it in and ensures that it is in the
> right register for the code to get at. The next time it is accessed,
> the same code (i.e. without having to take a different route) accesses
> the loaded data in the same simple way (i.e. without having to be
> concerned about the case where the datum is unloaded) and gets the
> datum because it is now loaded.
Yeah, that's the basic idea. SIGSEGV is being used to create
the equivalent of a demand-paged virtual memory, within the
user level (as opposed to the kernel level). ObjectStore
assigns your database to virtual address space, and then
uses mmap/mprotect/SIGSEGV to keep track of which pages
are read, written, or need to be brought in from the server.
This is all explained in our CACM paper. Google for
"Lamb Orenstein Weinreb". Unfortunately it is copyright
by the ACM so you need to have access to the ACM Digital
Library. None of it is secret, though.
>
> It's actually simpler than usual - there are fewer decision paths in
> the runtime code - all of the decisions and bending over backward is
> done in the trap handler, which can arrange for just about any context
> to be presented back to the user code.
>
> I must repeat: I don't know the ObjectStore code personally,
I personally wrote all the stuff being described above. If
you have any questions, I'll be happy to answer them.
but it
> was described to me, and all of my encounters with it tend to be
> consistent with my own experience in programming-by-trap. We actually
> do a lot of it in Allegro CL itself, although we don't use traps for
> functionality that is likely to occur very often. Instead we trap on
> things like unbound-variable errors for symbol-value accesses, so that
> you have restarts that include setting the symbol's value and/or
> continuing with the value at the point of the trap. A programmatic
> interface to such global variable accesses would be much more bulky,
> since the locus of code would have to temporarily jump out of the code
> to an error handler, and then find its way back into the same code
> path. With a trap, the handler has the complete state of the running
> lisp and can come back to that point after handling the error. We use
> the same technique on our instruction-level breakpoint stepper, which
> I describe (along with some of the concepts of user trap handling) in
> ftp://ftp.franz.com/pub/duane/break.ps
Cool. I actually never learned about AllegroStore.
Sorry if you had trouble interacting with ODI. I don't
know who you talked to but I apologize on their behalf.
We tried hard to be cooperative with anyone building
something on top of ObjectStore, in general.
>
From: Duane Rettig
Subject: Re: are relational databases not needed when you use lisp? paul graham did not
Date:
Message-ID: <o08x47ermz.fsf@gemini.franz.com>
Daniel Weinreb <···@alum.mit.edu> writes:
> Cool. I actually never learned about AllegroStore.
I never learned AllegroStore either, at least from the user point of
view - I was always busy in the bowels of the implementation (mostly
where they clashed).
> Sorry if you had trouble interacting with ODI. I don't
> know who you talked to but I apologize on their behalf.
> We tried hard to be cooperative with anyone building
> something on top of ObjectStore, in general.
I don't think we had any trouble with ODI. The major problems we had
to solve were technical ones, especially dealing with the fact that we
both used the same signal for different purposes.
--
Duane Rettig ·····@franz.com Franz Inc. http://www.franz.com/
555 12th St., Suite 1450 http://www.555citycenter.com/
Oakland, Ca. 94607 Phone: (510) 452-2000; Fax: (510) 452-0182
From: Richard M Kreuter
Subject: Re: are relational databases not needed when you use lisp? paul graham did not
Date:
Message-ID: <87ejdzx51x.fsf@progn.net>
Daniel Weinreb <···@alum.mit.edu> writes:
> SIGSEGV is being used to create the equivalent of a demand-paged
> virtual memory, within the user level (as opposed to the kernel
> level). ObjectStore assigns your database to virtual address space,
> and then uses mmap/mprotect/SIGSEGV to keep track of which pages are
> read, written, or need to be brought in from the server. This is
> all explained in our CACM paper. Google for "Lamb Orenstein
> Weinreb". Unfortunately it is copyright by the ACM so you need to
> have access to the ACM Digital Library. None of it is secret,
> though.
There's a publically available paper that cites your paper and calls
their technique similar titled "Pointer Swizzling at Page Fault Time"
by Paul R. Wilson and Sheetal V. Kakkad, linked here:
http://citeseer.ist.psu.edu/92467.html
From: Kent M Pitman
Subject: Re: are relational databases not needed when you use lisp? paul graham did not
Date:
Message-ID: <ud4tkqwuy.fsf@nhplace.com>
Maciej Katafiasz <········@gmail.com> writes:
> Den Wed, 05 Dec 2007 08:17:43 -0800 skrev Duane Rettig:
>
> > [...] but we knew that instantiation of objects was done by explicitly
> > forcing a SEGV and handling it [...]
>
> Now, it's hard to reach an informed conclusion from such a short passage,
> but instantiation through SEGV sounds so outlandish that I just have to
> ask for details, if there are any you can share without breaching any
> contracts.
MACLISP used to manage WITHOUT-INTERRUPTS in a similarly bizarre way.
It wanted to bind interrupts, but there was no way to bind anything
but a variable. So it had a variable called
+INTERNAL-WITHOUT-INTERRUPTS that had its value cell on a pure page
[that's what we called read-only memory]. If you bound it, it would
signal a pure page error and Lisp would decipher the stack frame to
figure out whether to set interrupts on or off. The value of that "variable"
never changed, so you had to read it by reading a completely alternate
facility, (STATUS NOINTERRUPT). It was all kind of weird and indirect.
From: Matthew D. Swank
Subject: Re: are relational databases not needed when you use lisp? paul graham did not
Date:
Message-ID: <pan.2007.12.06.02.27.05.440228@gmail.com>
On Wed, 05 Dec 2007 00:25:47 -0500, Ken Tilton wrote:
> I find myself fascinated by RDF triple stores, and have my eye on
> Redland as an open persistent storage solution for Lisp apps. yeah, to
> hell with persistent CLOS, gavino talked me out of OO. :)
What is your interest in RDF?
Matt
--
"You do not really understand something unless you
can explain it to your grandmother." -- Albert Einstein.
From: Ken Tilton
Subject: Re: are relational databases not needed when you use lisp? paul graham did not
Date:
Message-ID: <ZrJ5j.4850$_%2.4373@newsfe10.lga>
Matthew D. Swank wrote:
> On Wed, 05 Dec 2007 00:25:47 -0500, Ken Tilton wrote:
>
>
>>I find myself fascinated by RDF triple stores, and have my eye on
>>Redland as an open persistent storage solution for Lisp apps. yeah, to
>>hell with persistent CLOS, gavino talked me out of OO. :)
>
>
> What is your interest in RDF?
I think triples are a better way to represent data. Triples are to
objects what objects are to tables and rows, and what dynamic typing is
to static. Triples have the power of natural language without the
ambiguity. In brief, triples are the Lisp Way of data. I think even PG
and RPG would like them.
I hope Redland is good, it would be cool having persistence for free.
kzo
--
http://www.theoryyalgebra.com/
"In the morning, hear the Way;
in the evening, die content!"
-- Confucius
Ken Tilton <···········@optonline.net> writes:
> Matthew D. Swank wrote:
> > On Wed, 05 Dec 2007 00:25:47 -0500, Ken Tilton wrote:
> >
> >> I find myself fascinated by RDF triple stores, and have my eye on
> >> Redland as an open persistent storage solution for Lisp apps. yeah,
> >> to hell with persistent CLOS, gavino talked me out of OO. :)
> > What is your interest in RDF?
>
> I think triples are a better way to represent data. Triples are to
> objects what objects are to tables and rows, and what dynamic typing
I think you have this backwards/upside down. Triples are basically a
(sparse) representation of adjacency matrices (which are typically and
more directly modeled by tables). From a representational POV, all
this stuff is equivalent (including objects or structures which can be
modeled as sets of triples, [after all it's pretty easy to see a that
slot basically encodes a single triple]). Things start to separate
out when you curry in the context of the particular task at hand and
the underlying details of specific implementations of the
representations.
> Triples have the power of natural language without the
> ambiguity.
Have you been drinking before writing this??
/Jon
--
'j' - a n t h o n y at romeo/charley/november com
From: Ken Tilton
Subject: Re: are relational databases not needed when you use lisp? paul graham did not
Date:
Message-ID: <475afae1$0$32065$607ed4bc@cv.net>
jayessay wrote:
> Ken Tilton <···········@optonline.net> writes:
>
>
>>Matthew D. Swank wrote:
>>
>>>On Wed, 05 Dec 2007 00:25:47 -0500, Ken Tilton wrote:
>>>
>>>
>>>>I find myself fascinated by RDF triple stores, and have my eye on
>>>>Redland as an open persistent storage solution for Lisp apps. yeah,
>>>>to hell with persistent CLOS, gavino talked me out of OO. :)
>>>
>>>What is your interest in RDF?
>>
>>I think triples are a better way to represent data. Triples are to
>>objects what objects are to tables and rows, and what dynamic typing
>
>
> I think you have this backwards/upside down. Triples are basically a
> (sparse) representation of adjacency matrices (which are typically and
> more directly modeled by tables). From a representational POV, all
> this stuff is equivalent (including objects or structures which can be
> modeled as sets of triples, [after all it's pretty easy to see a that
> slot basically encodes a single triple]). Things start to separate
> out when you curry in the context of the particular task at hand and
> the underlying details of specific implementations of the
> representations.
If you have to use so many big words, you must be wrong.
>
>
>
>>Triples have the power of natural language without the
>>ambiguity.
>
>
> Have you been drinking before writing this??
I think our different perspectives arise from me being a serious
application developer and you being a Usenet asshole wannabe.
Please follow up, I want to see if my killfile is working.
From: Russell McManus
Subject: Re: are relational databases not needed when you use lisp? paul graham did not
Date:
Message-ID: <87ve78io3t.fsf@thelonious.cl-user.org>
Ken Tilton <···········@optonline.net> writes:
> jayessay wrote:
> > Ken Tilton <···········@optonline.net> writes:
> >
> >>Matthew D. Swank wrote:
> >>
> >>>On Wed, 05 Dec 2007 00:25:47 -0500, Ken Tilton wrote:
> >>>
> >>>
> >>>>I find myself fascinated by RDF triple stores, and have my eye on
> >>>>Redland as an open persistent storage solution for Lisp apps. yeah,
> >>>>to hell with persistent CLOS, gavino talked me out of OO. :)
> >>>
> >>>What is your interest in RDF?
> >>
> >>I think triples are a better way to represent data. Triples are to
> >>objects what objects are to tables and rows, and what dynamic typing
> > I think you have this backwards/upside down. Triples are basically a
> > (sparse) representation of adjacency matrices (which are typically and
> > more directly modeled by tables). From a representational POV, all
> > this stuff is equivalent (including objects or structures which can be
> > modeled as sets of triples, [after all it's pretty easy to see a that
> > slot basically encodes a single triple]). Things start to separate
> > out when you curry in the context of the particular task at hand and
> > the underlying details of specific implementations of the
> > representations.
>
> If you have to use so many big words, you must be wrong.
>
> >
> >>Triples have the power of natural language without the
> >>ambiguity.
> > Have you been drinking before writing this??
>
> I think our different perspectives arise from me being a serious
> application developer and you being a Usenet asshole wannabe.
>
> Please follow up, I want to see if my killfile is working.
You fell victim to one of the classic blunders! The most famous is
never get involved in a land war in Asia, but only slightly less
well-known is this: never go in against Ken Tilton when drinking is on
the line! Ha ha ha ha ha ha ha! Ha ha ha ha ha ha ha! Ha ha ha...
-russ
From: Ken Tilton
Subject: Re: are relational databases not needed when you use lisp? paul graham did not
Date:
Message-ID: <475c78dd$0$32090$607ed4bc@cv.net>
Russell McManus wrote:
> Ken Tilton <···········@optonline.net> writes:
>
>
>>jayessay wrote:
>>
>>>Ken Tilton <···········@optonline.net> writes:
>>>
>>>
>>>>Matthew D. Swank wrote:
>>>>
>>>>
>>>>>On Wed, 05 Dec 2007 00:25:47 -0500, Ken Tilton wrote:
>>>>>
>>>>>
>>>>>
>>>>>>I find myself fascinated by RDF triple stores, and have my eye on
>>>>>>Redland as an open persistent storage solution for Lisp apps. yeah,
>>>>>>to hell with persistent CLOS, gavino talked me out of OO. :)
>>>>>
>>>>>What is your interest in RDF?
>>>>
>>>>I think triples are a better way to represent data. Triples are to
>>>>objects what objects are to tables and rows, and what dynamic typing
>>>
>>>I think you have this backwards/upside down. Triples are basically a
>>>(sparse) representation of adjacency matrices (which are typically and
>>>more directly modeled by tables). From a representational POV, all
>>>this stuff is equivalent (including objects or structures which can be
>>>modeled as sets of triples, [after all it's pretty easy to see a that
>>>slot basically encodes a single triple]). Things start to separate
>>>out when you curry in the context of the particular task at hand and
>>>the underlying details of specific implementations of the
>>>representations.
>>
>>If you have to use so many big words, you must be wrong.
>>
>>
>>>>Triples have the power of natural language without the
>>>>ambiguity.
>>>
>>>Have you been drinking before writing this??
>>
>>I think our different perspectives arise from me being a serious
>>application developer and you being a Usenet asshole wannabe.
>>
>>Please follow up, I want to see if my killfile is working.
>
>
> You fell victim to one of the classic blunders! The most famous is
> never get involved in a land war in Asia, but only slightly less
> well-known is this: never go in against Ken Tilton when drinking is on
> the line! Ha ha ha ha ha ha ha! Ha ha ha ha ha ha ha! Ha ha ha...
I should have made it clear to Jon that his job was at stake.
Let's save his yammering about representational Turing equivalence in
the context of an engineering discussion for the C.L.L. Blooper show.
Overall his babbling makes for a nice segue into a larger theme of most
folks not understanding RDF, because most people just read hype. Just as
actual Lisp programmers overruled McCarthy on sexpr/mexpr, actually
programming with triples is needed to get past the (mistaken) hype machine.
Unfortunately the combination of "c.l.l." and "actual programming"...
inconceivable!
I just wish financial realities did not make it necessary for me to
concentrate on the Algebra software. I think it is time for KennyCL
featuring an OpenGL-based GUI lib, an RDF substitute for CLOS,
persistence via Redland, and... and... I know I am forgetting something.
Now if only the so-called cliki young tigers could stop preening on
#lisp and get SBCL stabilized on win32.
kt
--
http://www.theoryyalgebra.com/
"In the morning, hear the Way;
in the evening, die content!"
-- Confucius
Ken Tilton <···········@optonline.net> writes:
> Let's save his yammering about representational Turing equivalence in
> the context of an engineering discussion for the C.L.L. Blooper show.
I note you missed the speicific bits about how the context and task
(and implementations) change the expressiveness and choices. So, I
was not talking merely "turing equivalence" of representations. Also,
you still had it backwards/upside down.
> Overall his babbling makes for a nice segue into a larger theme of
> most folks not understanding RDF, because most people just read
> hype.
Could be. But I've been working in this general area (KR) for 15
years and specifically with/on RDF and triple stores and such for
about 4 years or so. How long have you been dabbling with it?
> Just as actual Lisp programmers overruled McCarthy on sexpr/mexpr,
> actually programming with triples is needed to get past the
> (mistaken) hype machine.
Right. Go for it. You will learn about the trade offs soon enough.
Maybe your application will be perfect for them.
> Unfortunately the combination of "c.l.l." and "actual
> programming"... inconceivable!
Speaking of yourself?
> concentrate on the Algebra software. I think it is time for KennyCL
> featuring an OpenGL-based GUI lib, an RDF substitute for CLOS,
> persistence via Redland
That GUI lib sounds great. I think I could offer you a better
substrate (not just for CLOS, but in general) and with persistence.
Certainly outperforms the current offerings...
/Jon
--
'j' - a n t h o n y at romeo/charley/november com
Ken Tilton <···········@optonline.net> writes:
> jayessay wrote:
> > Ken Tilton <···········@optonline.net> writes:
> >
> >>Matthew D. Swank wrote:
> >>
> >>>On Wed, 05 Dec 2007 00:25:47 -0500, Ken Tilton wrote:
> >>>
> >>>
> >>>>I find myself fascinated by RDF triple stores, and have my eye on
> >>>>Redland as an open persistent storage solution for Lisp apps. yeah,
> >>>>to hell with persistent CLOS, gavino talked me out of OO. :)
> >>>
> >>>What is your interest in RDF?
> >>
> >>I think triples are a better way to represent data. Triples are to
> >>objects what objects are to tables and rows, and what dynamic typing
>
> > I think you have this backwards/upside down. Triples are basically a
> > (sparse) representation of adjacency matrices (which are typically and
> > more directly modeled by tables). From a representational POV, all
> > this stuff is equivalent (including objects or structures which can be
> > modeled as sets of triples, [after all it's pretty easy to see a that
> > slot basically encodes a single triple]). Things start to separate
> > out when you curry in the context of the particular task at hand and
> > the underlying details of specific implementations of the
> > representations.
>
> If you have to use so many big words, you must be wrong.
Who would have thought that stating the obvious would have touched a
nerve? Heh.
> >>Triples have the power of natural language without the
> >>ambiguity.
> > Have you been drinking before writing this??
>
> I think our different perspectives arise from me being a serious
> application developer and you being a Usenet asshole wannabe.
And here I was trying to give you a graceful exit. Seriously claiming
that "triples have the power of natural language" just makes you look
like a fool. I guess this goes under the heading of "no good deed
goes unpunished"...
Perspectives are good. I have no idea if you really are a "serious"
application developer. But you say it enough and I have no reason to
doubt it. OTOH, you are definitely wrong on your second point: I
don't claim or desire to be a "usenet * wannabe" for any value of *.
If you like triples - use them. I don't care.
> Please follow up, I want to see if my killfile is working.
Whatever...
/Jon
--
'j' - a n t h o n y at romeo/charley/november com
> Had we written in Java, it's likely that we would have
> used Hibernate, and had less work to do ourselves.
the last time i used Hibernate two years ago it was, well, quite
inflexible.
> I don't currently know of an existing available
> open-source object-to-relational system for Common
> Lisp, at least not one that would meet our needs.
we have http://common-lisp.net/project/cl-perec/ using
http://common-lisp.net/project/cl-rdbms/ and they are both free.
i don't want to suggest that cl-perec would be useful for something
like what ITA is working on, but i think its design has the place for
all those optimizations, if someone had enough free time.
- attila
> Had we written in Java, it's likely that we would have
> used Hibernate, and had less work to do ourselves.
> I don't currently know of an existing available
> open-source object-to-relational system for Common
> Lisp, at least not one that would meet our needs.
cl-perec is an object-to-realtional mapping tool for Common Lisp and
it is open source.
Two out of three is not that bad...
levy
ps: We are using it in a production system.
From: Dmitriy Ivanov
Subject: Re: are relational databases not needed when you use lisp? paul graham did not
Date:
Message-ID: <fj5nea$j2$1@news.aha.ru>
"Daniel Weinreb" <···@alum.mit.edu> wrote:
DW> I don't currently know of an existing available
DW> open-source object-to-relational system for Common
DW> Lisp, at least not one that would meet our needs.
What is your opinion about Xanalys/LispWork CommonSQL? For it, open-source
counterparts do exist, e.g. CLSQL and YSQL. Can it be considered an
object-relational system?
--
Sincerely,
Dmitriy Ivanov
lisp.ystok.ru
From: Daniel Weinreb
Subject: Re: are relational databases not needed when you use lisp? paul graham did not
Date:
Message-ID: <Qnw5j.2332$UG1.731@trnddc01>
Dmitriy Ivanov wrote:
> "Daniel Weinreb" <···@alum.mit.edu> wrote:
>
> DW> I don't currently know of an existing available
> DW> open-source object-to-relational system for Common
> DW> Lisp, at least not one that would meet our needs.
>
> What is your opinion about Xanalys/LispWork CommonSQL? For it, open-source
> counterparts do exist, e.g. CLSQL and YSQL. Can it be considered an
> object-relational system?
> --
> Sincerely,
> Dmitriy Ivanov
> lisp.ystok.ru
>
>
Sorry, I'm not familiar with those. Anything with SQL in its
name is almost certainly a relational-type database rather
than an object-oriented-type database, though.
From: Andreas Thiele
Subject: Re: are relational databases not needed when you use lisp? paul graham did not
Date:
Message-ID: <fj3ju9$sh2$03$1@news.t-online.com>
"gavino" <·········@gmail.com> schrieb im Newsbeitrag ·········································@n20g2000hsh.googlegroups.com...
> paul graham said he did not use a relational db for viaweb
>
> can one do webapps with lisp and avoid use of a relational db?
Yes, there are some alternatives. Although I use relational DBs, I don't appreciate them
very much. They are not very handy. It's simply marketing for me. The customer is happy
when he has access to the relational DB and can do reporting the same way throughout his
company.
At least there is 'Allegro Cache' a commercial solution by franz. I think the idea behind it
can be found in 'elephant' (http://common-lisp.net/project/elephant/). But there are other
solutions out there. I'm sure people will mention very soon :)
Andreas
On Dec 4, 8:15 pm, "Andreas Thiele" <······@nospam.com> wrote:
> Yes, there are some alternatives. Although I use relational DBs, I don't appreciate them
> very much. They are not very handy. It's simply marketing for me.
No not marketing, you simply have a solid toolchain from different
vendors and don't bind
yourself to strange solutions. It's important for mission critical.
Especially if the application
grows and you need a realiable and good scaling data store the easiest
and best way are still RDBMS.
On Dec 4, 9:16 am, llothar <·······@web.de> wrote:
> On Dec 4, 8:15 pm, "Andreas Thiele" <······@nospam.com> wrote:
>
> > Yes, there are some alternatives. Although I use relational DBs, I don't appreciate them
> > very much. They are not very handy. It's simply marketing for me.
>
> No not marketing, you simply have a solid toolchain from different
> vendors and don't bind
> yourself to strange solutions. It's important for mission critical.
> Especially if the application
> grows and you need a realiable and good scaling data store the easiest
> and best way are still RDBMS.
Could you please tell me what units "solid", "strange", "easiest",
"important", "good" and "best" are measured in? Otherwise, they
are marketing.
Reliability is usually measured in FITS, do you have any numbers?
--S
From: Ken Tilton
Subject: Re: are relational databases not needed when you use lisp? paul graham did not
Date:
Message-ID: <e_e5j.2230$BI5.397@newsfe09.lga>
smallpond wrote:
> On Dec 4, 9:16 am, llothar <·······@web.de> wrote:
>
>>On Dec 4, 8:15 pm, "Andreas Thiele" <······@nospam.com> wrote:
>>
>>
>>>Yes, there are some alternatives. Although I use relational DBs, I don't appreciate them
>>>very much. They are not very handy. It's simply marketing for me.
>>
>>No not marketing, you simply have a solid toolchain from different
>>vendors and don't bind
>>yourself to strange solutions. It's important for mission critical.
>>Especially if the application
>>grows and you need a realiable and good scaling data store the easiest
>>and best way are still RDBMS.
>
>
> Could you please tell me what units "solid", "strange", "easiest",
> "important", "good" and "best" are measured in?
That would be the fud.
> Otherwise, they
> are marketing.
check.
kt
On Dec 4, 11:02 am, smallpond <·········@juno.com> wrote:
> On Dec 4, 9:16 am, llothar <·······@web.de> wrote:
>
> > On Dec 4, 8:15 pm, "Andreas Thiele" <······@nospam.com> wrote:
>
> > > Yes, there are some alternatives. Although I use relational DBs, I don't appreciate them
> > > very much. They are not very handy. It's simply marketing for me.
>
> > No not marketing, you simply have a solid toolchain from different
> > vendors and don't bind
> > yourself to strange solutions. It's important for mission critical.
> > Especially if the application
> > grows and you need a realiable and good scaling data store the easiest
> > and best way are still RDBMS.
>
> Could you please tell me what units "solid", "strange", "easiest",
> "important", "good" and "best" are measured in? Otherwise, they
> are marketing.
They are measured in units of ease, of course. It is a lot easier to
smack your data around 'til it fits into tables than it is to touch
the filesystem. It gets harder, each year. to remember how the
filesystem works.
On Dec 4, 8:15 am, "Andreas Thiele" <······@nospam.com> wrote:
> "gavino" <·········@gmail.com> schrieb im ····················································@n20g2000hsh.googlegroups.com...
>
> > paul graham said he did not use a relational db for viaweb
>
> > can one do webapps with lisp and avoid use of a relational db?
>
> At least there is 'Allegro Cache' a commercial solution by franz. I think the idea behind it
> can be found in 'elephant' (http://common-lisp.net/project/elephant/). But there are other
> solutions out there. I'm sure people will mention very soon :)
Interestingly, elephant uses a relational-db for it's objectstore:
http://common-lisp.net/project/elephant/doc/Getting-Started.html#Getting-Started
But I think we always have this question in our heads of how to map
complex programming data structures to the relational data model...
even trees have not one but two different popular ways of getting shoe-
horned into databases - nested sets and adjacency models.
Perhaps we should start all programming projects from a database and
simply build new database operators to accomodate to program specs.
That way we dont have the object-relational mismatch.
Just brainstorming...
From: Edi Weitz
Subject: Re: are relational databases not needed when you use lisp? paul graham did not
Date:
Message-ID: <uzlwqxwab.fsf@agharta.de>
On Tue, 4 Dec 2007 07:55:17 -0800 (PST), "metaperl.com" <········@gmail.com> wrote:
> Interestingly, elephant uses a relational-db for it's objectstore:
> http://common-lisp.net/project/elephant/doc/Getting-Started.html#Getting-Started
The page you're linking to specifically mentions that you can choose
between an SQL database or BerkeleyDB. And BerkeleyDB is /not/ a
relational DB. Furthermore, Elephant has what they call a
"multi-store architecture" which means that these are "just"
interchangeable backends. There's nothing preventing you from writing
your own backend in Lisp, for example.
Edi.
--
Lisp is not dead, it just smells funny.
Real email: (replace (subseq ·········@agharta.de" 5) "edi")
From: Dan Muller
Subject: Re: are relational databases not needed when you use lisp? paul graham did not
Date:
Message-ID: <vrn5j.2920$Vq.524@nlpi061.nbdc.sbc.com>
Edi Weitz <········@agharta.de> writes:
> "metaperl.com" <········@gmail.com> wrote:
>
>> Interestingly, elephant uses a relational-db for it's objectstore:
>
> The page you're linking to specifically mentions that you can choose
> between an SQL database or BerkeleyDB. And BerkeleyDB is /not/ a
> relational DB.
Neither is an SQL DBMS a relational DBMS, for that matter. But that's
a topic covered at length in different venues...
Hi gavino,
On Dec 4, 7:56 am, gavino <·········@gmail.com> wrote:
> paul graham said he did not use arelationaldb for viaweb
Yes, Graham mentioned there was no database, see:
http://209.85.135.104/search?q=cache:h6H2sLL-EEYJ:www.paulgraham.com/vwfaq.html+viaweb+database+%22didn%27t+use%22&hl=en&ct=clnk&cd=1&gl=us&client=opera
However, take note most of his VCs thought they were nuts. And having
been involved in similar startup situation, they had a point. Viaweb
created online stores for merchants w/ customers, orders, etc. Every
business dealing w/ that kind of data puts it into a database so you
can query it later, e.g., to report sales, do incident management, and
a thousand other things. The VCs can be slimy but are not total
idiots.
So I'd disagree w/ Graham on that point, and I'm sure at some point
there appeared some sort of database at ViaWeb being fed w/ the data
produced by the Lisp-based app. No business could be run w/out one.
Back in 2002 I had decided I wanted to do some Oracle database work in
some alternative to Java/Perl. It was Graham's success w/ Lisp at
ViaWeb that led me to use Lisp, specifically CLISP. Out of that came
CLISP's Oracle module, which really worked well for me since.
In my 15 yrs or so dealing w/ relational databases, I've yet to
encounter a general purpose programming language that was by nature
more or less well-suited to apps dealing w/ manipulation of relational
database based data. Except possibly that Lisp has some features that
put it slightly above others. (I can elaborate more on that if there
is interest).
Not all Lisp fans dislike databases, either. Phil Greenspun has this
to say about Oracle:
http://209.85.135.104/search?q=cache:ZrCM-0MiSCYJ:philip.greenspun.com/wtr/oracle-tips.html+greenspun+%22oracle+lets+me+sleep+at+night%22&hl=en&ct=clnk&cd=1&gl=us&client=opera
... and on this I agree w/ Phil completely.
> can one do webapps withlispand avoid use of arelationaldb?
You should think carefully why you are avoiding a relational database.
Are you sure you don't want one? They're not hard to learn to use,
and it's not an accident declarative queries in SQL have been around
and used to good effect since the '70s. Just as Lisp's longevity is
also no accident.
If your data lends itself to representation a relational model, and
you put your data into a decent database early, the time may come when
you need to do analysis and reporting on that data. This is
particularly true of "business" data like sales, reservations,
customers, etc. I think it was mentioned that ITA uses Lisp for part
of their reservation system and Oracle RAC as the data house. None
of that is surprising to me.
Happy hacking,
John Hinsdale