From: gavino
Subject: is a relational db a god tool or not really worth it?
Date: 
Message-ID: <1171874348.638770.88710@m58g2000cwm.googlegroups.com>
http://www.paulgraham.com/vwfaq.html Paul Graham states here that he
did not use a relational or other database in his work.

Is a database not needed for lisp webapps?
Can one get a fast app without one?
What is the tradeoff?
It seems most web setups include a db as a matter of course such as php
+mysql, or ruby on rails etc.

What are the pros and cons and once one starts to scale does a
relational db become mandatory?

From: Tim Bradshaw
Subject: Re: is a relational db a god tool or not really worth it?
Date: 
Message-ID: <1171891551.148684.30390@m58g2000cwm.googlegroups.com>
On Feb 19, 8:39 am, "gavino" <·········@gmail.com> wrote:

> Can one get a fast app without one?

The performance of typical Unix filesystems is pretty good, if you
treat them right.

> What is the tradeoff?

Well, at some point you will probably need to scale beyond one
machine.  So now you either need to expose the filesystem to multiple
machines over NFS or similar, which will be exciting in all the wrong
ways, or you write some kind of server which sits on one machine and
talks to all the others, doing the fs stuff behind the scenes.

And perhaps you want transactions and rollback, so you write all that
into your server, which is a lot of work, and very hard to get right
("every sufficiently large system contains a buggy, poorly
implemented ...").

And now your server is the bottleneck, so you need to run it on a
bigger, multiprocessor machine.  You now have to deal with making it
scale, and revisit all the assumptions you made in the transactional
stuff above.

And of course, you want to be able to search lots of data fast, so you
need to generate indices, and update them, and so on.

You probably want a language for talking to your system which is a bit
flexible and abstract, so you can do all this reimplementation without
breaking things.

What you've just written smells suspiciously like a database.

Of course, you don't always need one, and there is a kind of database
cult, whose members (before they paint you green and bury you in a
pit) will encourage you to use one in all sorts of places where you
don't, actually, need one (why does my mail reader use SQL?).

--tim
From: Pascal Bourguignon
Subject: Re: is a relational db a god tool or not really worth it?
Date: 
Message-ID: <87ejom8e80.fsf@thalassa.informatimago.com>
"Tim Bradshaw" <··········@tfeb.org> writes:
> Of course, you don't always need one, and there is a kind of database
> cult, whose members (before they paint you green and bury you in a
> pit) will encourage you to use one in all sorts of places where you
> don't, actually, need one (why does my mail reader use SQL?).

Because we need persistence (of some data, like greylists, etc), and
programmers, being the lazy bums we are want it easy.

So until we use an OS like eros (and perhaps coyotos, I've not
followed what's the decision of the week regarding transparent
persistence), using a SQL database is an easy way to get persistent
data in any application.

Perhaps mmap(2) is under utilised?  
On the other hand, mysql_query might be more portable...


-- 
__Pascal Bourguignon__                     http://www.informatimago.com/

NEW GRAND UNIFIED THEORY DISCLAIMER: The manufacturer may
technically be entitled to claim that this product is
ten-dimensional. However, the consumer is reminded that this
confers no legal rights above and beyond those applicable to
three-dimensional objects, since the seven new dimensions are
"rolled up" into such a small "area" that they cannot be
detected.
From: Tim Bradshaw
Subject: Re: is a relational db a god tool or not really worth it?
Date: 
Message-ID: <1171902594.261535.84830@v33g2000cwv.googlegroups.com>
On Feb 19, 1:32 pm, Pascal Bourguignon <····@informatimago.com> wrote:
> "Tim Bradshaw" <··········@tfeb.org> writes:
> >(why does my mail reader use SQL?).
>
> Because we need persistence (of some data, like greylists, etc), and
> programmers, being the lazy bums we are want it easy.

I understand that mail readers need persistence.  The issue was: why
the too-often-seen use-a-vast-SQL-database rather than use-the-unix-
filesystem-with-perhaps-some-lightweight-non-SQL-database.     You
blame it on laziness, which is plausiblle, but I think we all know
it's actually the men in robes whispering magic spells in your ear &
painting you green.

--tim
From: Kirk  Sluder
Subject: Re: is a relational db a god tool or not really worth it?
Date: 
Message-ID: <kirk-7A1965.12033619022007@newsclstr02.news.prodigy.com>
In article <·······················@v33g2000cwv.googlegroups.com>,
 "Tim Bradshaw" <··········@tfeb.org> wrote:

> I understand that mail readers need persistence.  The issue was: why
> the too-often-seen use-a-vast-SQL-database rather than use-the-unix-
> filesystem-with-perhaps-some-lightweight-non-SQL-database.     You
> blame it on laziness, which is plausiblle, but I think we all know
> it's actually the men in robes whispering magic spells in your ear &
> painting you green.
> 
> --tim

Which mail readers in specific?  Most that I know about use vanilla 
mbox. The few exceptions are MSOutlook which uses a proprietary 
storage system, and Apple Mail.app which stores messages in a 
Maildir like structure with an sqlite (I think) metadata index.
From: Tim Bradshaw
Subject: Re: is a relational db a god tool or not really worth it?
Date: 
Message-ID: <ercsf0$3qn$1$8300dec7@news.demon.co.uk>
On 2007-02-19 17:03:36 +0000, Kirk  Sluder <····@nospam.jobsluder.net> said:

> Which mail readers in specific?  Most that I know about use vanilla
> mbox. The few exceptions are MSOutlook which uses a proprietary
> storage system, and Apple Mail.app which stores messages in a
> Maildir like structure with an sqlite (I think) metadata index.

I wasn't thinking of specific instances, and it may be there are none.  
What I had in mind were things like various insanity which I think went 
on with netscape ... ah, yes, from http://www.jwz.org/doc/mailsum.html:

--cut--
My code had summary files that were on the order of 2% of the size of 
the folder they were summarizing, and was blazingly fast in all 
respects. The 4.0 code had an overhead closer to 30% (last time I 
checked) and was insanely slow, not to mention extremely fragile: their 
summary files got corrupted all the time.

But they were databases and C++ so they were better.
--cut--

Relatedly I think the Mozilla calendar (OK, not a mail reader) uses an 
SQL database.  And (perhaps) Mail.app uses SQL after all, for its 
indices.

From the same page as above: "once those database worms eat into your 
brain, it's hard to ever get anything practical done again. To a 
database person, every nail looks like a thumb."  Couldn't have put it 
better myself.

--tim
From: gavino
Subject: Re: is a relational db a god tool or not really worth it?
Date: 
Message-ID: <1171917385.194582.98690@j27g2000cwj.googlegroups.com>
On Feb 19, 11:06 am, Tim Bradshaw <····@tfeb.org> wrote:
> On 2007-02-19 17:03:36 +0000, Kirk  Sluder <····@nospam.jobsluder.net> said:
>
> > Which mail readers in specific?  Most that I know about use vanilla
> > mbox. The few exceptions are MSOutlook which uses a proprietary
> > storage system, and Apple Mail.app which stores messages in a
> > Maildir like structure with an sqlite (I think) metadata index.
>
> I wasn't thinking of specific instances, and it may be there are none.
> What I had in mind were things like various insanity which I think went
> on with netscape ... ah, yes, fromhttp://www.jwz.org/doc/mailsum.html:
>
> --cut--
> My code had summary files that were on the order of 2% of the size of
> the folder they were summarizing, and was blazingly fast in all
> respects. The 4.0 code had an overhead closer to 30% (last time I
> checked) and was insanely slow, not to mention extremely fragile: their
> summary files got corrupted all the time.
>
> But they were databases and C++ so they were better.
> --cut--
>
> Relatedly I think the Mozilla calendar (OK, not a mail reader) uses an
> SQL database.  And (perhaps) Mail.app uses SQL after all, for its
> indices.
>
> From the same page as above: "once those database worms eat into your
> brain, it's hard to ever get anything practical done again. To a
> database person, every nail looks like a thumb."  Couldn't have put it
> better myself.
>
> --tim

Perhaps a division of labor is in order?
dbs for some parts of problem and non for other?
Funny though because part of using a db is that it wants to contain
all the data so it can be sued to get the answer to any query.  once
data is outside fo it then it can't.......
From: Kirk  Sluder
Subject: Re: is a relational db a god tool or not really worth it?
Date: 
Message-ID: <kirk-245B71.15442619022007@newsclstr02.news.prodigy.com>
In article <·····················@news.demon.co.uk>,
 Tim Bradshaw <···@tfeb.org> wrote:

> On 2007-02-19 17:03:36 +0000, Kirk  Sluder <····@nospam.jobsluder.net> said:
> 
> > Which mail readers in specific?  Most that I know about use vanilla
> > mbox. The few exceptions are MSOutlook which uses a proprietary
> > storage system, and Apple Mail.app which stores messages in a
> > Maildir like structure with an sqlite (I think) metadata index.
>
> Relatedly I think the Mozilla calendar (OK, not a mail reader) uses an 
> SQL database.  And (perhaps) Mail.app uses SQL after all, for its 
> indices.

I actually like how they did this for Mail.app from an end-user 
perspective because it enables extremely quick searches across 
multiple mailboxes and makes opening large mailboxes extremely 
snappy.  The consequence is that I don't notice when my mailbox 
sizes creep up to the thousands.  To my memory mbox really starts to 
drag with increasing mailbox size.

Of course one of the reasons they hacked around this way was because 
they went with maildir-style storage with one message per file 
rather than flat mbox stores containing hundreds of messages.
 
> From the same page as above: "once those database worms eat into your 
> brain, it's hard to ever get anything practical done again. To a 
> database person, every nail looks like a thumb."  Couldn't have put it 
> better myself.

I'm a bit catholic in regards to databases, mbox + cache file is 
just another database to me.  It's interesting to me how sqlite is 
growing into that niche between "roll you own file format" and "run 
a database server" that was previously the domain of bdb. 


> 
> --tim
From: Tim Bradshaw
Subject: Re: is a relational db a god tool or not really worth it?
Date: 
Message-ID: <erd574$ced$1$8300dec7@news.demon.co.uk>
On 2007-02-19 20:44:25 +0000, Kirk  Sluder <····@nospam.jobsluder.net> said:
> 
> I actually like how they did this for Mail.app from an end-user
> perspective because it enables extremely quick searches across
> multiple mailboxes and makes opening large mailboxes extremely
> snappy.  The consequence is that I don't notice when my mailbox
> sizes creep up to the thousands.  To my memory mbox really starts to
> drag with increasing mailbox size.

I think Mail.app works pretty well, though I think I'd assumed that it 
somehow hooked into Spotlight (which of course probably uses some db of 
its own, so, well).

> 
> Of course one of the reasons they hacked around this way was because
> they went with maildir-style storage with one message per file
> rather than flat mbox stores containing hundreds of messages.

That must play better with Spotlight etc, because it must make it 
easier to have a single object (file) to point at.

> 
> I'm a bit catholic in regards to databases

Me too, witness the articles in this thread.  Cults are bad, using the 
right tool is good.  I spend much of my life looking after (machines 
which run) databases=, some of them even doing useful work.

--tim
From: Kirk  Sluder
Subject: Re: is a relational db a god tool or not really worth it?
Date: 
Message-ID: <kirk-B85C20.22122219022007@newsclstr02.news.prodigy.com>
In article <·····················@news.demon.co.uk>,
 Tim Bradshaw <···@tfeb.org> wrote:

> On 2007-02-19 20:44:25 +0000, Kirk  Sluder <····@nospam.jobsluder.net> said:
> > 
> > I actually like how they did this for Mail.app from an end-user
> > perspective because it enables extremely quick searches across
> > multiple mailboxes and makes opening large mailboxes extremely
> > snappy.  The consequence is that I don't notice when my mailbox
> > sizes creep up to the thousands.  To my memory mbox really starts to
> > drag with increasing mailbox size.
> 
> I think Mail.app works pretty well, though I think I'd assumed that it 
> somehow hooked into Spotlight (which of course probably uses some db of 
> its own, so, well).

From what I've read it does both, (which is the downside).  
Spotlight searches are used for searching on message content, while 
the Mail.app db is used for Subject, From and To searches.

> --tim
From: gavino
Subject: Re: is a relational db a god tool or not really worth it?
Date: 
Message-ID: <1171916968.461682.21510@a75g2000cwd.googlegroups.com>
INteresting I saw EROS a high availability (capability?) os.  They
unplugged the power and the os was back up in like 1 minute. Haev you
tried it Mr B?
From: ···············@gmail.com
Subject: Re: is a relational db a god tool or not really worth it?
Date: 
Message-ID: <1171892632.284865.74910@m58g2000cwm.googlegroups.com>
On 19 Feb, 08:39, "gavino" <·········@gmail.com> wrote:
> http://www.paulgraham.com/vwfaq.htmlPaul Graham states here that he
> did not use a relational or other database in his work.

Your questions don't really make sense. I realise many others on here
have an opinion of why your questions don't make sense but I'll try to
offer some answers anyway.

> Is a database not needed for lisp webapps?

This doesn't really make sense. The requirements for a database depend
on the type of application you're building, not on the programming
language used. Admittedly, with all the clever features that Lisp
provides you can probably squeeze a few extra miles out of it than say
PHP but ultimately if you have a lot of relational data then a
relational database makes sense. If you have a lot of flat structures
then you may be able to make something with the file system alone (and
some nifty code). If you have only a small a mount of data then maybe
some in memory structures could be in order. What are you building?

> Can one get a fast app without one?

Below a certain volume/complexity of data you'll be faster without
one. Above a certain volume/complexity you'll be faster with one.
Again, what are you building?

> What is the tradeoff?

The usual: speed/complexity/scalability.

> It seems most web setups include a db as a matter of course such as php
> +mysql, or ruby on rails etc.

You're probably right that many apps that have one don't need one.
There's a certain comfort in using one I guess, especially when it
becomes a habit. But it should depend on what you're building. What
are you building?

> What are the pros and cons and once one starts to scale does a
> relational db become mandatory?

If it's the data that's scaling then a DB makes sense. If it's just
number of hits on a fairly static site then it makes no difference.
What are you building?

HTH,
Phil
From: gavino
Subject: Re: is a relational db a god tool or not really worth it?
Date: 
Message-ID: <1171917166.050143.118140@s48g2000cws.googlegroups.com>
Why do my questions not make sense?  Perhaps a bit of perspective.  I
am aunix admin and I don't program nor have I developed webapps.  I
suspect most of what already exiasts because I see the herd mentality
and how many technologies used suck in my workplaces.  I hope that
lesser popularity free software may be more potent adn am learning to
use as much of it as I can in the shortest time.   I ask these things
hoping that other free software users perhaps have opinions.  I am not
a troll at all.
From: ···············@gmail.com
Subject: Re: is a relational db a god tool or not really worth it?
Date: 
Message-ID: <1171918676.140898.130260@v33g2000cwv.googlegroups.com>
On Feb 19, 8:32 pm, "gavino" <·········@gmail.com> wrote:
> Why do my questions not make sense?  Perhaps a bit of perspective.  I
> am aunix admin and I don't program nor have I developed webapps.  I
> suspect most of what already exiasts because I see the herd mentality
> and how many technologies used suck in my workplaces.  I hope that
> lesser popularity free software may be more potent adn am learning to
> use as much of it as I can in the shortest time.   I ask these things
> hoping that other free software users perhaps have opinions.  I am not
> a troll at all.

Your questions didn't make sense as you hadn't provided any context.
If I went onto alt.transport (or whatever...) and posted a question
saying "Should I buy a bike? I've heard they're good but maybe I
should get a car instead? Or should I just walk?" nobody would be able
to help me as I've not mentioned where I live and where I need to get
to. Perhaps if you could tell us roughly what you want your new
program to do then we could provide better advice.

For example, if you're a Unix admin and you want to write a web app to
view print queues, create new users accounts remotely and diagnose
network problems then you probably don't need much of a DB, if any.
But if you want to write a website for your department or company and
that needs user logins, newsletter subscriptions and a user editable
wiki then maybe it would be easier to use a DB. But then you'd
probably be better off with a canned portal/wiki framework (probably
written in PHP or Perl or some such).

Seriously, what do you want to do?!

Phil
From: George Neuner
Subject: Re: is a relational db a god tool or not really worth it?
Date: 
Message-ID: <t75lt29r98n47m4gt72pi4nfv7qoseq7mt@4ax.com>
On 19 Feb 2007 12:32:46 -0800, "gavino" <·········@gmail.com> wrote:

>Why do my questions not make sense? 

I may be all wrong, but my impression is that you don't think much
about your questions before posting them.  For example, you started
this thread with the observation that "Paul Graham states that he did
not use a database" and then ask whether databases have worth.

What's wrong with this question?  

1) Just because Paul Graham's opinions have been published doesn't
make them correct - being published simply makes them well known.

2) The majority of applications do not need an industrial strength
database.  Viaweb happens to be one that could go either way.

3) Graham's decision not to use a database for Viaweb is neither a
condemnation of databases in general nor a negative opinion on the use
of databases for web applications - it was an opinion that a database
was not needed for that particular project.  Another developer with
Graham's experience may well have decided the opposite.


You frequently post little "what about this" and "compare and contrast
these for me" types of questions.  You may think you are innocently
soliciting opinions from people who have used [whatever the topic is]
but the impression you leave is that either you are unable to research
things on your own - which makes you no friends in c.l.l - or that you
are deliberately trying to cause friction between their advocates.


>Perhaps a bit of perspective.  I am a unix admin and I don't program
>nor have I developed webapps.  I suspect most of what already exists
>because I see the herd mentality and how many technologies used suck
>in my workplaces.  I hope that lesser popularity free software may be
>more potent adn am learning to use as much of it as I can in the
>shortest time.   I ask these things hoping that other free software 
>users perhaps have opinions.  I am not a troll at all.

How is it that you are certain the tools "suck" when you have never
used them.  It's far more likely that the developers suck.  Software
development is an art - being a good technical programmer is only a
small part of it.  IMO the vast majority of "professional" developers
would best benefit the community by finding other work.

Your background is irrelevant.  If you want to learn programming then
learn programming.  Web applications are not a specialty - they can be
written in almost any language (though I personally wouldn't pick
assembler).  Pick a language and start writing.  If you don't like the
result, try something else.  Continually asking questions about which
language/library/system is best leveraged for some particular task is
pointless until you gain some experience.

George
--
for email reply remove "/" from address
From: Pascal Bourguignon
Subject: Re: is a relational db a god tool or not really worth it?
Date: 
Message-ID: <871wkl5hjl.fsf@thalassa.informatimago.com>
George Neuner <·········@comcast.net> writes:

> [...]
> You frequently post little "what about this" and "compare and contrast
> these for me" types of questions.  You may think you are innocently
> soliciting opinions from people who have used [whatever the topic is]
> but the impression you leave is that either you are unable to research
> things on your own - which makes you no friends in c.l.l - or that you
> are deliberately trying to cause friction between their advocates.

And the point of an opinion is to be personal: don't even try to copy
the others' opinion, just have  your own!  That means, read and try
YOURSELF the items you ask about, and make up your OWN opinion.


> [...]
> Your background is irrelevant.  If you want to learn programming then
> learn programming.  Web applications are not a specialty - they can be
> written in almost any language (though I personally wouldn't pick
> assembler).  Pick a language and start writing.  If you don't like the
> result, try something else.  Continually asking questions about which
> language/library/system is best leveraged for some particular task is
> pointless until you gain some experience.
>
> George
> --
> for email reply remove "/" from address

-- 
__Pascal Bourguignon__                     http://www.informatimago.com/

ATTENTION: Despite any other listing of product contents found
herein, the consumer is advised that, in actuality, this product
consists of 99.9999999999% empty space.
From: Chris Parker
Subject: Re: is a relational db a god tool or not really worth it?
Date: 
Message-ID: <1172073824.969104.207890@p10g2000cwp.googlegroups.com>
> 3) Graham's decision not to use a database for Viaweb is neither a
> condemnation of databases in general nor a negative opinion on the use
> of databases for web applications - it was an opinion that a database
> was not needed for that particular project.  Another developer with
> Graham's experience may well have decided the opposite.
>

I always got the impression that PG didn't really care for relational
databases, and wanted a lisp-only solution.

I disagree with PG on this.  I am not a big fan of relational db
theory, but it is well-understood and most databases now days are
actual database servers and use SQL, which means getting the data out
is really easy.
From: Vassil Nikolov
Subject: Re: is a relational db a god tool or not really worth it?
Date: 
Message-ID: <yy8vfy8yd9it.fsf@eskimo.com>
On 21 Feb 2007 08:03:45 -0800, "Chris Parker" <··········@gmail.com> said:
| ...
| but it is well-understood and most databases now days are
| actual database servers and use SQL, which means getting the data out
| is really easy.

  Hmmm...  Define "really easy"...

  ---Vassil.

-- 
Our programs do not have bugs; it is just that the users' expectations
differ from the way they are implemented.
From: Chris Parker
Subject: Re: is a relational db a god tool or not really worth it?
Date: 
Message-ID: <1172443849.247350.309630@p10g2000cwp.googlegroups.com>
On Feb 21, 11:52 pm, Vassil Nikolov <···············@pobox.com> wrote:
> On 21 Feb 2007 08:03:45 -0800, "Chris Parker" <··········@gmail.com> said:
> | ...
> | but it is well-understood and most databases now days are
> | actual database servers and use SQL, which means getting the data out
> | is really easy.
>
>   Hmmm...  Define "really easy"...
>
>   ---Vassil.
>
> --
> Our programs do not have bugs; it is just that the users' expectations
> differ from the way they are implemented.


"SELECT * FROM foo" is a lot easier than parsing plists in most
languages.  Unless you know that you are going to be working only in
lisp ever, a SQL database is a darn good way to store data.
From: Vassil Nikolov
Subject: Re: is a relational db a god tool or not really worth it?
Date: 
Message-ID: <yy8v649nm11b.fsf@eskimo.com>
On 25 Feb 2007 14:50:49 -0800, "Chris Parker" <··········@gmail.com> said:

| On Feb 21, 11:52 pm, Vassil Nikolov <···············@pobox.com> wrote:
|| On 21 Feb 2007 08:03:45 -0800, "Chris Parker" <··········@gmail.com> said:
|| | ...
|| | but it is well-understood and most databases now days are
|| | actual database servers and use SQL, which means getting the data out
|| | is really easy.
|| 
|| Hmmm...  Define "really easy"...


| "SELECT * FROM foo" is a lot easier than parsing plists in most
| languages.  Unless you know that you are going to be working only in
| lisp ever, a SQL database is a darn good way to store data.

  But "easier" is not always "easy" (I don't know what there is to
  "parsing plists", but that is probably a different matter).

  Let's see, what do we need in order to get the data out of a database:

  For the more interesting cases, we need to authenticate ourselves
  before the database.  Do we use a password, and where do we keep it
  and generally how do we manage it?  (Compare to file access, where
  this comes "for free".)

  Then we probably don't want to create a fresh connection every time
  we access the database, so we either need to implement a connection
  pool, or have someone do that for us.  Again, this is something that
  does not apply to file access.

  I am not saying that accessing databases is awfully hard, but it
  isn't as easy as it might appear to be, especially compared to
  accessing files.

  ---Vassil.


-- 
mind mate, n.
  One of two persons mentally compatible with each other (cf. soul mate).
From: Kirk  Sluder
Subject: Re: is a relational db a god tool or not really worth it?
Date: 
Message-ID: <kirk-2F9250.12464928022007@newsclstr03.news.prodigy.net>
In article <················@eskimo.com>,
 Vassil Nikolov <···············@pobox.com> wrote:

>   But "easier" is not always "easy" (I don't know what there is to
>   "parsing plists", but that is probably a different matter).
> 
>   Let's see, what do we need in order to get the data out of a database:
> 
>   For the more interesting cases, we need to authenticate ourselves
>   before the database.  Do we use a password, and where do we keep it
>   and generally how do we manage it?  (Compare to file access, where
>   this comes "for free".)

It's unfortunate how "database" has come to mean, "a server process 
that communicates with other server processes using SQL." But I'm in 
full agreement that in many cases it seems like the design process 
runs along the line of "MySQL is the solution, now what is your 
problem?"

Certainly, SQL servers are great solutions for many problems, but 
there are a lot of micro-sized "database" problems where you don't 
need to worry about multiple clients trying to modify the same set 
of data.  

>   ---Vassil.
From: George Neuner
Subject: Re: is a relational db a god tool or not really worth it?
Date: 
Message-ID: <h53qt2p51cn65uibdl8pk3hs49v0enrkva@4ax.com>
On 21 Feb 2007 08:03:45 -0800, "Chris Parker" <··········@gmail.com>
wrote:

>> 3) Graham's decision not to use a database for Viaweb is neither a
>> condemnation of databases in general nor a negative opinion on the use
>> of databases for web applications - it was an opinion that a database
>> was not needed for that particular project.  Another developer with
>> Graham's experience may well have decided the opposite.
>>
>
>I always got the impression that PG didn't really care for relational
>databases, and wanted a lisp-only solution.

I agree, but the OP is a newbie who lacks the experience to evaluate
Graham's opinions in context.  The OP read that Graham didn't use a
DBM for Viaweb and inferred from it that Lisp might somehow make DBMs
superfluous.

I sought to point out that fame doesn't make Graham's opinions correct
- it only makes them well known.  What he chose to do for his project
should not necessarily guide anyone else.


>I disagree with PG on this.  I am not a big fan of relational db
>theory, but it is well-understood and most databases now days are
>actual database servers and use SQL, which means getting the data out
>is really easy.

My MS was in database technology.  Theory notwithstanding, I have a
lot of practical problems with the relational model.

It's difficult to fit many storage problems easily into RM - the
"relations" needed for decent performance do not fall naturally out of
normalization.  A great many projects end up using redundancy and
internal row dependencies to get performance and pay the price in
query complexity and difficulty coordinating updates.  And don't get
me started on the problems with SQL as a language.  But there aren't
too many options nowadays for medium to large scale DB projects - the
relational model has essentially taken over.

From what I have read, ISTM that Viaweb might have been a pretty good
fit for a Codasyl database, but at the time Graham developed Viaweb,
Codasyl was all but extinct.

There have been a lot of proposals for new OO and functional models
but none have gained any traction and many of the proposed models are
easily cast as extensions of Codasyl while being much harder to fit to
RM.  RM has the theory behind it but the conceptual ease with which
Codasyl fits so many real world problems makes me wonder if abandoning
it wasn't naive.

We now return to your regularly scheduled c.l.l

George
--
for email reply remove "/" from address
From: Chris Parker
Subject: Re: is a relational db a god tool or not really worth it?
Date: 
Message-ID: <1172443661.920293.326370@s48g2000cws.googlegroups.com>
>
> My MS was in database technology.  Theory notwithstanding, I have a
> lot of practical problems with the relational model.
>

I used to work with one of the original relational db guys from IBM,
and he had huge problems with relational db theory.
From: George Neuner
Subject: Re: is a relational db a god tool or not really worth it?
Date: 
Message-ID: <m725u2pd38h2e3jca7kom5dee857ru95k2@4ax.com>
On 25 Feb 2007 14:47:42 -0800, "Chris Parker" <··········@gmail.com>
wrote:

>
>>
>> My MS was in database technology.  Theory notwithstanding, I have a
>> lot of practical problems with the relational model.
>>
>
>I used to work with one of the original relational db guys from IBM,
>and he had huge problems with relational db theory.
>

Many of the expressed criticisms of RM were not problems with the
theory per se, but rather dissatisfaction with SQL.  SQL implements
Codd's relational algebra (though simple things like set difference
are hard to express) - researchers generally preferred the more
expressive relational calculus.  Also many people felt that SQL was
too procedural in many respects.  The language QUEL, while also too
procedural, is closer to what Codd originally intended with his
unrealized "Alpha" query language.

Most of the actual theoretical criticism of the relational model has
to do with its (lack of) treatment of nulls.  Although the model
specifically allows for nulls (ie. missing data), neither the algebra
nor calculus specifies what to do with them.  Admitting nulls clashes 
with the requirement that all relation attributes have a value,
results in fuzzy comparisons and leaves unspecified the results of
operations like set product and difference.  Without nulls, the model
is sound - with them it has a big hole in it.  Codd tried to integrate
nulls through extensions of the entity model and by defining different
multi-state logics to deal with them, but few researchers were happy
with what he came up with.  And no one was happy with SQL's ad hoc
treatment of model nulls nor that it used null for unrelated purposes
(there are currently about 20 different uses of null in SQL).

Fabian Pascal has supposedly come up with a treatment of nulls as meta
data which avoids multi-valued logic problems by separating the
notions of "missing" and "not applicable".  I haven't seen his paper
yet so I can't comment on his solution.


Anyway - we've wandered far off topic.

George
--
for email reply remove "/" from address
From: Christopher Browne
Subject: Re: is a relational db a god tool or not really worth it?
Date: 
Message-ID: <87irdoim0y.fsf@wolfe.cbbrowne.com>
Centuries ago, Nostradamus foresaw when George Neuner <·········@comcast.net> would write:
> Fabian Pascal has supposedly come up with a treatment of nulls as meta
> data which avoids multi-valued logic problems by separating the
> notions of "missing" and "not applicable".  I haven't seen his paper
> yet so I can't comment on his solution.

The "authoritative" paper on this seems to be this one by Hugh Darwen:
<http://web.onetel.com/~hughdarwen/TheThirdManifesto/Missing-info-without-nulls.pdf>

What it essentially does is to decompose the data model further, so
that you might have some additional tables, and use an equivalent to
an outer join in order to assemble queries that indicate various sorts
of 'something missing.'
-- 
let name="cbbrowne" and tld="cbbrowne.com" in name ^ ·@" ^ tld;;
http://linuxdatabases.info/info/slony.html
"By golly,  I'm beginning  to think Linux  really *is* the  best thing
since sliced bread." -- Vance Petree, Virginia Power
From: George Neuner
Subject: Re: is a relational db a god tool or not really worth it?
Date: 
Message-ID: <i6g7u2psdvq4411ct5d3uj46gouupcdutd@4ax.com>
On Mon, 26 Feb 2007 17:36:45 -0500, Christopher Browne
<········@acm.org> wrote:

>Centuries ago, Nostradamus foresaw when George Neuner <·········@comcast.net> would write:
>> Fabian Pascal has supposedly come up with a treatment of nulls as meta
>> data which avoids multi-valued logic problems by separating the
>> notions of "missing" and "not applicable".  I haven't seen his paper
>> yet so I can't comment on his solution.
>
>The "authoritative" paper on this seems to be this one by Hugh Darwen:
><http://web.onetel.com/~hughdarwen/TheThirdManifesto/Missing-info-without-nulls.pdf>
>
>What it essentially does is to decompose the data model further, so
>that you might have some additional tables, and use an equivalent to
>an outer join in order to assemble queries that indicate various sorts
>of 'something missing.'

Thanks for the pointer.

George
--
for email reply remove "/" from address
From: Alan Crowe
Subject: Re: is a relational db a god tool or not really worth it?
Date: 
Message-ID: <86r6sbrbjm.fsf@cawtech.freeserve.co.uk>
Christopher Browne <········@acm.org> writes:
> 
> The "authoritative" paper on this seems to be this one by Hugh Darwen:
> <http://web.onetel.com/~hughdarwen/TheThirdManifesto/Missing-info-without-nulls.pdf>
> 
> What it essentially does is to decompose the data model further, so
> that you might have some additional tables, and use an equivalent to
> an outer join in order to assemble queries that indicate various sorts
> of 'something missing.'

Thanks for the pointer. It is an interesting read. I cannot
help thinking that he goes easy on the evils on NULL.

I tried thinking about this and contemplated an
anthropologist building a database of family relationships
in a wild tribe he is studying. He tries to build a table

(person mother father)

but 2/3's of the fathers are missing. His geneticist
colleague still insists that one needs to know whether men
father two children each or some four and others zero.

How should the anthropologist treat NULL? If NULL = NULL he
is a magic father with lots of children. If NULL =/= NULL
there are more fathers than men.

There are two problems. 

The epistemological problem is that the anthropologist
doesn't have the data he needs. He will have to fill in the
blanks in various ways and attempt to pin down the range of
conclusions that are consistent with the data he does have.

The teleological problem is that the point of a database is
to automate processing. When you let NULLs into your
database you are leave manual work for later. Either you grovel
about manually to ensure that the same NULLs get different
interpretations in different queries or you tolerate wrong
answers. Either way you are abandoning your goal of
automatic processing.

Alan Crowe
Edinburgh
Scotland
From: Christopher Browne
Subject: Re: is a relational db a god tool or not really worth it?
Date: 
Message-ID: <87abywi4yd.fsf@wolfe.cbbrowne.com>
Centuries ago, Nostradamus foresaw when Alan Crowe <····@cawtech.freeserve.co.uk> would write:
> Christopher Browne <········@acm.org> writes:
>> 
>> The "authoritative" paper on this seems to be this one by Hugh Darwen:
>> <http://web.onetel.com/~hughdarwen/TheThirdManifesto/Missing-info-without-nulls.pdf>
>> 
>> What it essentially does is to decompose the data model further, so
>> that you might have some additional tables, and use an equivalent to
>> an outer join in order to assemble queries that indicate various sorts
>> of 'something missing.'
>
> Thanks for the pointer. It is an interesting read. I cannot
> help thinking that he goes easy on the evils on NULL.

Yes, but it seems to me that this "cure" is almost as bad as the
disease.

NULLs have their problems; everyone that has used SQL has doubtless
had some unfortunate surprises as a result of the unexpected
introduction of NULLs.

But if you have to create a vastly more complex model to specifically
contemplate the possible bits of missing information, queries will get
hairier because you have more relations.  They may be somewhat simpler
to understand, individually, than queries in a "nullable" world, but
if the model's 8 times more complex, this doesn't seem like much of a
win to me...

> Either way you are abandoning your goal of automatic processing.

That's certainly the root of the problem.  NULLs allow you to create a
somewhat simpler model than you can build if you are forced into
circumlocutions in order to evade them.

Having to "special case" things is an abandonment of automatic
processing.

The approach I tend to take is to try to avoid NULLs when possible.
And if having them makes the model otherwise a lot simpler, then maybe
they're providing some kind of value (despite being null :-)).
-- 
wm(X,Y):-write(X),write(·@'),write(Y). wm('cbbrowne','gmail.com').
http://cbbrowne.com/info/postgresql.html
A  student,  in hopes  of  understanding  the  Lambda-nature, came  to
Greenblatt.  As they spoke a  Multics system hacker walked by.  "Is it
true", asked the  student, "that PL-1 has many of  the same data types
as  Lisp?"   Almost before  the  student  had  finished his  question,
Greenblatt shouted, "FOO!", and hit the student with a stick.
From: Stefan Nobis
Subject: Re: is a relational db a god tool or not really worth it?
Date: 
Message-ID: <871wk8gd2j.fsf@snobis.de>
Christopher Browne <········@acm.org> writes:

> But if you have to create a vastly more complex model to
> specifically contemplate the possible bits of missing information,
> queries will get hairier because you have more relations.

One of the points of the third manifesto is exactly this: SQL and
todays RDBMS are just too far away from the relational algebra. In
fact with SQL it's just impossible to create (true) relations but you
get mere tables.

So in a good world with good products, which are implementations of
the relational model, and a good query language inspired for example
by Tutorial D, all these problems will vanish.

I'm far too inexperienced to judge about these issues, but that's how
I understand what I've read from Date and Darwen until now.

> That's certainly the root of the problem.  NULLs allow you to create
> a somewhat simpler model than you can build if you are forced into
> circumlocutions in order to evade them.

> Having to "special case" things is an abandonment of automatic
> processing.

Yes, but with NULLs you are just pushing thinking and handling
problems from the model to the application layer. So no real win in
using NULLs. In most (all?) examples I've seen so far, it would've
been quite easy to throw out all NULLs without any new table (or only
very few new tables).

-- 
Stefan.
From: Christopher Browne
Subject: Re: is a relational db a god tool or not really worth it?
Date: 
Message-ID: <87y7mfhhue.fsf@wolfe.cbbrowne.com>
Martha Stewart called it a Good Thing when Stefan Nobis <······@gmx.de> wrote:
> Christopher Browne <········@acm.org> writes:
>
>> But if you have to create a vastly more complex model to
>> specifically contemplate the possible bits of missing information,
>> queries will get hairier because you have more relations.
>
> One of the points of the third manifesto is exactly this: SQL and
> todays RDBMS are just too far away from the relational algebra. In
> fact with SQL it's just impossible to create (true) relations but you
> get mere tables.
>
> So in a good world with good products, which are implementations of
> the relational model, and a good query language inspired for example
> by Tutorial D, all these problems will vanish.
>
> I'm far too inexperienced to judge about these issues, but that's how
> I understand what I've read from Date and Darwen until now.

Well, in Lisp, the fact that you generally don't have any of the
pointer problems found in C/C++ eliminates pretty well the whole set
of bugs resulting from accessing invalid bits of memory.

But that has NOT led to Lisp having no bugs; it has facilities that
permit having *new* kinds of bugs previously unavailable to
programmers.

The belief that "all problems will vanish" when we migrate to
"Tutorial D" or "Production D" seems like a willingness to drink
whatever koolaid may be offered ...

>> That's certainly the root of the problem.  NULLs allow you to create
>> a somewhat simpler model than you can build if you are forced into
>> circumlocutions in order to evade them.
>
>> Having to "special case" things is an abandonment of automatic
>> processing.
>
> Yes, but with NULLs you are just pushing thinking and handling
> problems from the model to the application layer. So no real win in
> using NULLs. In most (all?) examples I've seen so far, it would've
> been quite easy to throw out all NULLs without any new table (or only
> very few new tables).

Most of the NULLs I see I don't like.  But that's not the same as
"all"...

(string-equal "Most" "All")
NIL
-- 
let name="cbbrowne" and tld="gmail.com" in String.concat ·@" [name;tld];;
http://linuxdatabases.info/info/wp.html
"You can measure a programmer's perspective by noting his attitude on
the continuing vitality of FORTRAN." -- Alan Perlis
From: Kirk  Sluder
Subject: Re: is a relational db a god tool or not really worth it?
Date: 
Message-ID: <kirk-2E5E34.11581319022007@newsclstr02.news.prodigy.com>
> Is a (relational) database not needed for lisp webapps? 

Well, a database is simply any system that stores data.  What form 
of database you use depends on your needs.  For a small quantity of 
data, config files are enough. Mail systems tend to use filesystem 
databases. Wikis have been built using version control systems as 
back-ends.  XML files are another possibility, and so are simple 
Berkeley-style databases. 

> Can one get a fast app without one? 

Depends on your needs and what kind of application.  For cases where 
you have a small amount of data, keeping it all in memory by loading 
it as an image file may be better than using an external process.

> What is the tradeoff? 

Depends on the application, and your server needs.  On the one hand, 
SQL databases such as mySQL and postgresql provide a lot of 
functions and ways to deal with limited memory that you don't have 
to build yourself.  On the other hand, you usually end up talking to 
an external process, that has a bit of maintenance load.  A 
"middle-way" alternative is something like sqlite which provides an 
SQL interface to structures loaded from a file and stored in memory.  



sqlite saves quite a bit on inter-process communication, but can 
bloat for large datasets.
> It seems most web setups include a db as a matter of course such as php 
> +mysql, or ruby on rails etc. 

Yes, most languages include support for multiple forms of data 
persistence, including SQL databases.
From: Alex Mizrahi
Subject: Re: is a relational db a god tool or not really worth it?
Date: 
Message-ID: <45d9d485$0$90267$14726298@news.sunsite.dk>
(message (Hello 'gavino)
(you :wrote  :on '(19 Feb 2007 00:39:08 -0800))
(

 g> http://www.paulgraham.com/vwfaq.html Paul Graham states here that he
 g> did not use a relational or other database in his work.

 g> Is a database not needed for lisp webapps?

i think triple store is much better.

http://dannyayers.com/2004/12/30/are-triplestores-good-databases/

 g> Can one get a fast app without one?

i think AllegroGraph is very fast:

http://www.franz.com/products/allegrograph/

)
(With-best-regards '(Alex Mizrahi) :aka 'killer_storm)
"?? ???? ??????? ?????") 
From: baalbek
Subject: Re: is a relational db a god tool or not really worth it?
Date: 
Message-ID: <45df8b71$0$5391$c83e3ef6@nn1-read.tele2.net>
gavino wrote:
> http://www.paulgraham.com/vwfaq.html Paul Graham states here that he
> did not use a relational or other database in his work.
> 
> Is a database not needed for lisp webapps?
> Can one get a fast app without one?
> What is the tradeoff?
> It seems most web setups include a db as a matter of course such as php
> +mysql, or ruby on rails etc.
> 
> What are the pros and cons and once one starts to scale does a
> relational db become mandatory?
> 

You appear as extremely gullible, or as a neophyte just taking on some 
"computer" work (for your father-in-law?).

Whatever, I doubt you are a professional, as not having any knowledge of 
relational databases today as a professional is highly unlikely...

But whatever, yes, relational databases ARE worth it, until we get 
object databases that works adequately, notwithstanding this Paul Graham 
dude's web-app-with-no-database!

baalbek
From: Tim Bradshaw
Subject: Re: is a relational db a god tool or not really worth it?
Date: 
Message-ID: <1172491815.978871.37130@h3g2000cwc.googlegroups.com>
On Feb 24, 12:48 am, baalbek <····@bgoark.no> wrote:

>
> But whatever, yes, relational databases ARE worth it, until we get
> object databases that works adequately, notwithstanding this Paul Graham
> dude's web-app-with-no-database!

They *may* be worth it.  I think everyone has worked somewhere where
there were batch jobs which sucked all the data out of some relational
database every hour into flat files to drive the search engines.  Of
course, this is often because of dreadful schema design in the
database.  But often it's because the things that a serious RDBMS buys
you (ACIDity is the big one) just turned out not to matter.

I mean, what's the biggest web app in the world?  Do they  use a
relational database to drive it?  I bet they don't.

--tim
From: Christopher Browne
Subject: Re: is a relational db a god tool or not really worth it?
Date: 
Message-ID: <87odnhhxx8.fsf@wolfe.cbbrowne.com>
In an attempt to throw the authorities off his trail, baalbek <···@bgoark.no> transmitted:
> gavino wrote:
>> http://www.paulgraham.com/vwfaq.html Paul Graham states here that he
>> did not use a relational or other database in his work.
>> Is a database not needed for lisp webapps?
>> Can one get a fast app without one?
>> What is the tradeoff?
>> It seems most web setups include a db as a matter of course such as php
>> +mysql, or ruby on rails etc.
>> What are the pros and cons and once one starts to scale does a
>> relational db become mandatory?
>
> You appear as extremely gullible, or as a neophyte just taking on some
> "computer" work (for your father-in-law?).
>
> Whatever, I doubt you are a professional, as not having any knowledge
> of relational databases today as a professional is highly unlikely...
>
> But whatever, yes, relational databases ARE worth it, until we get
> object databases that works adequately, notwithstanding this Paul
> Graham dude's web-app-with-no-database!

You can get away with other approaches if the application has no need
for coordinated concurrent updates.

And it's possible to design applications to minimize the need for that
sort of thing.

But at some point, there is a need to have multiple users doing
concurrent updates where you *can't* just let them blindly update the
data, because they may be trampling on one another's data.  At that
point, you need the concurrency control that a "real database system"
provides, and creating that from scratch would be expensive and
(particularly if this is a "bag on the side") exceedingly error-prone.
-- 
(reverse (concatenate 'string "ofni.secnanifxunil" ·@" "enworbbc"))
http://linuxfinances.info/info/slony.html
Rules  of the  Evil Overlord  #26. "No  matter how  attractive certain
members  of the  rebellion  are,  there is  probably  someone just  as
attractive who  is not desperate to  kill me. Therefore,  I will think
twice before ordering a prisoner sent to my bedchamber."
<http://www.eviloverlord.com/>