Howdy Folks!
I'm not at all saddened to say that I'm leaving Lisp land for Erlang. I
wrote a complete implementation of my poker engine in Lisp but hit a
roadblock trying to write a scalable server. I thoroughly enjoyed my
time with Lisp but I don't think it's the universal hammer. I'm a
pragmatist and Erlang solves my problems far better than Lisp.
Here are my reasons, please flame away!
http://wagerlabs.com/2005/04/how-to-compete-with-big-boys.html
Thanks, Joel
well, congratulations on finding such a scalable language. I checked
it out and can't say I'd follow you, though it makes sense for your
application. The syntax seems pretty obtuse. I write applications
that similarly must be scalable- but I found postgres replication
pretty straightforward, allowing me to stick with the pristine beauty
of Lisp (natively multithreaded with sbcl). Anyway, you're doing what
they say, I suppose- using the language that best fits your needs.
Joseph
"Joseph Wecker" <···············@emcop.com> writes:
> well, congratulations on finding such a scalable language. I checked
> it out and can't say I'd follow you, though it makes sense for your
> application. The syntax seems pretty obtuse. I write applications
> that similarly must be scalable- but I found postgres replication
> pretty straightforward, allowing me to stick with the pristine beauty
> of Lisp (natively multithreaded with sbcl).
If it's not proprietary info, would you care to say a few words
about how you use postgres replication, and what problems it solves
for you? AFAIK, at the moment, there is only single master -> multiple
slaves replication for postgres, so I'm imagining you have a fairly
specific problem where that's adequate, but, nonetheless, I'd be
very interested to read about your experiences.
Thanks!
Alain,
> If it's not proprietary info, would you care to say a few words
> about how you use postgres replication, and what problems it solves
> for you?
I'm not using PostgreSQL at the moment. There are a few significant
replication threads on the net, though, and I got the idea that it's
not a panacea.
Alain Picard wrote:
> for you? AFAIK, at the moment, there is only single master -> multiple
> slaves replication for postgres, so I'm imagining you have a fairly
Have you seen PGCluster ? http://pgfoundry.org/projects/pgcluster/
From front-page: "PGCluster is the synchronous replication system of
the multi-master composition for PostgreSQL"
Svenne
Svenne Krap <············@krap.dk> writes:
>
> Have you seen PGCluster ? http://pgfoundry.org/projects/pgcluster/
Thanks, I'll look at that. My impression was that SLONY was
the currently best/most stable option for replication, but
I'll have a look at this.
Clinging to sanity, Alain Picard <············@memetrics.com> mumbled into her beard:
> Svenne Krap <············@krap.dk> writes:
>> Have you seen PGCluster ? http://pgfoundry.org/projects/pgcluster/
>
> Thanks, I'll look at that. My impression was that SLONY was
> the currently best/most stable option for replication, but
> I'll have a look at this.
Slony-I is only a single-master (to multiple slaves) system.
Certainly getting a lot more development attention than PGCluster...
Slony-II, which is in hopefully coming-towards-sorta-working stages of
prototyping, is to be a multi-master system.
I suspect that even it won't be an alternative to Erlang/Mnesia for
the OP's sort of application in view of the fact that Mnesia was
designed with replication as the basic underpinning, as opposed to as
an add-on.
--
(reverse (concatenate 'string "moc.liamg" ·@" "enworbbc"))
http://linuxdatabases.info/info/slony.html
"The most precisely-explained and voluminously-documented user
interface "rule" can and will be shot to pieces with the introduction
of a single new priority consideration." -- Michael Peck
Hey, np
We're actually using Slony. Joel is right- it's not a panacea.
Though, of course, I don't know that clustering would be either. For a
nice little description of the difference between replication and
clustering, btw, see http://brianray.chipy.org/postgreSQL .
The problem it solves so far: redundancy. The problem it doesn't solve
is: we're getting another few hundred customers- slap another box in
and watch it expand to handle the new requests...
I'm working on tweaking it a bit, though, and with the right
configuration, it also solves the problem of seamless database upgrades
(both Postgres and our actual databases)- just let one of the slaves
take over, switch a second slave to be a slave of the first slave,
upgrade the first server, and go around the chain.
Really with scalability we're talking about a number of different
technical issues with different solutions. We have threading, so we
have concurrency, we have replication, modules, access to running lisp
processes etc., and with the backend being able to do the same, we have
seamless modifications, we have redundancy- and as far as the language
constructs that make it easy to do all that... well, I really like what
erlisp is doing. Also keep in mind this little project-
http://www.iro.umontreal.ca/~etos/ - "ETOS is an efficient
implementation of Erlang. The ETOS compiler transforms Erlang programs
into Scheme programs." So, obviously, it's just a matter of someone
putting some work into it- not some limitation of the language [again,
obviously- that's our religion, right :) ]
That's just a quick summary- hope it helps :) Slony has a great
community and great docs, so that can get the creative juices flowing,
too.
-Joseph
>>>>> "Joseph" == Joseph Wecker <···············@emcop.com> writes:
Joseph> Also keep in mind this little project-
Joseph> http://www.iro.umontreal.ca/~etos/ - "ETOS is an efficient
Joseph> implementation of Erlang. The ETOS compiler transforms Erlang
Joseph> programs into Scheme programs."
One thing Erlang does well is massive parallelism. I heard about an
experiment to find the upper limit by pumping requests into a
webserver that would handle each request in a separate. They gave up
trying to break it after reaching several hundred thousand
processes. Competing setups using windows and unix threading had
collapsed much before that.
As far as I know, ETOS focuses on sequential programs, ie. not so much
on very many parallel processes, and this is also not something you
can easily recover as part of translation. If the underlying VM does
not do threading very cheaply, you will have difficulties in competing
with Erlangs multiprocessing.
That being said, the original article describing ETOS claimed that for
the benchmarks that was included, ETOS generated scheme code actually
ran faster than Erlang in many cases, suggesting that allthough Erlang
does multiple processes well, it is not blindingly fast otherwise. The
origianl article is some years old now and thus the Erlang compiler
may have caught up. On the other hand, ETOS is also still being
developed :-)
I do a fair bit of Erlang programming myself. Being a die-hard lisp
fanatic, I find that for a language as advanced and dynamic as Erlang
it falls just a bit short of lisp in many respects. For instance,
Erlang has a concept of records, but they work through compiler magic
so you can not just copy and paste code involving records into the
REPL as a lisp programmer can. Erlang is much better than C or Java
but it is nowhere near being Lisp.
------------------------+-----------------------------------------------------
Christian Lynbech | christian ··@ defun #\. dk
------------------------+-----------------------------------------------------
Hit the philistines three times over the head with the Elisp reference manual.
- ·······@hal.com (Michael A. Petonic)
Joel Reymont wrote:
> Howdy Folks!
>
> I'm not at all saddened to say that I'm leaving Lisp land for Erlang.
> I wrote a complete implementation of my poker engine in Lisp but hit
> a roadblock trying to write a scalable server. I thoroughly enjoyed
> my time with Lisp but I don't think it's the universal hammer. I'm a
> pragmatist and Erlang solves my problems far better than Lisp.
>
> Here are my reasons, please flame away!
>
> http://wagerlabs.com/2005/04/how-to-compete-with-big-boys.html
Like-minded people are interested in merging the two tools. Perhaps
you'll be able to return in the future...
http://www.dirkgerrits.com/erlisp/
We seem to be in a low valley as far as parallel Lisps go. Not that I'm
familiar with the topic though.
Joel Reymont wrote:
>Here are my reasons, please flame away!
>
>
Where is your ode? I was expecting an ode.
--
Cheers, www.indiegamedesign.com
Brandon Van Every Seattle, WA
"Troll" - (n.) Anything you don't like.
Usage: "He's just a troll."
Brandon J. Van Every wrote:
> Joel Reymont wrote:
>
>> Here are my reasons, please flame away!
>>
>>
> Where is your ode? I was expecting an ode.
>
Alright, fine, I will help you get started:
Erlang, O, Erlang
You are such a better lang
You entered my life with
A crash and a bang
[Now others please continue, or modify at will.]
--
Cheers, www.indiegamedesign.com
Brandon Van Every Seattle, WA
20% of the world is real.
80% is gobbledygook we make up inside our own heads.
Brandon J. Van Every wrote:
> Brandon J. Van Every wrote:
>
>> Joel Reymont wrote:
>>
>>> Here are my reasons, please flame away!
>>>
>>>
>> Where is your ode? I was expecting an ode.
>>
> Alright, fine, I will help you get started:
>
> Erlang, O, Erlang
> You are such a better lang
> You entered my life with
> A crash and a bang
Isn't it a hash and a bang?
#!/usr/bin/erlang maybe
Well, I've never tried Erlang. Might in the future, but now it's
Lisp's turn.
>
> [Now others please continue, or modify at will.]
>
--
No man is good enough to govern another man without that other's
consent. -- Abraham Lincoln
O, Erlang
You have more threads*
Than a baker has bread.
True computing distributed
You have contributed,
And Mnesia is smartsa.
O, Erlang
You have the bling
But also the blang --
But for unmutable structures
You'd have me captured.
*Poetic license, as Erlang has processes.
gk
"Brandon J. Van Every" <·····················@mycompanyname.com> wrote in message news:<···················@newsread1.news.atl.earthlink.net>...
> Brandon J. Van Every wrote:
>
> > Joel Reymont wrote:
> >
> >> Here are my reasons, please flame away!
> >>
> >>
> > Where is your ode? I was expecting an ode.
> >
> Alright, fine, I will help you get started:
>
> Erlang, O, Erlang
> You are such a better lang
> You entered my life with
> A crash and a bang
>
>
> [Now others please continue, or modify at will.]
Joel Reymont wrote:
> Howdy Folks!
>
> I'm not at all saddened to say that I'm leaving Lisp land for Erlang. I
> wrote a complete implementation of my poker engine in Lisp but hit a
> roadblock trying to write a scalable server. I thoroughly enjoyed my
> time with Lisp but I don't think it's the universal hammer. I'm a
> pragmatist and Erlang solves my problems far better than Lisp.
>
> Here are my reasons, please flame away!
>
Have you wrote the erlang version already?
Wade
On Tue, 19 Apr 2005 05:22:28 -0700, Joel Reymont wrote:
> Howdy Folks!
>
> I'm not at all saddened to say that I'm leaving Lisp land for Erlang. I
> wrote a complete implementation of my poker engine in Lisp but hit a
> roadblock trying to write a scalable server. I thoroughly enjoyed my
> time with Lisp but I don't think it's the universal hammer. I'm a
> pragmatist and Erlang solves my problems far better than Lisp.
From your writeup I got the impression that Erlang looks likely to solve
your server problems, not that it has. You're in the cheerleading stage.
I took a cursory look at Erlang one of the previous times the language was
discussed in this forum and my conclusion was and is that the lack of
mutable arrays makes it a hopeless candidate for a decent general purpose
language.
<http://groups.google.co.nz/groups?threadm=3ADC4207.D32D8C4C%40cse.uconn.edu>
This FAQ entry sets out problems Erlang is unsuitable for:
<http://erlang.org/faq/t1.html#AEN43>
Anything where constant-factors are crucial for performance, such as
image processing, signal processing, sorting large volumes of data,
device drivers and low-level protocol termination.
Most (all?) large systems developed using Erlang make heavy use of C
for low-level code, leaving Erlang to manage the parts which tend to be
complex in other languages, like controlling systems spread across
several machines and implementing complex protocol logic.
Do you know what Lisp is unsuitable for? Nothing whatsoever. While there
may be practical reasons to interface with C there is no design limitation
of the language that makes this necessary for efficiency. Your poker
server may not initially hit the limitations of Erlang but you may hit
them eventually (and you may find ways to work around them).
<http://groups.google.co.nz/groups?selm=nid6e73jk7.fsf%40rian.du.uab.ericsson.se&output=gplain>
We have a binary type in erlang which is a reference counted blob of
data, which works well.
Two problems, the first is that erlang data is non-mutable (like
Java's Strings) i.e. you can't update the damn binary anyway (*) so
for dynamic data it's rather expensive to have large arrays, which
needs to be rebuilt when anything changes.
[...]
(*) Ok, we can provide an api for doing that but thats against the
functional philosophy, it will probably happen anyway since we already
let opengl write to the binary in calls like glReadPixels.
> Here are my reasons, please flame away!
>
> http://wagerlabs.com/2005/04/how-to-compete-with-big-boys.html
You conclude with "And did I mention that Erlang is free?" Erlang is more
encumbered than the public domain/permissively licensed implementations
available for Unix-like operating systems: <http://erlang.org/EPLICENSE>
[As you state that the client will "most likely written in Python and
using the PyGame API" there is no reason to limit yourself to
implementations that run upon Windows].
I would typically wish you the best of success but I'd rather it not be
built upon a foundation of gambling addiction.
Regards,
Adam
Adam Warner wrote:
> Do you know what Lisp is unsuitable for? Nothing whatsoever. While
there may be practical reasons to interface with C there is no design
limitation of the language that makes this necessary for efficiency.
Your poker server may not initially hit the limitations of Erlang but
you may hit
them eventually (and you may find ways to work around them).
I did intend to provoke a lively discussion so I did not emphasize that
I actually *love* Lisp :D. It's just that of the Erlang evils that you
described and the evils of writing clustering, etc. in Lisp I chose the
lesser ones.
Cheers, Joel
Adam Warner <······@consulting.net.nz> wrote in message news:<······························@consulting.net.nz>...
>
.....
>
> From your writeup I got the impression that Erlang looks likely to solve
> your server problems, not that it has. You're in the cheerleading stage.
>
> I took a cursory look at Erlang one of the previous times the language was
> discussed in this forum and my conclusion was and is that the lack of
> mutable arrays makes it a hopeless candidate for a decent general purpose
> language.
>
it may well be true that Erlang is not a general purpose language, but
Joel is looking for the right solution to a very specific problem
which doesn't even have very complex logic, so it may very well be the
best solution in his case.
-klaus.
On Wed, 20 Apr 2005 09:15:12 -0700, klaus wrote:
>> From your writeup I got the impression that Erlang looks likely to
>> solve your server problems, not that it has. You're in the cheerleading
>> stage.
>>
>> I took a cursory look at Erlang one of the previous times the language
>> was discussed in this forum and my conclusion was and is that the lack
>> of mutable arrays makes it a hopeless candidate for a decent general
>> purpose language.
>>
> it may well be true that Erlang is not a general purpose language, but
> Joel is looking for the right solution to a very specific problem which
> doesn't even have very complex logic, so it may very well be the best
> solution in his case.
Indeed. But Joel also states: "I thoroughly enjoyed my time with Lisp but
I don't think it's the universal hammer. I'm a pragmatist and Erlang
solves my problems far better than Lisp."
Lisp *is* a universal hammer and one can write a domain specific language
with Erlang-like qualities in Lisp while the converse is not true. Someone
who *loves* Lisp as much as Joel ("I actually *love* Lisp") Reymont could
find a practical way to make Lisp work in this server domain.
"...Please don't assume Lisp is only useful for Animation and Graphics,
AI, Bioinformatics, B2B and E-Commerce, Data Mining, EDA/Semiconductor
applications, Expert Systems, Finance, Intelligent Agents, Knowledge
Management, Mechanical CAD, Modeling and Simulation, Natural Language,
Optimization, Research, Risk Analysis, Scheduling, Telecom, and Web
Authoring just because these are the only things they happened to
list." - Kent M. Pitman
Does anyone have a citation for this now-famous quotation? I just pasted
it from one of the many places it's duplicated.
Regards,
Adam
Adam Warner wrote:
> On Wed, 20 Apr 2005 09:15:12 -0700, klaus wrote:
...
>
> Does anyone have a citation for this now-famous quotation? I just
pasted
> it from one of the many places it's duplicated.
>
> Regards,
> Adam
http://slashdot.org/interviews/01/11/03/1726251.shtml
On Wed, 20 Apr 2005 21:13:16 -0700, Olivier Drolet wrote:
>> Does anyone have a citation for this now-famous quotation? I just
>> pasted it from one of the many places it's duplicated.
>
> http://slashdot.org/interviews/01/11/03/1726251.shtml
Thanks Oliver. I'll have to keep a look out for the NetSettler pseudonym.
Regards,
Adam
actually, there is at least one thing that would be, I think, extremely
difficult for a DSL implemented in lisp to simulate, that erlang
handles nicely...super-lightweight threads. Erlang threads are much
lighter-weight than any native threads I am aware of, and lisp ones are
mostly native or heavier. I would love to be proven wrong on this,
though.
···········@gmail.com writes:
> actually, there is at least one thing that would be, I think, extremely
> difficult for a DSL implemented in lisp to simulate, that erlang
> handles nicely...super-lightweight threads. Erlang threads are much
> lighter-weight than any native threads I am aware of, and lisp ones are
> mostly native or heavier. I would love to be proven wrong on this,
> though.
Common Lisp does not any einclude multiprocessing facilities.
Most (or all?) Lisp implementations have super-lightweight threads,
which are implemented and scheduled inside Lisp itself.
Some Lisp implementations support native threads.
(However, most people would prefer native threads.)
Why you think lightweight threads would be hard to
implement this for Lisp is beyond my imagination.
······@news.dtpq.com (Christopher C. Stacy) writes:
> Why you think lightweight threads would be hard to
> implement this for Lisp is beyond my imagination.
Erlang doesn't have threads. Erlang has processes. Processes don't
share memory (heap objects), but they share code. Implementing
lightweight processes is a challenge in Lisp, because the Lisp coding
style relies heavily on global variables and mutation. E.g., a vanilla
CMUCL core has a about 1000 globally bound (non-constant) special
variables. How do you deal with those 1000 variables and say 500
processes? I doubt that you want to create a separate copy for each
process. Implementing some copy-on-write scheme is certainly tricky
and will most likely make access to global variables slower.
Erlang has also many other features like a module system, code
distribution, code updates, or error handling across process
boundaries. It would be a lot of work to duplicate/integrate those
features in Lisp.
Helmut.
Helmut Eller <········@stud3.tuwien.ac.at> writes:
> ······@news.dtpq.com (Christopher C. Stacy) writes:
>
> > Why you think lightweight threads would be hard to
> > implement this for Lisp is beyond my imagination.
>
> Erlang doesn't have threads. Erlang has processes. Processes don't
> share memory (heap objects), but they share code. Implementing
> lightweight processes is a challenge in Lisp, because the Lisp coding
> style relies heavily on global variables and mutation. E.g., a vanilla
> CMUCL core has a about 1000 globally bound (non-constant) special
> variables. How do you deal with those 1000 variables and say 500
> processes? I doubt that you want to create a separate copy for each
> process. Implementing some copy-on-write scheme is certainly tricky
> and will most likely make access to global variables slower.
>
> Erlang has also many other features like a module system, code
> distribution, code updates, or error handling across process
> boundaries. It would be a lot of work to duplicate/integrate those
> features in Lisp.
I guess you better just stick with Erlang, then.
······@news.dtpq.com (Christopher C. Stacy) writes:
> Helmut Eller <········@stud3.tuwien.ac.at> writes:
>
>> Erlang has also many other features like a module system, code
>> distribution, code updates, or error handling across process
>> boundaries. It would be a lot of work to duplicate/integrate those
>> features in Lisp.
>
> I guess you better just stick with Erlang, then.
This is a particularly foolish response when one considers Helmut's
record of improving many Lisp programmers' lot through his active
development of the SLIME environment for emacs.
For the sake of balance, I wish to make it clear that I do not
consider recognition of the fact that it might be tricky to implement
certain features in Lisp in a fashion useful to the general community
grounds for excommunication, and also to express my earnest wish that
Helmut doesn't stick with Erlang to the exclusion of Lisp, as the Lisp
community would thereby lose a good contributor.
Christophe
On Thu, 21 Apr 2005 09:48:27 +0100, Christophe Rhodes <·····@cam.ac.uk> wrote:
> This is a particularly foolish response when one considers Helmut's
> record of improving many Lisp programmers' lot through his active
> development of the SLIME environment for emacs.
Thanks, Christophe. I can only second that. Harsh responses to
obvious trolls are understandable and might even be necessary
sometimes. It is absolutely childish to alienate everyone who dares
to mention that there /might/ be things in CL that could be improved,
though.
Helmut hasn't posted here for months and I don't think he has ever
posted anything remotely inaccurate or trollish. Even if one is
ignorant enough not to be aware of his contributions to the Lisp
community there's no reason to try to shoo him away because of one
mention of Erlang instead of answering with a technical refutation.
(Unless one can't refute his claims, of course. Then it might look
like a better strategy to start shouting.)
Cheers,
Edi.
--
Lisp is not dead, it just smells funny.
Real email: (replace (subseq ·········@agharta.de" 5) "edi")
I didn't think my response was harsh in the slightest.
I simply suggested that if he ought to use the language
that he prefers
I made that recommendation because it looks to me like Helmut
is demanding certain language features which Lisp does not
have, rather than considering the purpose of those features.
He wants to do functional programming, and has identified
language elements of Erlang that will help with that.
You can do the same sorts of things in Lisp, but at a little bit
different abstraction level. But they will not be quite like
the style that he wants. I was under the impression he knows
enough about Lisp to figure that out, and doesn't want it.
He wants Erlang, I'm not interested in a language war.
I'm not even interested in "converting" anybody.
Therefore, I suggested that he should stick with Erlang.
What's "harsh" about that?
······@news.dtpq.com (Christopher C. Stacy) writes:
> I didn't think my response was harsh in the slightest.
> I simply suggested that if he ought to use the language
> that he prefers
I also don't think the response was harsh. Actually, it was a
reasonable suggestion.
> I made that recommendation because it looks to me like Helmut
> is demanding certain language features which Lisp does not
> have, rather than considering the purpose of those features.
> He wants to do functional programming, and has identified
> language elements of Erlang that will help with that.
I didn't say that I want to do functional programming. I tried (but
failed) to explain that the process abstraction as provided by Erlang
_is_ difficult to implement efficiently, if you want to support _at
the same time_ Lisp programming style.
> You can do the same sorts of things in Lisp, but at a little bit
> different abstraction level.
Well, you can also do it in assembly, but at a little bit different
abstraction level.
There's a lot of Lisp code out there which is not written with threads
in mind. If you want to use that code in a multi threaded
environment, you have to re-examine it for synchronization bugs.
That's a tough business. Such reviews are not necessary if you have a
process abstraction.
> But they will not be quite like
> the style that he wants. I was under the impression he knows
> enough about Lisp to figure that out, and doesn't want it.
> He wants Erlang, I'm not interested in a language war.
> I'm not even interested in "converting" anybody.
> Therefore, I suggested that he should stick with Erlang.
>
> What's "harsh" about that?
Nothing.
Helmut.
Helmut Eller <········@stud3.tuwien.ac.at> writes:
> There's a lot of Lisp code out there which is not written with threads
> in mind. If you want to use that code in a multi threaded
> environment, you have to re-examine it for synchronization bugs.
> That's a tough business. Such reviews are not necessary if you have a
> process abstraction.
This is true. There is nothing provided in Lisp that will ensure
that arbitrary programs don't communicate with each other.
It is up to the programmer to prevent that.
However, it seems like it would be pretty easy in Lisp to write a program
that guaranteed this. You would simply have to declare process boundaries
as in Erlang modules. It is trivial to prove that if you are only using
functions (and variables) that belong to you, that you are only using
using functions (and variables) that belong to you. This can be done
by trivial lexical analysis. (You have to complain about uses of EVAL,
and complain or do considerable analysis to track the use of INTERN,
of course.) Once you have controlled all lexical access in this way,
the program should be provably safe in this sense.
I don't see offhand why that's not just a matter of a few Lisp macros,
whose names are DEFPROCESS, DEFUN-FOR-PROCESS, and DEFVAR-FOR-PROCESS.
Existing code could be trivially retrofitted.
I admit that I haven't thought about this for more than the time
it took to dash this response off. Is there some reason why this
would not work?
······@news.dtpq.com (Christopher C. Stacy) writes:
> Helmut Eller <········@stud3.tuwien.ac.at> writes:
>
>> There's a lot of Lisp code out there which is not written with threads
>> in mind. If you want to use that code in a multi threaded
>> environment, you have to re-examine it for synchronization bugs.
>> That's a tough business. Such reviews are not necessary if you have a
>> process abstraction.
>
> This is true. There is nothing provided in Lisp that will ensure
> that arbitrary programs don't communicate with each other.
> It is up to the programmer to prevent that.
>
> However, it seems like it would be pretty easy in Lisp to write a program
> that guaranteed this. You would simply have to declare process boundaries
> as in Erlang modules. It is trivial to prove that if you are only using
> functions (and variables) that belong to you, that you are only using
> using functions (and variables) that belong to you. This can be done
> by trivial lexical analysis. (You have to complain about uses of EVAL,
> and complain or do considerable analysis to track the use of INTERN,
> of course.) Once you have controlled all lexical access in this way,
> the program should be provably safe in this sense.
Can I also complain about (setf (readtable-case *readtable*) :downcase)?
How about (defun foo (n) (lambda () (incf n)))?
> I don't see offhand why that's not just a matter of a few Lisp macros,
> whose names are DEFPROCESS, DEFUN-FOR-PROCESS, and DEFVAR-FOR-PROCESS.
Hard to say what those macros mean.
> Existing code could be trivially retrofitted.
>
> I admit that I haven't thought about this for more than the time
> it took to dash this response off. Is there some reason why this
> would not work?
If it is indeed as trivial as you say, then you should get the next
Turing Award.
--
Helmut "the other guy" Eller.
Helmut Eller <········@stud3.tuwien.ac.at> writes:
> Can I also complain about (setf (readtable-case *readtable*) :downcase)?
You can't access either READTABLE-CASE or *READTABLE* in your
process unless you declare that it's part of it. If you take
a global variable from some other program like that, and stick
it into your process, presumably you intended for them to be
sharing state in that way.
> How about (defun foo (n) (lambda () (incf n)))?
What about it? Each time FOO is called, a new anonymous function
object will be created. It's no different from an array or string.
Are you worried that this object will escape from the process somehow?
> If it is indeed as trivial as you say,
> then you should get the next Turing Award.
Perhaps we are miscommunicating about what "it" is.
Could you elaborate?
······@news.dtpq.com (Christopher C. Stacy) writes:
> Helmut Eller <········@stud3.tuwien.ac.at> writes:
>
> > Can I also complain about (setf (readtable-case *readtable*) :downcase)?
>
> You can't access either READTABLE-CASE or *READTABLE* in your
> process unless you declare that it's part of it. If you take
> a global variable from some other program like that, and stick
> it into your process, presumably you intended for them to be
> sharing state in that way.
And you could share it, but just bind it and use it
within that contour, rather than setting it globally.
And you could easily enforce that by having a way in DEFPROCESS
to say that some variables can only be bound, and making
DEFUN-FOR-PROCESS check to make sure that's what you did.
You asked earlier what those macros do; I thought it was obvious
that they establish a data structure that describes your program
containment boundary in terms of restrictions on the lexical
elements of your program; then it walks over your source code
to check that those restrictions are not violated,
else it will refuse to compile.
······@news.dtpq.com (Christopher C. Stacy) writes:
> Helmut Eller <········@stud3.tuwien.ac.at> writes:
>
>> Can I also complain about (setf (readtable-case *readtable*) :downcase)?
>
> You can't access either READTABLE-CASE or *READTABLE* in your
> process unless you declare that it's part of it. If you take
> a global variable from some other program like that, and stick
> it into your process, presumably you intended for them to be
> sharing state in that way.
Why not? Is this a runtime check or a compile time check? Can the
variable only be part of a single thread?
*READTABLE* is in the CL standard (not some other program) and quite a
few programs access it. Do I have to rewrite all those programs?
Including the Lisp implementation itself?
>> How about (defun foo (n) (lambda () (incf n)))?
>
> What about it? Each time FOO is called, a new anonymous function
> object will be created. It's no different from an array or string.
> Are you worried that this object will escape from the process somehow?
Yes. That two threads use the variable to "communicate".
>> If it is indeed as trivial as you say,
>> then you should get the next Turing Award.
>
> Perhaps we are miscommunicating about what "it" is.
> Could you elaborate?
"it" means "that arbitrary programs don't communicate with each other"
where "communicate" includes interfere, e.g., by incrementing a shared
variable at the wrong moment.
Helmut.
Helmut Eller <········@stud3.tuwien.ac.at> writes:
> ······@news.dtpq.com (Christopher C. Stacy) writes:
>
> > Helmut Eller <········@stud3.tuwien.ac.at> writes:
> >
> >> Can I also complain about (setf (readtable-case *readtable*) :downcase)?
> >
> > You can't access either READTABLE-CASE or *READTABLE* in your
> > process unless you declare that it's part of it. If you take
> > a global variable from some other program like that, and stick
> > it into your process, presumably you intended for them to be
> > sharing state in that way.
>
> Why not? Is this a runtime check or a compile time check?
Lisp macros, which I suggested, are (for purposes of this discussion)
at compile-time. (That's why I talked about preventing invalid programs
from compiling.) I thought you were familiar with Lisp macros.
> Can the variable only be part of a single thread?
Previously you suggested we not use the word "thread",
and made it sound like you very much didn't want
to share variables:
> Erlang doesn't have threads. Erlang has processes.
> Processes don't share memory (heap objects),
> but they share code.
What I described was a way to make sure that you don't accidently
share objects, by checking that no undesired lexical references
are permitted across "process" boundaries.
You had gone on to suggest that objects only cross process boundaries
by being copied, and that a problem with Lisp was that it "relied"
upon mutable state. Your example of an anonymous function did not
cross any boundary. Now you're saying that you actually do want
to "share" mutable data, by passing objects (such as that closure,
or arrays, etc.) between processes.
If you want to share objects, share objects (by specifying that
they are shared in the DEFPROCESS macros). If you want some mutex
protection around them, then you'll be using some WITH-MUTEX form
(which the DEFUN-IN-PROCESS macro could insert into your source
code for you), which will expand into whatever algorithm you want.
> *READTABLE* is in the CL standard (not some other program) and quite a
> few programs access it. Do I have to rewrite all those programs?
> Including the Lisp implementation itself?
If those programs are changing the global value of *READTABLE*,
and you don't want that to happen, then you will have to rewrite
those programs. Why don't they bind the variable, instead?
It's true that not all existing Lisp programs in the universe
would be ammenable to this kind of treatment. The question on
the table was about using Lisp to create a new programming
language within Lisp. But many (I think most) existing Lisp
programs would be trivially usable under this scheme, since
they don't typically set global variables.
Why do you think you need to rewrite the Lisp implementation?
> Yes. That two threads use the variable to "communicate".
> "it" means "that arbitrary programs don't communicate with each other"
> where "communicate" includes interfere, e.g., by incrementing a shared
> variable at the wrong moment.
I still don't know what you are asking for, and in particular
how you would claim that it is solved in Erlang in some way
that cannot be done by creating an embedded language within Lisp.
If you are asking for a multiprocessing toolkit that has mutual
exclusion primitives, that's not defined by the ANSI standard,
but already exists in all Lisp implementations that support multiprocessing.
People have been writing multiprocessing systems (including entire
operating systems) in Lisp for at least 30 years. The facilities
are usually called "processes" or "stack groups" in Lisp, and they
share the heap. Some Lisp implementations also have some support
for accessing the platform's native "threads" facility.
If you are asking for programs to be prevented from communicating
with each other, I suggested some fairly straighforward macros
(compile-time technology) for defining and enforcing lexical
boundaries. Those can be easily extended to automatically support
mutually exclusive access to specified shared data structures.
······@news.dtpq.com (Christopher C. Stacy) writes:
>> > You can't access either READTABLE-CASE or *READTABLE* in your
>> > process unless you declare that it's part of it. If you take
>> > a global variable from some other program like that, and stick
>> > it into your process, presumably you intended for them to be
>> > sharing state in that way.
>>
>> Why not? Is this a runtime check or a compile time check?
>
> Lisp macros, which I suggested, are (for purposes of this discussion)
> at compile-time. (That's why I talked about preventing invalid programs
> from compiling.) I thought you were familiar with Lisp macros.
As you surely know, a macro can also expand to code which performs a
runtime check. Your hypothetic macros weren't very well specified, so
I had to ask. Neither in message <·············@news.dtpq.com> nor in
<·············@news.dtpq.com> did you mention compile-time.
>> Can the variable only be part of a single thread?
>
> Previously you suggested we not use the word "thread",
> and made it sound like you very much didn't want
> to share variables:
>
>> Erlang doesn't have threads. Erlang has processes.
>> Processes don't share memory (heap objects),
>> but they share code.
I use the term "threads" for execution contexts which share mutable
objects, like the facilities supported by some CL implementations.
And I use "processes" for the execution contexts which don't share
state, like the facilities supported by Erlang (and OSes).
> What I described was a way to make sure that you don't accidently
> share objects, by checking that no undesired lexical references
> are permitted across "process" boundaries.
>
> You had gone on to suggest that objects only cross process boundaries
> by being copied, and that a problem with Lisp was that it "relied"
> upon mutable state.
My claim is that your macros don't catch all cases in which objects
are shared accidently and that copying would be safe (but inefficient).
Previously you said:
> Why you think lightweight threads would be hard to
> implement this for Lisp is beyond my imagination.
I can very well imagine that an abstraction as provided by Erlang is
very hard to implement in Lisp.
> Your example of an anonymous function did not
> cross any boundary. Now you're saying that you actually do want
> to "share" mutable data, by passing objects (such as that closure,
> or arrays, etc.) between processes.
The example was intended to show that your macros don't cover all
possible cases.
> If you want to share objects, share objects (by specifying that
> they are shared in the DEFPROCESS macros). If you want some mutex
> protection around them, then you'll be using some WITH-MUTEX form
> (which the DEFUN-IN-PROCESS macro could insert into your source
> code for you), which will expand into whatever algorithm you want.
>
>> *READTABLE* is in the CL standard (not some other program) and quite a
>> few programs access it. Do I have to rewrite all those programs?
>> Including the Lisp implementation itself?
>
> If those programs are changing the global value of *READTABLE*,
> and you don't want that to happen, then you will have to rewrite
> those programs. Why don't they bind the variable, instead?
Because the authors didn't think about multi-threading!
> It's true that not all existing Lisp programs in the universe
> would be ammenable to this kind of treatment. The question on
> the table was about using Lisp to create a new programming
> language within Lisp. But many (I think most) existing Lisp
> programs would be trivially usable under this scheme, since
> they don't typically set global variables.
>
> Why do you think you need to rewrite the Lisp implementation?
>
>> Yes. That two threads use the variable to "communicate".
>
>> "it" means "that arbitrary programs don't communicate with each other"
>> where "communicate" includes interfere, e.g., by incrementing a shared
>> variable at the wrong moment.
>
> I still don't know what you are asking for, and in particular
> how you would claim that it is solved in Erlang in some way
> that cannot be done by creating an embedded language within Lisp.
My point is, that it is hard to provide the following two desirable
properties:
1. a guarantee that one process doesn't interfere with another process
2. that existing Lisp code can be used without modifications
Erlang provides the guarantee. (But can't execute Lisp at all.)
Your scheme neither provides a guarantee nor can existing code be used
without modifications. However, it might be a reasonable tradeoff
given the circumstances in the Lisp world.
If there were a Lisp implementation which provides 1., then 2. would
follow automatically. But nobody has written such an implementation.
Helmut.
Helmut Eller wrote:
> My point is, that it is hard to provide the following two desirable
> properties:
>
> 1. a guarantee that one process doesn't interfere with another
> process
> 2. that existing Lisp code can be used without modifications
>
> Erlang provides the guarantee. (But can't execute Lisp at all.)
One high-profile Lisp program with this property seems to be Paul
Graham's Viaweb. He said somewhere that they used the Unix process
abstraction instead of threads, i.e. one CLISP process per client.
That might make some people groan but I reckon it's pretty clever.
Let me just mention why relying on programming discipline is not the
Erlang way with a quote from the first Erlang paper [1]:
In programming large systems, many small programming errors _will_
be made - we view this as inevitable. Formal systems and exhaustive
test procedures are currently not capable of ensuring fault free
software for systems of the size and complexity of modern telecomms
applications. Given that errors _will_ be made, we are interested
in the problem of detecting and handling those errors in such a
manner that the system as a whole exhibits satisfactory behaviour
in the presence of errors.
That philosophy may not be a lot of people's cup of tea but it's surely
a pretty interesting premise to build a programming language on. :-)
1: http://www.erlang.se/publications/iss90.ps
Helmut Eller <········@stud3.tuwien.ac.at> writes:
> My point is, that it is hard to provide the following two desirable
> properties:
>
> 1. a guarantee that one process doesn't interfere with another process
> 2. that existing Lisp code can be used without modifications
Since "existing Lisp code" could do absoluately anything, I definitely agree.
Helmut Eller wrote:
> ······@news.dtpq.com (Christopher C. Stacy) writes:
>
>
>>Why you think lightweight threads would be hard to
>>implement this for Lisp is beyond my imagination.
>
>
> Erlang doesn't have threads. Erlang has processes. Processes don't
> share memory (heap objects), but they share code.
And why is it a bad thing to share memory?? From this talk about
erlang it sounds like to get any shared information you have to use some kind
of inter-process communication protocol. Big deal. Processes that do
any kind of useful work need lots of logic and lots of data. Really
lightweight processes, by definition, cannot do much since they
would consume processor amd memory resources and hence cease to be
"lightweight".
Erlang's claims (PR) do not stand up to close scrutiny. Many Telco
Switches have the same fail over and reliablity characteristics, AND,
they are not written in Erlang. Also the notion about automatic fail-over
can also be a debugging nightmare. I worked with some techs from
Nortel who had a fiber-backbone just like that. It some problem was
encountered by the software it would "fail-over" to another switch/
processor. Unfortunately when the system was deployed across Canada
a systematic failure started to occur. BEFORE the techs could even
figure out what the problem was, the problem moved to another
machine (maybe across the country), ran fine for a while, then
started to fail and moved on yet again. They had a terrible
time debugging what was even causing the problem.
> Implementing
> lightweight processes is a challenge in Lisp, because the Lisp coding
> style relies heavily on global variables and mutation. E.g., a vanilla
> CMUCL core has a about 1000 globally bound (non-constant) special
> variables. How do you deal with those 1000 variables and say 500
> processes?
(Rolling eyes to heaven) There is no Lisp coding style. Again
why is it better to run seperate "processes" in their own
memory space that "processes" in a shared memory space?
I doubt that you want to create a separate copy for each
> process. Implementing some copy-on-write scheme is certainly tricky
> and will most likely make access to global variables slower.
>
> Erlang has also many other features like a module system, code
> distribution, code updates, or error handling across process
> boundaries. It would be a lot of work to duplicate/integrate those
> features in Lisp.
>
> Helmut.
Wade Humeniuk wrote:
> why is it better to run seperate "processes" in their own
> memory space that "processes" in a shared memory space?
One long answer to this is Joe Armstrong's recent thesis, "Making
reliable distributed systems in the presence of software errors":
http://www.sics.se/~joe/thesis/armstrong_thesis_2003.pdf
Personally I think Scott McKay has it right:
Erlang is one of those rare languages -- like Lisp, APL, Snobol,
Prolog, ML -- that is worth looking at for its excellent ideas.
I can also recommend Helmut's multithreading support in SLIME [1] as an
excellent example of a simple and clear concurrent Lisp program (in
contrast to the hairy and incorrect pthreads-style mess that I had
made previously [2]). Ask me about it in Amsterdam if you're there &
interested.
1:
http://common-lisp.net/pipermail/slime-devel/2004-January/000738.html
2:
http://common-lisp.net/pipermail/slime-devel/2003-December/000425.html
"Luke Gorrie" <······@gmail.com> wrote in message news:<·······················@l41g2000cwc.googlegroups.com>...
> Personally I think Scott McKay has it right:
>
> Erlang is one of those rare languages -- like Lisp, APL, Snobol,
> Prolog, ML -- that is worth looking at for its excellent ideas.
One obstacle in adding ideas from interesting languages to Lisp
is the desire for seamless meshing. A losing proposition in an
imperfect world of mutually exclusive ideas: purely functional vs.
mutation, strict vs. lazy, call/cc vs. unwind-protect.
I'm fairly convinced that the right way to implement all this
"interesting stuff" in Common Lisp is to think of the seams and
borders required.
Taking Scheme for example: to support call/cc and not break the
promise made by unwind-protect Scheme cannot have unrestricted
access to the CL environment. The language boundary must eg.
enforce sanity by limiting the allowed class of arguments to immutable
objects, and prohibiting direct access to CL from Scheme -- symbol
value access and function calls to CL must likewise pass the boudary.
Such a boundary may sound like a harsh measure, but I fail to see
how to accommodate mutually exclusive semantics without one
-- though for some languages the boundary may be blurred with
little ill effect; keeping Scheme as an example, direct calls to Scheme
could be allowed with the caveat that call/cc will mess you up then.
In a cl-erlang CL would be in the same relation to Erlang as C is to
C-based Erlang: the Erlang would not be an extension to CL, but
CL would be the way to extend Erlang and reach beyond its limits.
Cheers,
-- Nikodemus "What lithp?"
Luke Gorrie wrote:
> Wade Humeniuk wrote:
>
>>why is it better to run seperate "processes" in their own
>>memory space that "processes" in a shared memory space?
>
>
> One long answer to this is Joe Armstrong's recent thesis, "Making
> reliable distributed systems in the presence of software errors":
>
> http://www.sics.se/~joe/thesis/armstrong_thesis_2003.pdf
>
I have no problem with most of the assertions in the doc, but
the author does acknowledge that it is necessary to structure
one's system so that there is "good" decomposition of the system
so that "good" and effective inter-process protocols can be
developed. (Concurrency oriented programming). It's kind
of like the assertion "with a sufficiently smart compiler" ...
I agree that a stable and good system can be written with
a strict message passing scheme. The people from Ericcson
did just that, Erlang helped some, but it was the Switch
development team that did most of the work in designing
the interacting processes. That is where most of the work
lay, and, of course they did a good job. They knew
what they were doing. The flip side is that it could have been
botched, just as OO design can be botched, or shared-memory
communication can be botched.
Wade
Wade Humeniuk wrote:
> And why is it a bad thing to share memory?? From this talk about
> erlang it sounds like to get any shared information you have to use some
> kind
> of inter-process communication protocol. Big deal. Processes that do
> any kind of useful work need lots of logic and lots of data. Really
> lightweight processes, by definition, cannot do much since they
> would consume processor amd memory resources and hence cease to be
> "lightweight".
A process can have lots of logic and data and still communicate
with other processes. Of course if you need several threads to
share that data, by all means use threads.
The advantage of (Unix) processes are that they have a very simple
interface, that simplifies programming. The lack of memory
sharing also allows you to let some processes run on remote
machines, while still communicating with read()/write().
I'd say that any data and logic that doesn't need to be shared
should be separated into separate processes (if concurrency is
desired; otherwise simple modular programming is of course
sufficient).
--
No man is good enough to govern another man without that other's
consent. -- Abraham Lincoln
Wade Humeniuk <··················@telus.net> writes:
> And why is it a bad thing to share memory??
Generally the reason given is that it removes a whole class of bugs.
Synchronization when updating data structures is no longer required.
Of course there is a trade-off involved; you have to copy data[1] a
lot of times to interact between threads, and the message passing
model imposes some constraints on the design of your program. The
Erlang folks just argue that this trade off is a no-brainer.
Having written multi-threaded programs over the years, I often find
myself often mimicing Erlang's approach, just to achieve some
predictability and reproducability. They have a strong argument to
make, IMHO.
-russ
[1] Of course I'm simplifying here a bit. Sometimes you don't have to
copy the data, if you have a sink abstraction where the data producing
thread can put the data into a queue, and then provably no longer
access the data...
Russell McManus <···············@yahoo.com> schrieb:
> Wade Humeniuk <··················@telus.net> writes:
>> And why is it a bad thing to share memory??
> Generally the reason given is that it removes a whole class of bugs.
> Synchronization when updating data structures is no longer required.
Transparent distribution of Erlang processes is another reason. It's
hard to share data when the code isn't running on the same computer.
Russell McManus wrote:
> Wade Humeniuk <··················@telus.net> writes:
>
>
>>And why is it a bad thing to share memory??
>
>
> Generally the reason given is that it removes a whole class of bugs.
> Synchronization when updating data structures is no longer required.
>
> Of course there is a trade-off involved; you have to copy data[1] a
> lot of times to interact between threads, and the message passing
> model imposes some constraints on the design of your program. The
> Erlang folks just argue that this trade off is a no-brainer.
>
I have programmed with a message passing model and (at least for
embedded systems) this introduces other problems. Pools of actual messages
should be managed (some kind of resource management), global and local flow control
between message queues to prevent overloading a process, process
failure detection, and assurance of message delivery and processing.
Logic and configuration is needed to
handle those "failure" situations. I have found there is no
free lunch. The worst thing that can happen is when the system
locks up or loses messages (because of resource limits).
This, at least in the systems I have seen, has been handled by
manager and watchdog processes, meaning the distributed ideal
needed authoritarian oversite. Maybe a way has been found around
this, I do not know. From my brief re-reading of the Erlang docs
this admin overhead does still seem to be there.
> Having written multi-threaded programs over the years, I often find
> myself often mimicing Erlang's approach, just to achieve some
> predictability and reproducability. They have a strong argument to
> make, IMHO.
>
I agree mimicing that approach can have benefits, like not worrying
about your state machines handling reentrant calls (only process one
message at a time). But like all things, if your code sucks the
system sucks.
> -russ
>
> [1] Of course I'm simplifying here a bit. Sometimes you don't have to
> copy the data, if you have a sink abstraction where the data producing
> thread can put the data into a queue, and then provably no longer
> access the data...
>
Wade Humeniuk <··················@telus.net> writes:
> Helmut Eller wrote:
> > ······@news.dtpq.com (Christopher C. Stacy) writes:
> >
> >> Why you think lightweight threads would be hard to implement this
> >> for Lisp is beyond my imagination.
> > Erlang doesn't have threads. Erlang has processes. Processes don't
> > share memory (heap objects), but they share code.
>
> And why is it a bad thing to share memory??
In Lisp, I would just refrain from using global variables.
I also might use closures. But it's programmer discipline,
not language restrictions, that provide that.
> Really lightweight processes, by definition, cannot do much
> since they would consume processor amd memory resources and
> hence cease to be "lightweight".
I think (or the other guy) was conflating the idea of a
seperated execution context (with its own heap) with the
scheduling of execution. In the implementation they were
referring to (is there only one Erlang?), this was done
without using platform-native threads.
The efficiency claim about "lightweight processes" simply
reflects a design problem in the Erlang implementation.
The word "process" is meant to imply that there is a
seperate data heap. If you conceptually map that directly
onto the operating system, you get the same word: "process".
Some operating systems have more overhead for the creation
of processes than these implementors wanted, because you're
going to be creating "processes" all the time. There is also
the problem of interprocess communication and copying the data
between such processes.
The Erlang solution is to implement in-process execution contexts.
The other guy used the word "threads". There is no reason why Erlang
could not use native threads, which ought to be more efficient
(particularly on a multi-processor machine) than using their own.
This is just an implementation detail.
The MDL language (back in the 70s), which was a forerunner
of Scheme and Common Lisp (and which competed with MACLISP
in some circles at MIT) had a data object that I think is
a lot like what Erlang has. They were called "processes",
and were used for things like co-routines.
Lisp's general answer to all of this, since then, is closures,
stack-groups, similar in-process "green threads", native threads,
and the usual scheduler and "process" synchronization features.
Lisp has had those features since before the word "thread" came
along - we always just used the word "process". That's why the
PROCESS-MUMBLE functions in the Lisp systems today are not called
THREAD-MUMBLE. It never meant "native process". Lisp machines
used to be the only operating systems that really had threads.
Nowaadays, systems like Unix have threads, so I guess it's
convenient that we hadn't already used that word up, because
some Lisp systems support both "green" and "native" threads.
> > Implementing lightweight processes is a challenge in Lisp,
> > because the Lisp coding style relies heavily on global
> > variables and mutation. E.g., a vanilla CMUCL core has
> > a about 1000 globally bound (non-constant) special
> > variables. How do you deal with those 1000 variables
> > and say 500 processes?
>
> (Rolling eyes to heaven) There is no Lisp coding style.
Lisp allows many different coding styles.
Some people see that as a liability.
This is purely a religous argument.
His observation about the implementation of CMUCL using lots
of global variables is specious. It has nothing to do with
the (lack of) difficulty of implementing Erlang in CMUCL.
One could implement, in CMUCL, an embedded domain language
that had "process" and heap semantics like Erlang.
(I don't think it would be very difficult, but of course that's
just an unsubstantiated claim, since I haven't done it.)
> > Erlang has also many other features like a module system,
> > code distribution, code updates, or error handling across
> > process boundaries. It would be a lot of work to
> > duplicate/integrate those features in Lisp.
Here he is referring to specific Erlang solutions to certain
general problems. Lisp has approaches to those problems.
However, Lisp does not include any distributed computing tools.
Erlang is oriented around message-passing program containers.
You could certainly implement that in Lisp (and again, I will
claim that it would not be very difficult.)
Certainly there have been embedded langauges for distributed
computing based on message passing, written in Lisp.
I may have been mistaken in assuming that the OP knew about the way
that Lisp approaches the general conceptual problems that Erlang
addresses with its "process" model. He certainly knows that Lisp
is Turing complete, and probably even that it is well-suited for
embedded languages.
I am left to conclude that he rejects those ideas,
which makes his complaint boil down to: "Lisp is not Erlang".
Hence my suggestion that he stick with Erlang.
I apologize if that advice sounded harsh, but it was actually
intended to merely acknowledge that Lisp, in fact, is not Erlang,
and that some people will like Erlang better. More power to them.
> Erlang's claims (PR) do not stand up to close scrutiny.
> Many Telco Switches have the same fail over and reliablity
> characteristics, AND, they are not written in Erlang.
Lisp has been successfully used for carrier-class telephony switching.
But that's just evidence presented to promote a religion, not a technical
comment about the need (or lack thereof) for object-oriented-cum-functional
programming using a distributed message-passing kernel, or the kind of
language semantics that would be most desirable for that, or anything about
how difficult it is to implement any of those things in any given language.
······@news.dtpq.com (Christopher C. Stacy) writes:
> In the implementation they were > referring to
[...]
> reflects a design problem in the Erlang implementation.
I don't want to slur Erlang here. Those comments are based on an
interpretation of what one of the posters was describing, which might
not accurately reflect actual Erlang implementations. There appears
to have been some terminology confusion by one of the posters,
and possible conflation of Erlang language elements with the
implementation details.
The point is: whether certain operating system facilities
(processes, threads, memory mapping, etc.) are the most
efficient way to implement virtual program containers
should merely be an implementation detail.
(Of course, such implementation details, might also be very
important for people trying to deliver programs with certain
performance characteristics.)
Christopher C. Stacy wrote:
>
> Lisp allows many different coding styles.
> Some people see that as a liability.
> This is purely a religous argument.
>
> His observation about the implementation of CMUCL using lots
> of global variables is specious. It has nothing to do with
> the (lack of) difficulty of implementing Erlang in CMUCL.
> One could implement, in CMUCL, an embedded domain language
> that had "process" and heap semantics like Erlang.
> (I don't think it would be very difficult, but of course that's
> just an unsubstantiated claim, since I haven't done it.)
>
>
LispWorks has many of the Erlang semantics already. Its
multiprocess system with mailboxes could be used to implement
a message passing system. Processes bind shared special
vars and are protected by one all inclusive handler-bind to
provide "crash protection". Like you said though, its up to
programmer discipline to stay within the model. Whether one
relys on the OS for system stability (for example to protect
other processes from another's failure) or rely on a certain
Lisps inherent robustness is no different. Erlang seems to
do just that in practice, rely on the OS to be reliable.
One of the things I find a little interesting is the analogy
between (i think) Flavors and CLOS. Though I have not used
Flavors, my understanding was that it had a message passing
like syntax. Though method dispatch in CLOS looks just like
a function call (or RPC). I suppose this argument will go
on longer between things like RPC, message passing, CORBA,
etc, etc.
Wade
As an Aside:
I would just like to point to people to the work done
with the Amoeba operating system. (which relied on a
RPC like mechanism, which granted, is a message passing
system deep down.)
http://www.cs.vu.nl/pub/amoeba/
http://www.cs.vu.nl/pub/amoeba/amoeba.html
>>>>> "Helmut" == Helmut Eller <········@stud3.tuwien.ac.at> writes:
Helmut> Erlang has also many other features like a module system,
The Erlang module system is a pale imitation of the lips package
system and the generality of symbols.
I do not how formally defined it is but you are in great trouble if
the module identification of a file does not match the file name.
There is no manipulable concept of the current module like we have in
lisp with *package* which means that allthough Erlang is a dynamic
language with a REPL and all, you can not just cut and paste code into
the REPL and expect it to work.
You can also only access that from a module which has been
exported. Some would hold that it is a feature, but as all such
features that tries to force socalled good behaviour on programmers,
it ends up tempting the programmers to circumvent the barrier (Erlang
has an 'export-all' directive).
I would claim that Erlang can be relatively straightforwardly
translated to Lisp when focusing on functionality rather than
characteristics. I still think it would be a challenge to find a Lisp
whose MP substrate would readily handle several hundred thousand
processes but I also seriously doubt whether that is a very usefull
characteristic.
------------------------+-----------------------------------------------------
Christian Lynbech | christian ··@ defun #\. dk
------------------------+-----------------------------------------------------
Hit the philistines three times over the head with the Elisp reference manual.
- ·······@hal.com (Michael A. Petonic)
Christian Lynbech <·········@defun.dk> writes:
>>>>>> "Helmut" == Helmut Eller <········@stud3.tuwien.ac.at> writes:
>
> Helmut> Erlang has also many other features like a module system,
>
> The Erlang module system is a pale imitation of the lips package
> system and the generality of symbols.
I think packages and modules serve slightly different purposes:
modules separate programs and packages separate data. In Lisp,
programs are data and that's important in particular for macros. In
Erlang, this programs==data aspect is less important. But modules are
IMO a more useful abstraction when we talk about code distribution and
updates.
E.g., it makes sense to say that "all nodes in a system use the same
version of a module". Saying "all nodes use the same version of a
package" sounds rather strange.
So while both packages and modules can be used as a namespace
mechanism, the concepts have additional roles: packages serve as a
hygiene mechanism for macros in Lisp and modules are useful for code
distribution and updates in Erlang. Modules are useful for the
"versioning" problem.
> I do not how formally defined it is but you are in great trouble if
> the module identification of a file does not match the file name.
>
> There is no manipulable concept of the current module like we have in
> lisp with *package* which means that allthough Erlang is a dynamic
> language with a REPL and all, you can not just cut and paste code into
> the REPL and expect it to work.
>
> You can also only access that from a module which has been
> exported. Some would hold that it is a feature, but as all such
> features that tries to force socalled good behaviour on programmers,
> it ends up tempting the programmers to circumvent the barrier (Erlang
> has an 'export-all' directive).
Yes, I totally agree. I think this is to some degree a deficiency of
the programming environment. For instance, I see no reason why the
REPL should not be able to bypass the module system, just like a
debugger should be able to call non-exported functions. Enforcing
module boundaries in regular code is more debatable.
But I think the differences between packages and modules that I
described above are unrelated to the question whether non-exported
functions/symbols can be referenced from the outside.
> I would claim that Erlang can be relatively straightforwardly
> translated to Lisp when focusing on functionality rather than
> characteristics. I still think it would be a challenge to find a Lisp
> whose MP substrate would readily handle several hundred thousand
> processes but I also seriously doubt whether that is a very usefull
> characteristic.
If you restrict yourself to Erlang, then I agree. Threads in Lisp are
efficiency wise not that far away from Erlang processes. But
restricting ourself to Erlang (a functional language without
assignments) is not very interesting.
What I tried to explain all the time was that threads and processes
are not the same and that processes are harder to implement than
threads. There are important differences between threads and
processes which become apparent if we use the non-functional parts of
Lisp, like assignment and mutation. These differences are
particularly important when we try to extend a language without
concurrency, like CL, with a concurrency mechanism or when we try to
extend a language without assignments, like Erlang, with assignments.
Programs written in CL can suddenly have synchronization bugs if we
execute them in a CL+threads environment. Such bugs simply don't
happen if we execute them in a CL+processes environment. Processes
are a backward compatible extension. Threads are not; threads are
primarily an efficiency hack.
Erlang programs will have similar bugs if we execute them in an
environment where assignments are allowed and Erlang processes are
implemented as threads. A faithful translation of Erlang to Lisp
should map processes to processes and not processes to threads.
But AFAIK, existing CL system only support threads (if we ignore for
the moment the possibility to use one OS process for one Lisp
process).
Here's a paper that describes many of the those issues better than I
can. And an implementation in the Java world. They call processes
"tasks" or "isolates" but the concept is pretty much the same.
Multitasking without Compromise: A Virtual Machine Evolution
http://research.sun.com/projects/barcelona/papers/oopsla01.pdf
Helmut.
Helmut Eller <········@stud3.tuwien.ac.at> writes:
> Programs written in CL can suddenly have synchronization bugs if we
> execute them in a CL+threads environment. Such bugs simply don't
> happen if we execute them in a CL+processes environment. Processes
> are a backward compatible extension. Threads are not; threads are
> primarily an efficiency hack.
People have been writing complex multi-threaded Lisp applications
(aoperating systems, window systems, network servers, etc.) since
the late 1970s, long before there was language support for doing
that in languages like C, and long before Erlang (or Java, or
any number of other languages) were conceived.
These were called "processes" by Lisp -- the word "thread" had
not yet been invented. We're talking about multiple threads of
execution in a single lareg address space sharing a single heap.
The single address space simultaneously contained all of the
applications as well as the entire (preemptive multitasking)
operating system. This system worked out extremely well for
tens of thousands of users over a couple of decades, in all
kinds of critical (including real-time) environments,
and it was not buggy.
Which Lisp programs are you reporting on,
that you had all these bugs?
······@news.dtpq.com (Christopher C. Stacy) writes:
> Which Lisp programs are you reporting on,
> that you had all these bugs?
Here's an example:
http://cvs.sourceforge.net/viewcvs.py/sbcl/sbcl/src/code/reader.lisp?rev=1.42&view=auto
This is SBCL's reader. The code is almost portable CL. And it has a
threading bug. Can you spot it? The problem is the global
*read-buffer*, a string that's used to implement various stuff
efficiently. But there's no code at all to synchronize concurrent
access/modifications to the buffer. If two unrelated threads call
READ at the same time, they will use the same buffer and can possibly
interfere with each other.
Is the bug also there in a single threaded implementation or in a
implementation with processes? I think not.
Now you could say that this code is not idiomatic CL. To some degree
I would agree. But real processes are still nice because they give
this warm fuzzy feeling that bugs like this one are not there at all.
Helmut.
Helmut Eller <········@stud3.tuwien.ac.at> wrote:
+---------------
| ······@news.dtpq.com (Christopher C. Stacy) writes:
| > Which Lisp programs are you reporting on,
| > that you had all these bugs?
|
| Here's an example:
|
| http://cvs.sourceforge.net/viewcvs.py/sbcl/sbcl/src/code/reader.lisp?rev=1.42&view=auto
|
| This is SBCL's reader. The code is almost portable CL. And it has a
| threading bug. Can you spot it? The problem is the global
| *read-buffer*, a string that's used to implement various stuff
| efficiently. But there's no code at all to synchronize concurrent
| access/modifications to the buffer. If two unrelated threads call
| READ at the same time, they will use the same buffer and can possibly
| interfere with each other.
+---------------
Interesting. The corresponding CMUCL file contains this macro:
;;; Recursive reader functions use with-read-buffer to allocate a
;;; fresh buffer. We currently allocate a fresh buffer only for the
;;; exported functions READ, READ-PRESERVING-WHITESPACE,
;;; READ-FROM-STRING, and READ-DELIMITED-LIST. Some internal
;;; functions like READ-TOKEN, INTERNAL-READ-EXTENDED-TOKEN and
;;; READ-STRING avoid the overhead for the allocation and clobber the
;;; current read-buffer.
(defmacro with-read-buffer (() &body body)
"Bind *read-buffer* to a fresh buffer and execute Body."
`(let* ((*read-buffer* (allocate-read-buffer))
(*read-buffer-length* (length *read-buffer*))
(*ouch-ptr* 0)
(*inch-ptr* 0))
(unwind-protect (progn ,@body)
(free-read-buffer *read-buffer*))))
Note that CMUCL's [and one would hope, SBCL's] MP package keeps bound
specials per-thread, thus the LET* means that each thread gets their own
private *READ-BUFFER*, *READ-BUFFER-LENGTH*, *OUCH-PTR*, & *INCH-PTR*.
-Rob
-----
Rob Warnock <····@rpw3.org>
627 26th Avenue <URL:http://rpw3.org/>
San Mateo, CA 94403 (650)572-2607
····@rpw3.org (Rob Warnock) writes:
> Note that CMUCL's [and one would hope, SBCL's] MP package keeps bound
> specials per-thread, thus the LET* means that each thread gets their own
> private *READ-BUFFER*, *READ-BUFFER-LENGTH*, *OUCH-PTR*, & *INCH-PTR*.
Just to make it clear: I wrote the macro you quoted for CMUCL to fix a
related bug. SBCL doesn't have the fix.
Helmut.
Helmut Eller <········@stud3.tuwien.ac.at> writes:
> ····@rpw3.org (Rob Warnock) writes:
>
> > Note that CMUCL's [and one would hope, SBCL's] MP package keeps bound
> > specials per-thread, thus the LET* means that each thread gets their own
> > private *READ-BUFFER*, *READ-BUFFER-LENGTH*, *OUCH-PTR*, & *INCH-PTR*.
>
> Just to make it clear: I wrote the macro you quoted for CMUCL to fix a
> related bug. SBCL doesn't have the fix.
If it really is using a global variable, then that's a horrible bug.
I don't believe the commercial Lisp implementations have that bug.
Lisp will not prevent you from writing buggy code,
because it assumes that you know what you're doing.
······@news.dtpq.com (Christopher C. Stacy) writes:
> Lisp will not prevent you from writing buggy code,
> because it assumes that you know what you're doing.
But the code was mostly correct as long SBCL had no threads. Don't
you think?
Helmut Eller <········@stud3.tuwien.ac.at> writes:
> ······@news.dtpq.com (Christopher C. Stacy) writes:
>
> > Lisp will not prevent you from writing buggy code,
> > because it assumes that you know what you're doing.
>
> But the code was mostly correct as long SBCL had no threads.
> Don't you think?
Most of the Lisp implementations I've used in the
last 25 years had threads, so I am very surprised
that someone would write that buggy code.
But if your point is that taking non-thread-safe code and
expecting it to run in the presence of threads will lead
to bugs, is there any disagreement on that point?
······@news.dtpq.com (Christopher C. Stacy) writes:
> But if your point is that taking non-thread-safe code and
> expecting it to run in the presence of threads will lead
> to bugs, is there any disagreement on that point?
No, not on that point. But apparently we disagree that processes
(without shared mutable objects) would be more useful than threads.
Non-thread-safe code will not lead to this kind of bugs in the
presence of processes.
Helmut.
Helmut Eller <········@stud3.tuwien.ac.at> writes:
> ······@news.dtpq.com (Christopher C. Stacy) writes:
>
>> But if your point is that taking non-thread-safe code and
>> expecting it to run in the presence of threads will lead
>> to bugs, is there any disagreement on that point?
>
> No, not on that point. But apparently we disagree that processes
> (without shared mutable objects) would be more useful than threads.
> Non-thread-safe code will not lead to this kind of bugs in the
> presence of processes.
Aren't you two talking past each other? I think in a lot lisps,
global specials are auto-magically thread specific, so the problem you
are talking about doesn't exist. But then again, perhaps I
misunderstand...
-russ
Russell McManus <···············@yahoo.com> writes:
> Helmut Eller <········@stud3.tuwien.ac.at> writes:
>
> > ······@news.dtpq.com (Christopher C. Stacy) writes:
> >
> >> But if your point is that taking non-thread-safe code and
> >> expecting it to run in the presence of threads will lead
> >> to bugs, is there any disagreement on that point?
> >
> > No, not on that point. But apparently we disagree that processes
> > (without shared mutable objects) would be more useful than threads.
> > Non-thread-safe code will not lead to this kind of bugs in the
> > presence of processes.
>
> Aren't you two talking past each other? I think in a lot lisps,
> global specials are auto-magically thread specific, so the problem you
> are talking about doesn't exist. But then again, perhaps I
> misunderstand...
No, you don't misunderstand; the code shown earlier in this thread [sic]
was safe at least with respect to globally special variables, because
they were "lambda-bound" (i.e. they appear as a lambda variable in a
lambda-list, or in a var location in a let form). Note however that
in most implementations globally-special variables are not thread-safe
in threads in which they are not lambda-bound, becasue the value slot
tends to be in a global place when the variable is not lambda-bound.
I presume that other implementations also have a way of simulating
such lambda-bindings, similar to Allegro CL's required-top-level-binding
and required-thread-binding macros. This is a way to get such variables
to be bound per REPL or thread, respectively, without having to
explicitly bind them at some point within the REPL or thread.
--
Duane Rettig ·····@franz.com Franz Inc. http://www.franz.com/
555 12th St., Suite 1450 http://www.555citycenter.com/
Oakland, Ca. 94607 Phone: (510) 452-2000; Fax: (510) 452-0182
Russell McManus <···············@yahoo.com> writes:
>> Aren't you two talking past each other? I think in a lot lisps,
>> global specials are auto-magically thread specific, so the problem you
>> are talking about doesn't exist. But then again, perhaps I
>> misunderstand...
Despite that this is not done in practice, making the variable
thread-local would not be sufficient. The value stored in the
variable also needs the be thread-local, because the problem still
exists if two thread-local variables point to the same buffer.
Duane Rettig <·····@franz.com> writes:
> No, you don't misunderstand; the code shown earlier in this thread [sic]
> was safe at least with respect to globally special variables, because
> they were "lambda-bound" (i.e. they appear as a lambda variable in a
> lambda-list, or in a var location in a let form).
I say it again to avoid confusion. The example that I cited is from
SBCL and the code that Rob Warnock quoted is from CMUCL. The CMUCL
code contains a fix for the bug (and incidentally I wrote the fix).
The SBCL code wasn't fixed.
If you still think that the variable is lambda-bound in my example
then please point out where in this file:
http://cvs.sourceforge.net/viewcvs.py/sbcl/sbcl/src/code/reader.lisp?rev=1.42&view=auto
Helmut.
Helmut Eller <········@stud3.tuwien.ac.at> writes:
> Russell McManus <···············@yahoo.com> writes:
>
> >> Aren't you two talking past each other? I think in a lot lisps,
> >> global specials are auto-magically thread specific, so the problem you
> >> are talking about doesn't exist. But then again, perhaps I
> >> misunderstand...
>
> Despite that this is not done in practice, making the variable
> thread-local would not be sufficient. The value stored in the
> variable also needs the be thread-local, because the problem still
> exists if two thread-local variables point to the same buffer.
Of course.
> Duane Rettig <·····@franz.com> writes:
>
> > No, you don't misunderstand; the code shown earlier in this thread [sic]
> > was safe at least with respect to globally special variables, because
> > they were "lambda-bound" (i.e. they appear as a lambda variable in a
> > lambda-list, or in a var location in a let form).
>
> I say it again to avoid confusion. The example that I cited is from
> SBCL and the code that Rob Warnock quoted is from CMUCL. The CMUCL
> code contains a fix for the bug (and incidentally I wrote the fix).
> The SBCL code wasn't fixed.
>
> If you still think that the variable is lambda-bound in my example
> then please point out where in this file:
>
> http://cvs.sourceforge.net/viewcvs.py/sbcl/sbcl/src/code/reader.lisp?rev=1.42&view=auto
I did not look in any files. The only code I _see_ earlier in this
thread was posted by Rob Warnock. And I was explicit in saying that
whatever thread-safety was present there was only with respect to
the variables themselves. Whethere there is any other non-thread-safe
functionality in that code must of course be considered, likely
for example in the allocate-read-buffer code, but that is attainable.
My comment was on globally-special variables _only_.
--
Duane Rettig ·····@franz.com Franz Inc. http://www.franz.com/
555 12th St., Suite 1450 http://www.555citycenter.com/
Oakland, Ca. 94607 Phone: (510) 452-2000; Fax: (510) 452-0182
Duane Rettig <·····@franz.com> wrote:
+---------------
| Helmut Eller <········@stud3.tuwien.ac.at> writes:
| > I say it again to avoid confusion. The example that I cited is from
| > SBCL and the code that Rob Warnock quoted is from CMUCL. The CMUCL
| > code contains a fix for the bug (and incidentally I wrote the fix).
| > The SBCL code wasn't fixed.
| >
| > If you still think that the variable is lambda-bound in my example
| > then please point out where in this file:
| > http://cvs.sourceforge.net/viewcvs.py/sbcl/sbcl/src/code/reader.lisp?rev=1.42&view=auto
|
| I did not look in any files. The only code I _see_ earlier in this
| thread was posted by Rob Warnock. And I was explicit in saying that
| whatever thread-safety was present there was only with respect to
| the variables themselves. Whethere there is any other non-thread-safe
| functionality in that code must of course be considered, likely
| for example in the allocate-read-buffer code, but that is attainable.
+---------------
And my apologies for any confusion I may have inadvertently caused,
though I did include a quote containing the above URL[1] *and* the
text "This is SBCL's reader ... And it has a threading bug", and I
did try to be clear that I was contrasting that with CMUCL by saying
"The corresponding CMUCL file contains this macro" before displaying it.
The only reason I stopped short of making a definitive statement about
SBCL having a bug was that I hadn't read *all* of the SBCL source, and
for all I knew there might have been some wrapper layer I couldn't see
that contained a fix. But Helmut says not, so no wonder I didn't see it! ;-}
-Rob
[1] Which URL I *did* go read, but failed (now understandably) to find
thread-localizing code similar to CMUCL's.
-----
Rob Warnock <····@rpw3.org>
627 26th Avenue <URL:http://rpw3.org/>
San Mateo, CA 94403 (650)572-2607
····@rpw3.org (Rob Warnock) writes:
> The only reason I stopped short of making a definitive statement
> about SBCL having a bug was that I hadn't read *all* of the SBCL
> source, and for all I knew there might have been some wrapper layer
> I couldn't see that contained a fix. But Helmut says not, so no
> wonder I didn't see it! ;-}
I will not stop short of definitively stating that sbcl has a bug
here, and indeed plenty of others due to the same kind of cause.
(Some are of a lesser severity than others, and some are obscured or
worked around by things like a *big-compiler-lock* or a *pcl-lock*).
This particular bug has been noted on the sbcl mailing list, and a
patch has been discussed (but not merged into CVS HEAD, by my
recollection because of unhappiness at the approach taken by the
patch, but I'm not sure about that). There are, however, plenty of
other places where bad things can happen; interested parties might
want to look at the file doc/internals/threading-specials distributed
with SBCL source, where some attempt at categorization and
classification is taking place.
Christophe
>>>>> "Helmut" == Helmut Eller <········@stud3.tuwien.ac.at> writes:
Helmut> Despite that this is not done in practice, making the variable
Helmut> thread-local would not be sufficient. The value stored in the
Helmut> variable also needs the be thread-local, because the problem still
Helmut> exists if two thread-local variables point to the same buffer.
It has occurred to me that Erlang does not really distinguish itself
in the discussion of threads versus processes, or rather that The
Erlang language is rather insensitive to the differences in the two
concepts.
In Erlang there are only two kinds of mutable objects, tuples and
lists. Erlang also only has the equivalent of Lisps 'equal', there is
no immediate concept of 'eq' (object identity). This means that the
implementation is free to copy the contents of an object when that
object is signalled from one process/thread to another process/thread
or even when objects are transferred between functions. If the Erlang
VM does (effectively) copy objects when they are moved out of a
function, then Erlang has process semantics, if the VM does not copy
objects it has something that much more resembles thread semantics.
However this distinction has very little to do with Erlang processes,
this is much more closely related to how the Erlang *language* as such
is defined (including the fact that Erlang has no global variables).
------------------------+-----------------------------------------------------
Christian Lynbech | christian ··@ defun #\. dk
------------------------+-----------------------------------------------------
Hit the philistines three times over the head with the Elisp reference manual.
- ·······@hal.com (Michael A. Petonic)
Thought I would post an update...
I finished my Erlang poker engine and I wasn't able to reuse anything
from the CL version. Erlang required me to "think different" so even
ideas weren't reusable, so this is a rewrite from scratch that I
completed in about 6 weeks.
I can run about 27,000 games in parallel although this taxes my laptop
quite a bit. I might be able to run more but I don't see the need. I'm
not using networking (sockets) for this test but this will be my next
goal.
I have a PowerBook G4 1.25Ghz with 512Mb of memory. Running 27K games
consumes about 600Mb of memory and takes around 15 minutes per 10K
games due to heavy swapping. I gave "top" a cursory look so this is a
probably reason. Each game runs in about 0.02 to 0.07 seconds,
depending on the number of games running.
I have one process for the game "base", one for the game logic, one for
the pot which also manages the side pots internally. Game limit (fixed,
pot, etc.) adds another process and so does the card deck. Player
"bots" add two processes per player. Assuming an average of 5 players
per game that's 135,000 players and 405,000 simultaneous processes.
Bots have a command queue that they use to pull their responses from. I
also give bots a certain time to respond. With 2-5K processes a timeout
of 100ms is sufficient but I had to increase it to 1 second to run all
27k games through. Without the increase 6-7 games would fail to run
through due to a bot timeout.
I believe Erlang uses less than 400 bytes per process and I'm wondering
if I could achieve the same results with a Common Lisp implementation.
Dealing with processes allowed me to implement the most natural
concurrent design whereas my Allegro CL implementation had to use
tricks like UCW continuations.
Thanks, Joel
Joel Reymont wrote:
> I have a PowerBook G4 1.25Ghz with 512Mb of memory. Running 27K games
> consumes about 600Mb of memory and takes around 15 minutes per 10K
> games due to heavy swapping. I gave "top" a cursory look so this is a
> probably reason. Each game runs in about 0.02 to 0.07 seconds,
> depending on the number of games running.
Swapping sucks. You should really upgrade your memory if you want
to run all those games (get a server with 2GB and enjoy...). How
many games can you run simultaneously in 450MB without swapping?
How fast do they run then?
--
Don't let school interfere with your education. -- Mark Twain
Joel Reymont wrote:
> I believe Erlang uses less than 400 bytes per process and I'm wondering
> if I could achieve the same results with a Common Lisp implementation.
> Dealing with processes allowed me to implement the most natural
> concurrent design whereas my Allegro CL implementation had to use
> tricks like UCW continuations.
Perhaps this message will interest you:
<http://people.csail.mit.edu/gregs/ll1-discuss-archive-html/msg03309.html>
I am ignorant of the inner workings of Erlang, but since Feeley has
used Gambit to implement Etos, I suspect that Gambit's thread system
has a lot in common with Erlang's.
<http://www.iro.umontreal.ca/~gambit/doc/gambit-c_11.html#SEC44>
--
Jens Axel Søgaard
On Thu, 28 Apr 2005 11:29:03 -0400, Russell McManus <···············@yahoo.com> wrote:
> Aren't you two talking past each other? I think in a lot lisps,
> global specials are auto-magically thread specific, so the problem
> you are talking about doesn't exist.
Only if you rebind them per thread.
Cheers,
Edi.
--
Lisp is not dead, it just smells funny.
Real email: (replace (subseq ·········@agharta.de" 5) "edi")
>>>>> "Helmut" == Helmut Eller <········@stud3.tuwien.ac.at> writes:
>> The Erlang module system is a pale imitation of the lips package
>> system and the generality of symbols.
Helmut> I think packages and modules serve slightly different purposes:
Helmut> modules separate programs and packages separate data. In Lisp,
Helmut> programs are data and that's important in particular for macros. In
Helmut> Erlang, this programs==data aspect is less important. But modules are
Helmut> IMO a more useful abstraction when we talk about code distribution and
Helmut> updates.
Helmut> E.g., it makes sense to say that "all nodes in a system use the same
Helmut> version of a module". Saying "all nodes use the same version of a
Helmut> package" sounds rather strange.
Now, I could easily be mistaken as I am nowhere near being a seasoned
Erlang expert, but so far I have not seen anything of the module
system that was not tightly connected to files, that is in Erlang the
concepts of namespaces and object files has been collapsed into one
concept, a module (which incidently is somewhat close to how C does
it).
In Lisp we have two concepts, one for namespaces (packages) and one
for managing files (systems). It would not be a major effort to set up
Lisp to collapse the two. Simply insist that each file is in a package
whose name is the same as the basename of the source file and one
could use the package name as arguments to (slightly extended versions
of) load and compile.
But in Lisp we still have the option of keeping the two concepts
apart. We can have logical namespaces that spans many files or a
single file setting up multiple namespaces.
<steam vent open>
Here is another frustrating aspect of Erlang I have just battled.
I have this function that implements an integrity check. It supposed
to be called with a record but somebody called it with a string
provoking an error in my function. No big deal, I thought, I just need
to robustofy it a bit by checking that the arguments are really
records before I operate on them.
So I tried using the record() guard and wrote a function clause
check(X,Y) when record(X,product) and record(Y,xmlElement) -> ...
Nope. The compile complains that this is an illegal guard.
Ok moving the check into the function, writing something like
if (is_tuple(X) and (element(1,X)==product)) -> ...
This was eaten by the compiler but I still got runtime errors when
calling 'check' with a non-record (Erlang has a tuple type and a
record is just a tuple with a symbol as the first element and
'element' extracts members from tuples).
Some significant headscratching and debug prinouts later, I find out
that 'and' is not shortcircuiting.
Back to the reference manual; aha the shortcircuiting version is
called 'andalso'.
But now the compiler is unhappy. One cannot use 'andalso' in a guard
and in Erlang there are no difference between the gurads used to
discriminate function clauses and the cases inside an 'if'.
So I need to move the calculation out of the 'if', store it in a
variable solely for the purpose of checking whether it is true or
false:
C1=(is_tuple(X) and (element(1,X)==product)),
if C1 -> ...
AAAAARRRRRRRGGGGGGGHHHHHHHHHHHH
I am sure that an experienced Erlang hacker would never had gotten
into this mess but to a Lisp programmer such as myself, this is a
totally bizarre and twisted world to live in. And I do not care just
how many cycles this single-assigment and guard-limitations can save
you in the VM, I HATE what it does to my programs.
</stream vent off>
------------------------+-----------------------------------------------------
Christian Lynbech | christian ··@ defun #\. dk
------------------------+-----------------------------------------------------
Hit the philistines three times over the head with the Elisp reference manual.
- ·······@hal.com (Michael A. Petonic)
Christian Lynbech <·········@defun.dk> writes:
> Now, I could easily be mistaken as I am nowhere near being a seasoned
> Erlang expert, but so far I have not seen anything of the module
> system that was not tightly connected to files, that is in Erlang the
> concepts of namespaces and object files has been collapsed into one
> concept, a module (which incidently is somewhat close to how C does
> it).
I'm also not a Erlang expert. But I think it does some things quite
well and is worth a closer look.
> In Lisp we have two concepts, one for namespaces (packages) and one
> for managing files (systems). It would not be a major effort to set up
> Lisp to collapse the two. Simply insist that each file is in a package
> whose name is the same as the basename of the source file and one
> could use the package name as arguments to (slightly extended versions
> of) load and compile.
It's certainly possible to do that, but is it really that easy?
Whenever I try to use one package per file, I usually have to create
an extra file just for the package definitions. I don't like that
because I prefer to keep the "export declaration" close to the
definition of the function.
> But in Lisp we still have the option of keeping the two concepts
> apart. We can have logical namespaces that spans many files or a
> single file setting up multiple namespaces.
Agreed, Lisp is more flexible in this regard. But how often do you
want to distribute the code for one module across multiple files? If
the code belongs together, why not put it the same file. If the code
isn't related, why not create two modules. I think that putting one
module in one file is a pretty good default and probably that what
most people expect. It makes it also easy to use low-tech tools, like
make. I think Erlang has also something like "-include", but I'm not
sure whether that solves this problem.
Another question is how difficult/easy/worthwhile it would be to extend
Erlang for more complicated scenarios.
[snip..]
> AAAAARRRRRRRGGGGGGGHHHHHHHHHHHH
>
> I am sure that an experienced Erlang hacker would never had gotten
> into this mess but to a Lisp programmer such as myself, this is a
> totally bizarre and twisted world to live in. And I do not care just
> how many cycles this single-assigment and guard-limitations can save
> you in the VM, I HATE what it does to my programs.
:-) You have my full sympathy.
Helmut.
On Wed, 20 Apr 2005 21:46:48 -0700, matthewknox wrote:
> actually, there is at least one thing that would be, I think, extremely
> difficult for a DSL [domain specific language] implemented in lisp to
> simulate, that erlang handles nicely...super-lightweight threads. Erlang
> threads are much lighter-weight than any native threads I am aware of,
> and lisp ones are mostly native or heavier. I would love to be proven
> wrong on this, though.
As the super lightweight threads are non-native/green threads this should
pose no conceptual difficulty. After all Erlang is typically implemented
in C or C++, which are highly imperative languages that are usually
coupled with native threading implementations like POSIX threads.
Only very few language implementations are general enough to be able to
build anything on top of them more or less successfully and efficiently.
Some Common Lisp implementations fit into this category.
By the way native threads are the future. They are becoming faster and
more lightweight on Linux, e.g. with 4k stacks in Linux 2.6. Plus CPUs are
becoming multicore. To utilise all processors in future systems you'll
want to use native threads so the operating system can allocate different
threads to different CPUs. With green threads the operating system only
sees a single thread and can at most allocate one CPU to a process.
Regards,
Adam
On Thu, 21 Apr 2005 18:01:10 +1200, Adam Warner wrote:
> By the way native threads are the future. They are becoming faster and
> more lightweight on Linux, e.g. with 4k stacks in Linux 2.6.
4k stacks provide some secondary benefit:
<http://www.ussg.iu.edu/hypermail/linux/kernel/0405.1/0489.html>
A better example is the new Native POSIX Thread Library (NPTL):
<http://www-106.ibm.com/developerworks/linux/library/l-web26/?ca=dgr-lnxw07KernelCompare>
The 2.6 kernel adapts the new thread library, Native POSIX Thread
Library (NPTL). This new library is based on a 1:1 model and full POSIX
compliance. A test done by Red Hat indicates that on an old IA-32 dual
450MHz PII Xeon system, 100,000 threads could be created and destroyed
in 2.3 seconds (with up to 50 threads running at any one time) using
NPTL.
Regards,
Adam
Hello!
Adam Warner <······@consulting.net.nz> wrote:
>By the way native threads are the future. They are becoming faster and
>more lightweight on Linux, e.g. with 4k stacks in Linux 2.6.
4k is still much more than what Erlang uses per process.
>Plus CPUs are
>becoming multicore. To utilise all processors in future systems you'll
>want to use native threads so the operating system can allocate different
>threads to different CPUs. With green threads the operating system only
>sees a single thread and can at most allocate one CPU to a process.
There's more in the world than n:n and 1:n threading. I.e. perhaps
some kind of n:m scheme would be worthwhile if you both want to use
multiprocessor machines (or multiprocessor machines in disguise) *and*
have really lightweight threads.
>Regards,
>Adam
Kind regards,
Hannah.
On Sun, 01 May 2005 13:59:29 +0000, Hannah Schroeter wrote:
> Hello!
>
> Adam Warner <······@consulting.net.nz> wrote:
>
>>By the way native threads are the future. They are becoming faster and
>>more lightweight on Linux, e.g. with 4k stacks in Linux 2.6.
>
> 4k is still much more than what Erlang uses per process.
That's a kernel stack. Userspace threads will reserve around 2MB per
thread by default for program stack space. On 32 bit platforms the number
of threads is thus rather limited: <http://www.kegel.com/c10k.html#threaded>
64 bit platforms fix this virtual memory limitation.
>>Plus CPUs are becoming multicore. To utilise all processors in future
>>systems you'll want to use native threads so the operating system can
>>allocate different threads to different CPUs. With green threads the
>>operating system only sees a single thread and can at most allocate one
>>CPU to a process.
>
> There's more in the world than n:n and 1:n threading. I.e. perhaps some
> kind of n:m scheme would be worthwhile if you both want to use
> multiprocessor machines (or multiprocessor machines in disguise) *and*
> have really lightweight threads.
Indeed. But with 64-bit computing thousands of native threads will be
feasible. Given these technology trends designing an application around
"really lightweight threads" could turn out to be premature optimisation.
Regards,
Adam
Hello!
Adam Warner <······@consulting.net.nz> wrote:
>On Sun, 01 May 2005 13:59:29 +0000, Hannah Schroeter wrote:
>> Adam Warner <······@consulting.net.nz> wrote:
>>>By the way native threads are the future. They are becoming faster and
>>>more lightweight on Linux, e.g. with 4k stacks in Linux 2.6.
>> 4k is still much more than what Erlang uses per process.
>That's a kernel stack. Userspace threads will reserve around 2MB per
>thread by default for program stack space. On 32 bit platforms the number
>of threads is thus rather limited: <http://www.kegel.com/c10k.html#threaded>
>64 bit platforms fix this virtual memory limitation.
But then, each thread actually touches at least one page per thread,
still making 4k per thread of *real* memory usage. Still much more
than Erlang uses.
>>>Plus CPUs are becoming multicore. To utilise all processors in future
>>>systems you'll want to use native threads so the operating system can
>>>allocate different threads to different CPUs. With green threads the
>>>operating system only sees a single thread and can at most allocate one
>>>CPU to a process.
>> There's more in the world than n:n and 1:n threading. I.e. perhaps some
>> kind of n:m scheme would be worthwhile if you both want to use
>> multiprocessor machines (or multiprocessor machines in disguise) *and*
>> have really lightweight threads.
>Indeed. But with 64-bit computing thousands of native threads will be
>feasible.
Only if you have enough *real* memory too. See above.
>Given these technology trends designing an application around
>"really lightweight threads" could turn out to be premature optimisation.
If you wanna handle say 100000 or 1m concurrent processes (from the
point of view of the application logic), you can decide whether you
want to use 1m * (4k + 4k) [user stack, at least one page touched,
plus kernel stack] = 8GB of real memory (or thrash around like hell),
or whether you use really lightweight threads with say 500 bytes
per thread, that is 1m * 500 = 500m (more managable), or whether you
use other means of multiplexing, making the mapping between high-level
application logic and actual code less straightforward.
>Regards,
>Adam
Kind regards,
Hannah.
Adam Warner wrote:
>Lisp *is* a universal hammer and one can write a domain specific
>language
>with Erlang-like qualities in Lisp while the converse is not true.
>Someone
>who *loves* Lisp as much as Joel ("I actually *love* Lisp") Reymont
>could
>find a practical way to make Lisp work in this server domain.
It's a time to market issue. I could have accomplished what I want in
Lisp but it would have taken me longer. I'm in no way bashing Lisp but
I'm a believer in specialization.
Erlang is specialized enough to let me achieive what I want quickly.
What's the point in persisting with Lisp then and what would I prove
/beside my programming chops/ by doing so?
Joel Reymont wrote:
> It's a time to market issue. I could have accomplished what I want in
> Lisp but it would have taken me longer. I'm in no way bashing Lisp but
> I'm a believer in specialization.
>
> Erlang is specialized enough to let me achieive what I want quickly.
> What's the point in persisting with Lisp then and what would I prove
> /beside my programming chops/ by doing so?
I think posts such as the one that started this thread are useful,
and not at all trollish, since they're data about what should be made
available (as libraries) to make Lisp more attractive. Thank you, Joel.
Paul
Joel Reymont wrote:
>
> It's a time to market issue. I could have accomplished what I want in
> Lisp but it would have taken me longer. I'm in no way bashing Lisp but
> I'm a believer in specialization.
>
> Erlang is specialized enough to let me achieive what I want quickly.
> What's the point in persisting with Lisp then and what would I prove
> /beside my programming chops/ by doing so?
>
Joel,
I have to point something out. You have not done your system
in Erlang yet. I hope it will allow to get it done quicker,
but the fact is you do not know. In the past (from your blog
entries) you have shown a tendency to bail when things become
difficult. Software development is not easy with the
best of tools. Perseverance is necessary to get that last
10% done (which we all know takes 90% of the effort).
As what you might prove by persisting, is to gain the experience
with Lisp and one of its implementations to "see" how to do it.
It takes time, to me "quickly" means hasty in your sentence.
Also by persisting with Lisp, and finding a solution you
might contribute to the body of knowledge about using Lisp and
benefit us all. That is how Lisp has evolved over its 40+ years,
by being used and modified.
Good Luck
Wade
Wade said:
>As what you might prove by persisting, is to gain the experience
>with Lisp and one of its implementations to "see" how to do it.
>It takes time, to me "quickly" means hasty in your sentence.
>Also by persisting with Lisp, and finding a solution you
>might contribute to the body of knowledge about using Lisp and
>benefit us all. That is how Lisp has evolved over its 40+ years,
>by being used and modified.
Wade, your comments are on the money. I have two excellent tools in my
toolset now and I will certainly try to contribute to the Lisp body of
knowledge in my next application.
Thanks, Joel