Dear all,
Where could I find information for making it easier to run multiple
tasks parallel in the same CL image ? These tasks are linked very
closely, it is basically an input process which checks stdin for
control commands, a loop which executes very fast commands and an
output process which sends responses to the control commands and
updates from the main processing loop.
Ideal would be real multi-threading, but my code is currently portable
between CLISP on Linux and Windows, and SBCL. I do not want to lose
this advantage (for me) by introducing incompatible constructs, or
adding too many compiler dependent processing.
Thinking about this problem, I do not expect an easy solution, because
it comes down to writing a dispatch process with cooperative multi-
tasking, but maybe there are sources of information which can make the
implementation of this part less complex.
Regards,
Jurgen
From: Pascal Costanza
Subject: Re: Portable constructs or ideas for multi-tasking/multi-threading/co-routines ?
Date:
Message-ID: <5pr4h2Fnpg1lU1@mid.individual.net>
jurgen_defurne wrote:
> Dear all,
>
> Where could I find information for making it easier to run multiple
> tasks parallel in the same CL image ? These tasks are linked very
> closely, it is basically an input process which checks stdin for
> control commands, a loop which executes very fast commands and an
> output process which sends responses to the control commands and
> updates from the main processing loop.
>
> Ideal would be real multi-threading, but my code is currently portable
> between CLISP on Linux and Windows, and SBCL. I do not want to lose
> this advantage (for me) by introducing incompatible constructs, or
> adding too many compiler dependent processing.
I don't know if this solves your problem, but check out the
portable-threads library of the GBBOpen project at http://gbbopen.org/
Pascal
--
My website: http://p-cos.net
Common Lisp Document Repository: http://cdr.eurolisp.org
Closer to MOP & ContextL: http://common-lisp.net/project/closer/
jurgen_defurne wrote:
> Dear all,
>
> Where could I find information for making it easier to run
> multiple tasks parallel in the same CL image ? These tasks
> are linked very
[[ ...]]
> Thinking about this problem, I do not expect an easy
> solution, because it comes down to writing a dispatch
> process with cooperative multi- tasking, but maybe there
> are sources of information which can make the
[[...]]
Well, we don't have full continuations, but we have a
hackish continuations library in the the arnesi
library. Using this, one can write a cooperative
multitasking toolkit. For example, a few months back, I
tried something like this in Scheme, then converted it to
CL. It is nothing new, and I've seen papers with similar
code dating back a long time (couldn't remember, was there
one by Dijkstra?).
However, during conversion to CommonLisp, there were
problems with arnesi's tail-calls. See this email (as far as
I can tell, there were no replies to it).
http://common-lisp.net/pipermail/bese-devel/2007-April/002956.html
Since then, I've looked at it again, and managed to fix
this. By changing the function 'schedule', and being careful
to workaround some some unintuitive 'kall' behaviour, I've
managed to get it to run without excessive memory usage
(otherwise, arnesi itself does perform tail-calls).
So this is what the test (probably modified a bit from the
attachment to the mailing-list above)
(defun p (&rest x)
(format t "~{~a~^ ~}~%" x)
;; for synchronisation
(finish-output))
(defun/cc test-print (x)
(p "First:" x)
(suspend) ;; Will probably rename this to 'yield' someday
(p "Second:" x)
(suspend)
(p "Third:" x))
(with-call/cc
(spawn (lambda () (test-print "1")))
(spawn (lambda () (test-print "2")))
(spawn (lambda () (test-print "3")))
(spawn* (lambda ()
(p "Hello")
(suspend)
(p "Hello, again")
(suspend)
(p "Bye")
(suspend))))
Running it gives us:
Hello
First: 1
First: 2
First: 3
Hello, again
Second: 1
Second: 2
Second: 3
Bye
Third: 1
Third: 2
Third: 3
Thus far, using a clisp-2.41 on cygwin on windows, I've been
able to demonstrate up to at least 500 'suspend's or context
switches per second, running 'do-nothing' tasks. If I
remember correctly, it's up to 2000 per second on SBCL (the
number of tasks is irrelevant, only context switches, so 50
tasks X 10 yields per seconds, or 500 tasks yielding once a
second are the same (not counting other (actual) work done
by tasks)).
;; Vapourware now:
Using this and a combination of read-char-no-hang (wrapped
into read-line-no-hang), I can probably (I've been meaning
to do it when I get the time) write a multi-threaded repl,
and the world will be mine! :) This will be just a hack, but
it'll also be purely CL! Another dream of mine is to
integrate this into cl-muproc, just for the hack value. I've
also wanted to integrate 'engines' based preemption into the
arnesi interpreter (hopefully it's a simple method
override), so that we get even closer to Erlang-ish
behaviour.
Anyway, if you want, the newer code (that actually works) is
available, and I should probably release it soonish (just
need time to polish and do a release etc). As should be
obvious from that email/attachment I sent, the code is rough
and needs work (lots of TODO's)
On Nov 13, 6:28 pm, szergling <···············@gmail.com> wrote:
> So this is what the test (probably modified a bit from the
> attachment to the mailing-list above)
>
> (defun p (&rest x)
> (format t "~{~a~^ ~}~%" x)
> ;; for synchronisation
> (finish-output))
>
Quick note. The finish-output is not really for synchronisation, since
there is no preemption, but it was there when I was debugging, just to
be sure. Don't worry too much about it.
s> Well, we don't have full continuations, but we have a
s> hackish continuations library in the the arnesi
s> library.
the old arnesi had CPS-transformer ("compiler") rather than interpreter.
wouldn't that be better?
i know it had problems with some constructs like throw, and didn't have
serializable continuations, but for some limited subset it might work well.
On Nov 13, 9:05 pm, "Alex Mizrahi" <········@users.sourceforge.net>
wrote:
> s> Well, we don't have full continuations, but we have a
> s> hackish continuations library in the the arnesi
> s> library.
>
> the old arnesi had CPS-transformer ("compiler") rather than
Yes, I've read one of Marco's posts mentioning it, but have not
been able to find it. Is there a copy of this somewhere?
> interpreter. wouldn't that be better? i know it had problems
Do you mean faster when you say better? Aside from that, I
believe the semantics would be more or less the same.
> with some constructs like throw, and didn't have serializable
> continuations, but for some limited subset it might work well.
Am I correct in guessing that the reason they went to using an
interpreter was because of serialisation?
While we are on this topic, which Lisp implementations have
serialisable closures? I have (partial) code for clisp, and
probably SBCL. One of the vendors might have that too, so a
compiled arnesi might be more serialisable than we think.
s>>> Well, we don't have full continuations, but we have a
s>>> hackish continuations library in the the arnesi
s>>> library.
??>>
??>> the old arnesi had CPS-transformer ("compiler") rather than
s> Yes, I've read one of Marco's posts mentioning it, but have not
s> been able to find it. Is there a copy of this somewhere?
you can find it in arnesi darcs repository
(http://common-lisp.net/project/bese/repos/arnesi_dev/) near the initial
import (20050706).
however i'm not sure at which point it is actually working -- it appears i
was using some older snapshot (i've made some web framework on ABCL with
it).
i don't know if this version is available anywhere, if you wish i can send
it by email.
??>> interpreter. wouldn't that be better? i know it had problems
s> Do you mean faster when you say better?
yes, it should be faster, i think..
you have mentioned some 500/2000 "switches per second", this seems
freaking..
i'm not sure what and how are you testing, but simple loop:
(defun/cc loopcc () (loop for i upfrom 1 do (let/cc cc cc)))
(defun test-loop (n) (loop with lcc = (loopcc) repeat n do (setf lcc
(funcall lcc nil))))
runs something like 5 million iterations with SBCL on my computer, and same
speed works more practical generator function
(http://groups.google.com/group/comp.lang.lisp/msg/31d7edd1d878f44c).
as i understand switches you're talking about are semantically equivalent to
generator iterations (each task is a generator..), so compiled CPS is kinda
2000x faster?
s> Aside from that, I believe the semantics would be more or less the
s> same.
i don't know how the new one works; at least old one was easy, more-or-less
:)
and probably it's easy to modify it if needed
??>> with some constructs like throw, and didn't have serializable
??>> continuations, but for some limited subset it might work well.
s> Am I correct in guessing that the reason they went to using an
s> interpreter was because of serialisation?
i'm guessing with you :). besides serialization, i remember there were some
complaints about CPS-transformer not working correctly with some constructs
like non-local control transfer.
s> While we are on this topic, which Lisp implementations have
s> serialisable closures? I have (partial) code for clisp, and
s> probably SBCL. One of the vendors might have that too, so a
s> compiled arnesi might be more serialisable than we think.
honestly i don't even know how serializaton of closures/continuations does
work. does it mean if i have a bug i can serialize CC and post it to
newsgroup, so people can try to debug it? :)
From: Madhu
Subject: Re: Portable constructs or ideas for multi-tasking/multi-threading/co-routines ?
Date:
Message-ID: <m36404jfux.fsf@robolove.meer.net>
[My newsserver (downstream from verio) cannot find the OP's original
article <·····························@s15g2000prm.googlegroups.com>
for some reason]
* szergling
| jurgen_defurne wrote:
|
|> Where could I find information for making it easier to run
|> multiple tasks parallel in the same CL image ? These tasks
|> are linked very
|
| [[ ...]]
|
|> Thinking about this problem, I do not expect an easy
|> solution, because it comes down to writing a dispatch
|> process with cooperative multi- tasking, but maybe there
|> are sources of information which can make the
|
| [[...]]
|
| Well, we don't have full continuations, but we have a
| hackish continuations library in the the arnesi
| library. Using this, one can write a cooperative
| multitasking toolkit.
No need for continuations. [Just closures and] "A few pages of CLOS
code provide simple multiple processes scheduling."
Olivier Clarrise's miniproc package which is included in CL-HTTP as part
of its portable CLIM-SYS package is also available from
<URL:http://lisp.erire.com/>
(miniproc.tar.gz) worked under CLISP and CMUCL when I checked last a few
years ago.
You can browse the code at <URL:http://lisp.erire.com/miniproc/>
--
Madhu
From: Vesa Karvonen
Subject: Re: Portable constructs or ideas for multi-tasking/multi-threading/co-routines ?
Date:
Message-ID: <fhbrnt$nd5$1@oravannahka.helsinki.fi>
jurgen_defurne <··············@pandora.be> wrote:
[...]
> Thinking about this problem, I do not expect an easy solution, because
> it comes down to writing a dispatch process with cooperative multi-
> tasking, but maybe there are sources of information which can make the
> implementation of this part less complex.
You might be interested in the following message by Stephen Weeks on
asynchronous programming in SML:
http://mlton.org/pipermail/mlton-user/2006-July/000856.html
Yes, I know that the code is in SML, but the ideas should not be
difficult to translate to Common Lisp. I've written a few
applications using the techniques described in the above message, and
they seem to lead to quite manageable and readable code.
-Vesa Karvonen
From: Matthew D. Swank
Subject: Re: Portable constructs or ideas for multi-tasking/multi-threading/co-routines ?
Date:
Message-ID: <pan.2007.11.14.23.59.39.257998@gmail.com>
On Tue, 13 Nov 2007 09:45:33 +0000, Vesa Karvonen wrote:
> jurgen_defurne <··············@pandora.be> wrote:
> [...]
>> Thinking about this problem, I do not expect an easy solution, because
>> it comes down to writing a dispatch process with cooperative multi-
>> tasking, but maybe there are sources of information which can make the
>> implementation of this part less complex.
>
> You might be interested in the following message by Stephen Weeks on
> asynchronous programming in SML:
>
> http://mlton.org/pipermail/mlton-user/2006-July/000856.html
>
> Yes, I know that the code is in SML, but the ideas should not be
> difficult to translate to Common Lisp. I've written a few
> applications using the techniques described in the above message, and
> they seem to lead to quite manageable and readable code.
>
> -Vesa Karvonen
Well, if your going to go that route, you might consider something like
serve-event in sbcl or the portable equivalent in GBBOpen. One gotcha is
the use of special variables since they're likely as not to lose their
bindings. Which brings up the question --how hard would it be to provide
"event local" contexts in event loop style programming like specials can
provide in threaded programming?
Matt
--
"You do not really understand something unless you can
explain it to your grandmother." -- Albert Einstein.