Am I being stupid in not having noticed any threading
capabilities in cmucl? Or is it possible to invoke the
pthreads libraries safely from cmucl?
The reason I ask is (apart from simple curiousity) that
there have been a couple of mentions of using sockets in lisp,
and naturally when using sockets (at least on the server side),
one would want a separate I/O and execution thread (Well,
often, anyway).
--
You are on the way to destruction.
Marcin Tustin <·····@witch.monkeys> writes:
> Am I being stupid in not having noticed any threading >
> capabilities in cmucl?
My question about multi-threading in Lisps is this:
I've noticed that ACL and LW use native threads on Win32 platforms, but
under Linux they tend to use their own scheduler. Some discussion about
this with other developers suggests that this is problematic for two
reasons. The first, and obvious reason, is that deferring the context
switching to the OS, and letting the OS handle scheduling is probably
much quicker, because it's done in kernel space, and because you're
adding more scheduling responsibility to the Lisp environment under
Linux.
The second reason is that using foreign libraries that are thread-safe
(e.g. an ODBC library) makes good sense when coupled with a
multi-threaded Lisp. For example, if you're using an ODBC interface in
your favorite CL implementation that does multiprocessing with stack
groups and not native threads, then you can't expect your ODBC library
now to be thead-safe (or, `multi-processing-safe').
I wonder why the Lisp vendors seem to have avoided using native threads
on Linux. I'd have thought that Linux would have made it even easier to
use threads than Win32.
dave
In article <···············@no-knife.mit.edu>,
Dave Bakhash <·····@alum.mit.edu> wrote:
>I wonder why the Lisp vendors seem to have avoided using native threads
>on Linux. I'd have thought that Linux would have made it even easier to
>use threads than Win32.
My guess is that it's because they developed their threading
implementations before Unix systems had native threading, and completely
reimplementing it to take advantage of native threading has not been high
priority.
--
Barry Margolin, ······@genuity.net
Genuity, Woburn, MA
*** DON'T SEND TECHNICAL QUESTIONS DIRECTLY TO ME, post them to newsgroups.
Please DON'T copy followups to me -- I'll assume it wasn't posted to the group.
On 03 Jul 2002 16:49:45 -0400, Dave Bakhash <·····@alum.mit.edu>
wrote:
>I wonder why the Lisp vendors seem to have avoided using native threads
>on Linux. I'd have thought that Linux would have made it even easier to
>use threads than Win32.
Linux native threads are really processes disguised as threads.
>> I wonder why the Lisp vendors seem to have avoided using native threads
>> on Linux. I'd have thought that Linux would have made it even easier to
>> use threads than Win32.
> Linux native threads are really processes disguised as threads.
How do Win32 threads differ?
joelh
* Joel Ray Holveck wrote:
>>> I wonder why the Lisp vendors seem to have avoided using native threads
>>> on Linux. I'd have thought that Linux would have made it even easier to
>>> use threads than Win32.
>> Linux native threads are really processes disguised as threads.
> How do Win32 threads differ?
I don't know, but I imagine they are lighter weight. As the parent
post said, Linux threads (at least those in several commonly available
distributions (kernels? I dunno how you know which version of Linux is
which, but it will be the kernel/libc that counts I guess)) are
basically processes without the non-shared memory. So making a new
thread and various other thread operations are *really* slow and
expensive in resources compared to implementations with better
threading systems, where threads are much lighter-weight objects than
processes. Solaris for instance has much lighter-weight threads, as
I'm sure do any of the other big-machine commercial Unices. I don't
know about Windows threads, but I suspect they are similarly
lightweight things (if anything about Windows can be said to be
`lightweight').
But there are at least two other issues with threads:
If you're a commercial vendor, you're probably targetting several
Unixes (and if you're not, you probably are too). And of course
`standard Posix threads' will turn out not actually to be standard,
and especially not to be standard in the interesting and curious
places that mean your system will be randomly flaky on different OSs.
If you're a C/C++ person, then the approach is one of: not do the
things that result in random flakyness, such as sophisticated memory
management; just be randomly flaky - that's pretty much a requirement
for C/C++ systems, after all; or finally spend a huge amount of money
making your system not randomly flaky on different OSs. Spending huge
amounts of money is easy if you're Oracle, but not so easy if you're a
Lisp vendor with a market which is somewhat smaller, and clients who
whine if they have to pay more than $30 per license.
Of course on Windows you only have one target platform, so getting
threads to work is easier (so long as the myriad Windows variants
don't have too much variation).
Secondly, doing a native multithreaded system is *really* hard, since
all sorts of assumptions that a non-native system can make aren't true
any more. For instance several threads may run at once, so there are
now a million places you have to check for thread safety, and the
memory allocator and GC have to be redone to cope with this. And if
you want the system to run on larger machines, you suddenly have to
worry about making sure that GC scales - meaning the GC itself has to
be multithreaded so it can be scheduled on several CPUS - or you'll be
eaten alive by Amdahls law. Some of these problems (multithreaded GC)
are probably semi-research problems. As far as I know all the systems
that do native threads generally do so by locking the heap and thus
making sure that actually, only one thread can access the heap at
once, which is something that will obviously not wash on a 32CPU
box... Corman Lisp seems to be an honourable exception to this (but
on Windows only, so no posix-threads nightmare to deal with), and I
think Roger Corman is going to talk about this at the ALU Lisp
conference - I wish I had time to explore it and to go to the
conference...
Finally, lots of lisp programmers and programs suffer from `stack
group syndrome[1]' - they think it's quite reasonable to use
WITHOUT-PREEMPTION instead of mutexes & semaphores because `it's all
just simulated multithreading on a single processor', and argue for
DYNAMIC-WIND-type constructs because, of course, you need a handle on
thread switches so you can do all sorts of interesting and weird stuff
like the LispMs did. So there are likely lots of programs, including
commercially important ones, which are going to really suck in terms
of performance on larger machines (WITHOUT-PREEMPTION) or just not
work at all (DYNAMIC-WIND / LETF) if the underlying Lisp starts using
native threads `natively'. Of course those programs can't make use of
larger machines at the moment (at least, not with threads, maybe by
processes and IPC), but they do at least work. From a vendor's point
of view, coming out with a native-threaded system which causes your
clients' 20-year-old mission-critical code to fail in random ways
might not be really a good move. Of course, by not coming out with
such a system you're ruling out applications which scale painlessly to
really big enterprise machines (I don't count multiple processes and
IPC as painless, sorry), but given non-huge-sums of money to spend on
implementation it's certainly less risky to keep your existing
customers happy rather than screw them in favour of a market that you
don't currently have.
All this sounds pretty pessimistic, and it is I suppose. However as
far as I know there are at least two systems - Corman Lisp and another
one which is not (I think) public targeted at (I think) Solaris and/or
Linux which aim to do native threads and scaling, so the situation is
not perhaps as bleak as I paint it.
--tim
Footnotes:
[1] You can tell by the pustules.
Tim Bradshaw <···@cley.com> writes:
> * Joel Ray Holveck wrote:
> >>> I wonder why the Lisp vendors seem to have avoided using native threads
> >>> on Linux. I'd have thought that Linux would have made it even easier to
> >>> use threads than Win32.
> >> Linux native threads are really processes disguised as threads.
>
> > How do Win32 threads differ?
>
> I don't know, but I imagine they are lighter weight.
Win32 processes are really threads disguised as processes
(Well, sort of).
--
Oh yeah! Does anybody feel like messing with a see-through motherfucker?
Marcin Tustin <·····@witch.cheese> writes:
> Tim Bradshaw <···@cley.com> writes:
>
> > * Joel Ray Holveck wrote:
> > >>> I wonder why the Lisp vendors seem to have avoided using native threads
> > >>> on Linux. I'd have thought that Linux would have made it even easier to
> > >>> use threads than Win32.
> > >> Linux native threads are really processes disguised as threads.
> >
> > > How do Win32 threads differ?
> >
> > I don't know, but I imagine they are lighter weight.
>
> Win32 processes are really threads disguised as processes
Win32 processes are so amazingly heavy-weight that you *need*
threads. That is not the case on a reasonable Unix platform.
--
Raymond Wiker Mail: ·············@fast.no
Senior Software Engineer Web: http://www.fast.no/
Fast Search & Transfer ASA Phone: +47 23 01 11 60
P.O. Box 1677 Vika Fax: +47 35 54 87 99
NO-0120 Oslo, NORWAY Mob: +47 48 01 11 60
Try FAST Search: http://alltheweb.com/
Raymond Wiker <·············@fast.no> writes:
> Win32 processes are so amazingly heavy-weight that you *need*
> threads. That is not the case on a reasonable Unix platform.
Yawn!
The Win32 process object can be a heavy-weight entity because of the
pervasive use and intrinsic support for threads in Windows NT. This
was not the case for Unix.
Windows NT is by no means the first operating system (and certainly
not the last) which made this design trade-off. The consequences of
this decision have no bearing on whether the system is viable for
delivering successful applications.
* Raymond Wiker wrote:
> Win32 processes are so amazingly heavy-weight that you *need*
> threads. That is not the case on a reasonable Unix platform.
Rubbish. Not only is it the case, it's so much the case that Unix
platforms typically provide three layers - traditional Unix processes,
LWPs and userland threads with all sorts of flexibility as to how they
interact (in particular as to how userland threads get attached to
LWPs, which are the things that actually get scheduled onto a
processor).
--tim
Tim Bradshaw wrote:
>
> * Raymond Wiker wrote:
>
> > Win32 processes are so amazingly heavy-weight that you *need*
> > threads. That is not the case on a reasonable Unix platform.
>
> Rubbish. Not only is it the case, it's so much the case that Unix
> platforms typically provide three layers - traditional Unix processes,
> LWPs and userland threads with all sorts of flexibility as to how they
> interact (in particular as to how userland threads get attached to
> LWPs, which are the things that actually get scheduled onto a
> processor).
The three layers in Win32 are called "processes", "threads", and "fibers".
Though I haven't measured it myself, rumour has it that, on
identical hardware, a Linux process switch takes about the same
time as an NT thread switch.
Ian Wild <···@cfmu.eurocontrol.be> writes:
> Tim Bradshaw wrote:
> > Rubbish. Not only is it the case, it's so much the case that Unix
> > platforms typically provide three layers - traditional Unix processes,
> > LWPs and userland threads with all sorts of flexibility as to how they
> > interact (in particular as to how userland threads get attached to
> > LWPs, which are the things that actually get scheduled onto a
> > processor).
>
> The three layers in Win32 are called "processes", "threads", and "fibers".
Wrong. Win32 fibers are very different from UNIX user-land LWPs.
Fibers are cooperatively scheduled and require the programmer to call
SwitchToFiber() specifying another execution context to schedule.
This is very much in contrast to the common UNIX user-land LWPs (and
Lisp-on-UNIX threads) which switch tasks upon receipt of a signal at
preprogrammed intervals.
If you believe the literature, fibers were added to make Windows NT a
better porting target for UNIX software packages which had rolled
their own thread substrates.
Tim Bradshaw <···@cley.com> writes:
> * Joel Ray Holveck wrote:
> >>> I wonder why the Lisp vendors seem to have avoided using native threads
> >>> on Linux. I'd have thought that Linux would have made it even easier to
> >>> use threads than Win32.
> >> Linux native threads are really processes disguised as threads.
>
> > How do Win32 threads differ?
>
> I don't know, but I imagine they are lighter weight. As the parent
> post said, Linux threads (at least those in several commonly available
> distributions (kernels? I dunno how you know which version of Linux is
> which, but it will be the kernel/libc that counts I guess)) are
> basically processes without the non-shared memory. So making a new
> thread and various other thread operations are *really* slow and
> expensive in resources compared to implementations with better
> threading systems, where threads are much lighter-weight objects than
> processes.
[...]
It might be interesting to know...
Dr. Edward G. Bradford from IBM has made some benchmarks and claims
that in his tests Linux shows better performance (about 1.5-2 times) than
Windows 2000/XP in creating new threads.
(see http://www-106.ibm.com/developerworks/linux/library/l-rt7/)
He also compared speed of synchronization mechanisms in both systems
(see http://www-106.ibm.com/developerworks/linux/library/l-rt5/).
And made a conclusion that semaphores and mutexes in Linux faster the
similar mechanisms (semaphores and mutexes) in Windows 2000/XP in several
times. The only exception is CriticalSection that are fastest and faster
than pthread_mutexes in Linux in ~5 times.
But inspite of such the loss in a speed, if to compare posix_mutexes and
CriticalSections, Linux (as in this tests) still shows better performance
in a task of some fractal calculations in multithreaded environments (~
5%). (See http://www-106.ibm.com/developerworks/linux/library/l-rt8/)
In article <··············@list.ru>, Aleksandr Skobelev wrote:
> It might be interesting to know...
>
> Dr. Edward G. Bradford from IBM has made some benchmarks and claims
> that in his tests Linux shows better performance (about 1.5-2 times) than
> Windows 2000/XP in creating new threads.
> (see http://www-106.ibm.com/developerworks/linux/library/l-rt7/)
>
> He also compared speed of synchronization mechanisms in both systems
> (see http://www-106.ibm.com/developerworks/linux/library/l-rt5/).
> And made a conclusion that semaphores and mutexes in Linux faster the
> similar mechanisms (semaphores and mutexes) in Windows 2000/XP in several
> times. The only exception is CriticalSection that are fastest and faster
> than pthread_mutexes in Linux in ~5 times.
But note that critical sections are hand-tuned assembly language.
Also note that critical sections are braindamaged crap. When there is
contention for a critical section, it will dynamically allocate a new event
object so that the threads have a place to suspend. If your program has
thousands of critical sections, it may potentially consume thousands of
handles. What's worse, if an event cannot be allocated, a Win32 exception will
be thrown. The allocation of handles in critical sections makes it difficult to
check a Win32 executable for handle leaks. There is a way to initialize
critical sections so tha the handles are preallocated, which is something
I once had to hack into a multithreaded server so that I could check its
true handle usage over time.
Linux mutexes do not consume any kernel handles at all; they are purely
data objects that consume only memory.
Also note that Linux has several kinds of mutexes; the type of mutex
is selected by specifying an attribute in pthread_mutex_init.
These mutexes perform differently.
The default mutex type has some extra baggage because it supports the timed out
operation pthread_mutex_timedlock. More importantly, its unlock operation
implements a kind of fair behavior whereby if there are waiting threads, the
mutex is given to the highest priority thread which has been waiting the
longest, even if another thread is prime and ready to seize the mutex.
So there is a period of time during which the critical region is protected,
but nobody is in it, from the time that the current owner of the lock removes
the next waiting thread, and the time that thread is actually dispatched
and runs the critical region.
There is a mutex type in glibc 2.2 which does not implement this ``fair''
behavior; the unlock operation immediately marks the mutex as available so
another thread can grab it (perhaps on another processor, of if a context
switch takes place). A waiting thread is also woken, but the mutex is already
available when this is happening. The acquire operation on type of mutex also
performs spinning, if you have a multiprocessor system. The spin count is
dynamically measured and a smoothed average of it is stored in the lock itself,
and is used to set an upper limit on the number of spins. So the locks try to
dynamically ``learn'' about longer or shorter critical sections, so as to not
spin too long to acquire short ones, or give up too early on the longer ones.
Win32 critical sections do are not dynamic. On a multiprocessor you can
individually configure a constant value for the spin count for each critical
section. Choosing such values is, of course, hardware-specific guesswork.
Moreover, critical sections have no provision in their programming interface
to specify a lock type; there is only one flavor. You cannot choose a
critical section that has no recursive locking, or a critical section
that has error checking. Maybe one day Microsoft will add ``Ex'' to all the
function names and get the interface less wrong.
Kaz Kylheku <···@ashi.footprints.net> writes:
> But note that critical sections are hand-tuned assembly language.
Many UNIX systems have various concurrency primitives expand out into
in-lined assembler too. This seems like a perfectly reasonable design
decision to me. These operations ought to be fast; use any edge you
have to make them fast!
> Also note that critical sections are braindamaged crap. When there is
> contention for a critical section, it will dynamically allocate a new event
> object so that the threads have a place to suspend.
Most Win32 synchronization primitives use kernel objects so you can do
ultra-cool things like call WaitForMultipleObjects() on all sorts of
random objects and later have the kernel call you back when any one of
them become ready. UNIX systems don't have this kind of interface and
that is a real pity. Although if they ever do, I suspect UNIX
programmers will have to give similar consideration to kernel object
consumption just as Win32 programmers do now.
Anyway, if your style of programming obviates the need for this sort
of functionality, use a user-level lock primitive.
Carl Shapiro <·············@panix.com> writes:
> Most Win32 synchronization primitives use kernel objects so you can do
> ultra-cool things like call WaitForMultipleObjects() on all sorts of
> random objects and later have the kernel call you back when any one of
> them become ready. UNIX systems don't have this kind of interface and
> that is a real pity.
Unix is a multi-headed beast. FreeBSD and OpenBSD have kqueue(2)
which provides similar (though not identical) functionality. NetBSD
should have it soon. I'm not sure if/when Linux will get something
like it. The idea was discussed almost two years ago on LKML but it
petered out with the 2.4 freeze :-< Solaris has /dev/poll which is
fine if you can represent your object of interest as a file-descriptor
but otherwise won't help.
·······@dino.dnsalias.com (Stephen J. Bevan) writes:
> Unix is a multi-headed beast. FreeBSD and OpenBSD have kqueue(2)
> which provides similar (though not identical) functionality. NetBSD
> should have it soon.
Yes, I am aware of kqueues. Unfortunately, kqueues are only good for
selecting on a very restricted set of types in comparison to its
equivalent interface in Win32. This limitation is rather frustrating.
However, kqueues are definately a step in the right direction!
Carl Shapiro <·············@panix.com> writes:
> Yes, I am aware of kqueues. Unfortunately, kqueues are only good for
> selecting on a very restricted set of types in comparison to its
> equivalent interface in Win32. This limitation is rather frustrating.
> ...
It is clear that WaitForMultipleObjects supports a variety of objects
(though considerably less if your are programming for WinCE).
However, it isn't clear to which ones cause frustration when using
kqueue. Perhaps that's because I've been brainwashed by the Unix way
of doing things. Of the objects I use regularly (mutex, process,
thread, timer) I can see how to wait on them (though not necessarily
with kqueue) and of the ones I don't I'm not clear on why I'd want to
wait on them.
Carl Shapiro <·············@panix.com> writes:
[...]
>
> Most Win32 synchronization primitives use kernel objects so you can do
> ultra-cool things like call WaitForMultipleObjects() on all sorts of
> random objects and later have the kernel call you back when any one of
> them become ready. UNIX systems don't have this kind of interface and
> that is a real pity.
[...]
I'm not sure if this is a 'real pity'. It might be more convinient to use
WaitFor...() especially when you try to do somethings in a first time. In
the other hand there is a very limited realization of select() in
Windows. If not speak that this realization is just broken (at least
under windows 95/98) and it seems that MS isn't going to fix it.
In article <···············@panix3.panix.com>, Carl Shapiro wrote:
> Most Win32 synchronization primitives use kernel objects so you can do
> ultra-cool things like call WaitForMultipleObjects() on all sorts of
> random objects and later have the kernel call you back when any one of
> them become ready.
With condition variables, you don't need this. Your program evaluates the
predicates that it is interested in and waits on a single condition
variable. Because you open code the scope of your lock and your own
evaluation, you can implement ultra-cool synchronization things yourself.
WaitForMultipleObjects is also braindamaged crap that should be avoided,
expecially as a communication mechanism within one process.
Silly Windows programmers fall into the trap of using events for
signaling among modules of a program, instead of using function calls.
Effectively, the proprietary Win32 synchronization objects become part of the
interface definition of a module, e.g.: ``pass me me a Microsoft event handle,
and I will notify you by signaling it''. The semantics of a proprietary
operating system are conflated into the design of an interface that might
instead have been perfectly platform-independent with the approach ``pass me an
object and I will notify you by invoking a method on it''.
Of course, when you use events for interfacing you eventually encounter a
situation whereby you need to multiplex two signals. That's when the
Windows programmer reaches for WaitForMultipleObjects to bail him or her out.
This doesn't solve any actual problem; it's a workaround for one's own
stupidity.
"Kaz Kylheku" <···@ashi.footprints.net> wrote in message ·················@luna.vcn.bc.ca...
> In article <···············@panix3.panix.com>, Carl Shapiro wrote:
> > Most Win32 synchronization primitives use kernel objects so you can do
> > ultra-cool things like call WaitForMultipleObjects() on all sorts of
> > random objects and later have the kernel call you back when any one of
> > them become ready.
>
> With condition variables, you don't need this. Your program evaluates the
> predicates that it is interested in and waits on a single condition
> variable. Because you open code the scope of your lock and your own
> evaluation, you can implement ultra-cool synchronization things yourself.
Evaluates the predicates when? Every thread switch? Or every scheduling
cycle? This involves returning to the context of your thread just to
see if there is any work to do.
> WaitForMultipleObjects is also braindamaged crap that should be avoided,
> expecially as a communication mechanism within one process.
No doubt there are better intra-process communication mechanisms.
> Effectively, the proprietary Win32 synchronization objects become part of the
> interface definition of a module, e.g.: ``pass me me a Microsoft event handle,
> and I will notify you by signaling it''. The semantics of a proprietary
> operating system are conflated into the design of an interface that might
> instead have been perfectly platform-independent with the approach ``pass me an
> object and I will notify you by invoking a method on it''.
How does an `event handle' differ from an `object'? How does `signalling' differ
from `invoking a method'?
On 04 Jul 2002 11:17:52 +0100, Tim Bradshaw <···@cley.com> wrote:
>* Joel Ray Holveck wrote:
>>>> I wonder why the Lisp vendors seem to have avoided using native threads
>>>> on Linux. I'd have thought that Linux would have made it even easier to
>>>> use threads than Win32.
>>> Linux native threads are really processes disguised as threads.
>
>> How do Win32 threads differ?
>
>I don't know, but I imagine they are lighter weight. As the parent
>post said, Linux threads (at least those in several commonly available
>distributions (kernels? I dunno how you know which version of Linux is
>which, but it will be the kernel/libc that counts I guess)) are
>basically processes without the non-shared memory. So making a new
>thread and various other thread operations are *really* slow and
>expensive in resources compared to implementations with better
>threading systems, where threads are much lighter-weight objects than
>processes. Solaris for instance has much lighter-weight threads, as
>I'm sure do any of the other big-machine commercial Unices. I don't
>know about Windows threads, but I suspect they are similarly
>lightweight things (if anything about Windows can be said to be
>`lightweight').
Yes, threads are very light-weight under Windows (as long as you don't
use MFC or some other Microsoft technology that changes that ;-)
A process has quite a bit more overhead, although it is basically a
thread with non-shared memory and various other baggage attached.
If we look at Oracle as an example, under Unix it runs each database
connection as a separate process. Under Win32 there is only one
process, with many separate threads in a pool. This is a common
implementation strategy, used by many servers, and represents the
relative difference of thread/process utility in Unix vs. Win32
operating systems. (BTW, I prefer Unix Oracle because the think the
multi-process design is more robust.)
>
>But there are at least two other issues with threads:
>
>If you're a commercial vendor, you're probably targetting several
>Unixes (and if you're not, you probably are too). And of course
>`standard Posix threads' will turn out not actually to be standard,
>and especially not to be standard in the interesting and curious
>places that mean your system will be randomly flaky on different OSs.
>If you're a C/C++ person, then the approach is one of: not do the
>things that result in random flakyness, such as sophisticated memory
>management; just be randomly flaky - that's pretty much a requirement
>for C/C++ systems, after all; or finally spend a huge amount of money
>making your system not randomly flaky on different OSs. Spending huge
>amounts of money is easy if you're Oracle, but not so easy if you're a
>Lisp vendor with a market which is somewhat smaller, and clients who
>whine if they have to pay more than $30 per license.
>
>Of course on Windows you only have one target platform, so getting
>threads to work is easier (so long as the myriad Windows variants
>don't have too much variation).
Fortunately the differences are fairly minor. Microsoft did a decent
job in this area. As long as you stick to the subset of functionality
that is shared by all, which is pretty good.
>
>Secondly, doing a native multithreaded system is *really* hard, since
>all sorts of assumptions that a non-native system can make aren't true
>any more. For instance several threads may run at once, so there are
>now a million places you have to check for thread safety, and the
>memory allocator and GC have to be redone to cope with this. And if
>you want the system to run on larger machines, you suddenly have to
>worry about making sure that GC scales - meaning the GC itself has to
>be multithreaded so it can be scheduled on several CPUS - or you'll be
>eaten alive by Amdahls law. Some of these problems (multithreaded GC)
>are probably semi-research problems. As far as I know all the systems
>that do native threads generally do so by locking the heap and thus
>making sure that actually, only one thread can access the heap at
>once, which is something that will obviously not wash on a 32CPU
>box... Corman Lisp seems to be an honourable exception to this (but
>on Windows only, so no posix-threads nightmare to deal with), and I
>think Roger Corman is going to talk about this at the ALU Lisp
>conference - I wish I had time to explore it and to go to the
>conference...
I am looking forward to this talk--sorry you won't be there, Tim.
>
>Finally, lots of lisp programmers and programs suffer from `stack
>group syndrome[1]' - they think it's quite reasonable to use
>WITHOUT-PREEMPTION instead of mutexes & semaphores because `it's all
>just simulated multithreading on a single processor', and argue for
>DYNAMIC-WIND-type constructs because, of course, you need a handle on
>thread switches so you can do all sorts of interesting and weird stuff
>like the LispMs did. So there are likely lots of programs, including
>commercially important ones, which are going to really suck in terms
>of performance on larger machines (WITHOUT-PREEMPTION) or just not
>work at all (DYNAMIC-WIND / LETF) if the underlying Lisp starts using
>native threads `natively'. Of course those programs can't make use of
>larger machines at the moment (at least, not with threads, maybe by
>processes and IPC), but they do at least work. From a vendor's point
>of view, coming out with a native-threaded system which causes your
>clients' 20-year-old mission-critical code to fail in random ways
>might not be really a good move. Of course, by not coming out with
>such a system you're ruling out applications which scale painlessly to
>really big enterprise machines (I don't count multiple processes and
>IPC as painless, sorry), but given non-huge-sums of money to spend on
>implementation it's certainly less risky to keep your existing
>customers happy rather than screw them in favour of a market that you
>don't currently have.
The WITHOUT-PREEMPTION stuff drives me nuts. I was taught (OK, taught
myself mainly) to think about threads as logically all running
simultaneously. If you plan on that, then everything will work fine
with or without multiple processors. If you assume preemption, and
assume single-thread-at-a-time, you have painted yourself into a
corner. It's not that hard to get the hang of synchronization, once
you realize you're stuck with it. Your coding style can go a long way
to eliminate problems. In Corman Lisp, any special variable can be
used as a per thread "global" variable, which makes it easy to
restrict data sharing to the few objects that actually require it
(because they fundamentally participate in the parallel processing
logic). I think leveraging special variables in this way actually
makes Common Lisp the easiest language I have used for multi-threaded
programming. In Java you pretty much had to avoid static variables
completely (now they have a clumsy workaround), whereas in Corman Lisp
I just bind all the special variables a thread will be using and off
it goes.
Corman Lisp was designed throughout to support OS-level, multi-cpu
threading. It assumes every thread runs at once. To implement
without-preemption you have to suspend other processes, and then
resume them. This turns what should presumably be a light-weight
operation into a heavyweight operation, with many kernel calls.
For this reason I have resisted trying to copy other lisp vendors'
implementationsand APIs. If there is one benefit to being a small
company establishing a new client base, it is the ability to do things
the way you think they ought to be done. We don't feel so burdened by
history.
I ought to mention that if it wasn't for some correspondence with
Rainer Joswig at an opportune time (early in Corman Lisp development)
we probably would not have implemented OS native multi-threading. He
told me if I was developing a new lisp system it ought to be there,
and I completely agreed (partly because I have a lot of respect for
Rainer). Adding it involved all kinds of redesigning, even at that
stage. I couldn't imagine taking that on now. It affected every bit of
the system, including the garbage collector, the run time model,
foreign interfaces, etc. etc. Implementing a multi-threaded lisp
system is a significant undertaking. I am not sure if retrofitting an
existing system is even feasible.
We used some tricks to allow the garbage collector to work despite the
chance it can be called between any arbitrary pair of instructions on
any thread. I will talk some about this at the conference (and many
other threading issues).
Roger
On 03 Jul 2002 16:49:45 -0400, Dave Bakhash <·····@alum.mit.edu> wrote:
> I wonder why the Lisp vendors seem to have avoided using native threads
> on Linux. I'd have thought that Linux would have made it even easier to
Duane Rettig explained here why ACL doesn't use pthreads on Linux. You may
try searching the comp.lang.lisp archive.
Paolo
--
EncyCMUCLopedia * Extensive collection of CMU Common Lisp documentation
http://www.paoloamoroso.it/ency/README
Marcin Tustin <·····@witch.monkeys> writes:
> Am I being stupid in not having noticed any threading
> capabilities in cmucl? Or is it possible to invoke the pthreads
> libraries safely from cmucl?
See <index.html#threads> in Paolo Amoroso's EncyCMUCLopedia
<http://www.paoloamoroso.it/ency/README>. The relevant part is:
"CMU CL provides a multi-processing extension via threads,
currently available only on the x86 port, based on the model
implemented by the CLIM graphical user interface manager for
Common Lisp.
This document, available at the Web site of Xanalys, is the
section of the CLIM manual that illustrates the functions for
creating, destroying, accessing and waiting for processes. Notice
that CMU CL provides other functions not discussed in the
document, and that its code base is not too interrupt safe
yet. The interrupt safety on the x86 port, however, is improving."
The document this text is referring to is
<http://www.xanalys.com/software_tools/reference/lwu41/climuser/GUID_335.HTM#HEADING335-0>.
Edi.
Marcin Tustin <·····@witch.monkeys> writes:
> Am I being stupid in not having noticed any threading
> capabilities in cmucl? Or is it possible to invoke the
> pthreads libraries safely from cmucl?
> The reason I ask is (apart from simple curiousity) that
> there have been a couple of mentions of using sockets in lisp,
> and naturally when using sockets (at least on the server side),
> one would want a separate I/O and execution thread (Well,
> often, anyway).
CMUCL has threading, but it uses it's own scheduler, so you have to be
careful when making certain system calls (e.g., blocking socket
operations). I think I've heard there's someone working on a
commercial fork with a shiny new interrupt-safe library that'll play
well with native threads--that'll be interesting (and hopefully
affordable :)).
If I wanted to write a server using CMUCL, I'd forget threads
entirely: CMUCL provides an event system that provides a fairly nice
wrapper around the select(2) system call. Read the manual sections or
documentation (they all have good docstrings) on SERVE-EVENT,
ADD-FD-HANDLER, REMOVE-FD-HANDLER, INVALIDATE-DESCRIPTOR,
WITH-FD-HANDLER, etc.
Gabe Garza
> If I wanted to write a server using CMUCL, I'd forget threads
> entirely: CMUCL provides an event system that provides a fairly nice
> wrapper around the select(2) system call. Read the manual sections or
> documentation (they all have good docstrings) on SERVE-EVENT,
> ADD-FD-HANDLER, REMOVE-FD-HANDLER, INVALIDATE-DESCRIPTOR,
> WITH-FD-HANDLER, etc.
I'm using that in one project, but I'm moving away from it to CMUCL's
MT. I haven't verified the following, but I figure it's got one of
the two problems:
1. Everything's on the same stack. (This is what I think is
happening.) This means that if A calls B, B blocks for I/O, and
M's handler is called, then M must return before B can continue.
-or-
2. Separate special bindings aren't being kept for each running
handler.
AFAICT, situation 1 is the case, but I haven't spent much time
verifying this.
Cheers,
joelh