Hello, All!
multiprocessing in cmucl 18e behaves very strange:
- when some function is running in a process toplevel repl is locked
- mp:with-timeout seems to be ignored
am i doing something wrong or it's just broken in cmucl?
where can i find documentation on cmucl mp?
and, by the way, about locks..
as i understand it's necessary for multithreaded application to do locking
when some data can be changed in multiple thread `simultaneously`. and it's
desired to have a lot of locks to avoid stalls. but performance can
degrade(or it may completely fall) if there are too many locks and they are
implemented as OS kernel objects. if lock is implemented as simple loop
there is no difference how many locks are there, but it can be less
efficient. so, how are locks implemented in cmucl, and how many is it ok to
have(orienting on Celeron1000 running RedHat7.3)?
With best regards, Alex Mizrahi.
>>>>> "am" == Alex Mizrahi <·········@xhotmail.com> writes:
am> multiprocessing in cmucl 18e behaves very strange:
am> - when some function is running in a process toplevel repl is locked
am> - mp:with-timeout seems to be ignored
things will run better if you have
#+mp (mp::startup-idle-and-top-level-loops)
in your ~/.cmucl-init file.
am> am i doing something wrong or it's just broken in cmucl?
am> where can i find documentation on cmucl mp?
the multiprocessing support in CMUCL is based on user-level coroutines
("green threads"), and not on kernel threads. It is only implemented
on x86 platforms, and is announced by the presence of the :mp
*feature*. Scheduling is cooperative: processes must explicitly yield
control, but input/output operations yield implicitly, so for many
purposes you don't need to insert explicit yield points. The
implementation is not very polished: interaction with the debugger and
with signals is imperfect. I wouldn't recommend relying on it for a
production system with many users.
There is no documentation other than the docstrings of symbols in the
MP package.
am> how are locks implemented in cmucl, and how many is it ok to
am> have(orienting on Celeron1000 running RedHat7.3)?
locks are represented in memory by a small structure, and use the
CMPXCHG compare-and-swap instruction, so they are quite efficient.
I expect that you'll run into scheduler limitations far more quickly
than problems with locks.
--
Eric Marsden <URL:http://www.laas.fr/~emarsden/>
(message (Hello 'Eric)
(you :wrote :on '(Wed, 15 Oct 2003 13:12:59 +0200))
(
am>> am i doing something wrong or it's just broken in cmucl?
am>> where can i find documentation on cmucl mp?
EM> the multiprocessing support in CMUCL is based on user-level
EM> coroutines ("green threads"), and not on kernel threads. It is only
EM> implemented on x86 platforms, and is announced by the presence of
EM> the :mp *feature*. Scheduling is cooperative: processes must
EM> explicitly yield control, but input/output operations yield
EM> implicitly, so for many purposes you don't need to insert explicit
EM> yield points. The implementation is not very polished: interaction
EM> with the debugger and with signals is imperfect. I wouldn't
EM> recommend relying on it for a production system with many users.
oh, that's bad.. while all goes ok it may be acceptable(and even good in
some points since i don't have to worry about locks all around), but some
long running or just infinite function that was not designed to be such can
ruin all the system.. i had to rerun lisp process 5 times yesterday to track
down that there is a bug - eternal loop - in plist-alist function in kmrcl
8-[
is there a free lisp for linux that supports real kernel threads? maybe sbcl
or ecl do?
)
(With-best-regards '(Alex Mizrahi) :aka 'killer_storm)
(prin1 "Jane dates only Lisp programmers"))
"Alex Mizrahi" <·········@xhotmail.com> writes:
>
> is there a free lisp for linux that supports real kernel threads? maybe sbcl
> or ecl do?
I think scieneer does.
On Thu, 16 Oct 2003 16:46:54 +0200, Alain Picard <·······················@optushome.com.au> wrote:
> "Alex Mizrahi" <·········@xhotmail.com> writes:
>
> > is there a free lisp for linux that supports real kernel threads?
> > maybe sbcl or ecl do?
>
> I think scieneer does.
Yes, but it's not free. SBCL (version 0.8.4 and higher) does, but
currently only on Linux x86. It still has some small rough edges but
it's quite usable already.
Edi.
Eric Marsden <········@laas.fr> writes:
> the multiprocessing support in CMUCL is based on user-level coroutines
> [...] The implementation is not very polished [...]
> I wouldn't recommend relying on it for a production system with many users.
Really? Two years ago I used CMUCL on x86 to implement a high-performance
web server via these multiprocessing threads. Seemed to work fine for me,
in production, with many users.
What failures would you have expected me to find?
_______________________________________________________________________________
Don Geddis http://don.geddis.org/ ···@geddis.org
Don Geddis <···@geddis.org> writes:
> Eric Marsden <········@laas.fr> writes:
> > the multiprocessing support in CMUCL is based on user-level coroutines
> > [...] The implementation is not very polished [...]
> > I wouldn't recommend relying on it for a production system with many users.
>
> Really? Two years ago I used CMUCL on x86 to implement a high-performance
> web server via these multiprocessing threads. Seemed to work fine for me,
> in production, with many users.
>
> What failures would you have expected me to find?
I have a web based report generator system that runs *nearly*
identically on ACL 6.2 on Windows (with native threads) and CMUCL
(Linux without native threads).. I do explicitly call mp:process-yield
(in CMU) and mp:process-allow-schedule (ACL) at certain points. I am
using Allegroserve / Portableallegroserve with some patches to support
Content-Disposition.
The biggest limitation of CMU is that during FFI calls, the FFI
function can't be interrupted. (At least this was the case as of 18c.
I haven't checked 18e since I just installed it recently.) So the rest
of the processes have to wait. (I use ClibPDF for PDF output; CL-PDF
was not ready when I began work on this project). Fortunately the PDF
FFI calls are effectively instantaneous, so there aren't any noticeable
pauses for the PDF functions. Unfortunately database FFI calls can be
much slower. I use FREETDS to talk to SQL Server and big queries can
be noticeable. Partly for this reason I deployed the system on Windows.
But I have comfortably used CMU for development and early testing with
no problems.
Jock Cooper
--
www.fractal-recursions.com
>>>>> "dg" == Don Geddis <···@geddis.org> writes:
ecm> the multiprocessing support in CMUCL is based on user-level coroutines
ecm> [...] The implementation is not very polished [...]
ecm> I wouldn't recommend relying on it for a production system with many users.
dg> Really? Two years ago I used CMUCL on x86 to implement a high-performance
dg> web server via these multiprocessing threads. Seemed to work fine for me,
dg> in production, with many users.
dg>
dg> What failures would you have expected me to find?
you're most likely to run into problems if you call out to foreign
libraries (even something simple like resolving a hostname will block
all threads), or have to use signals. CMUCL is also not very resistant
to networking errors. I'm glad it worked for you, though.
--
Eric Marsden <URL:http://www.laas.fr/~emarsden/>
Eric Marsden <········@laas.fr> writes:
> dg> What failures would you have expected me to find?
>
> you're most likely to run into problems if you call out to foreign
> libraries (even something simple like resolving a hostname will block
> all threads), or have to use signals. CMUCL is also not very resistant
> to networking errors. I'm glad it worked for you, though.
Note also that the compiler and fasloader and (possibly) the debugger
are all slightly thread-dubious. For a web server this probably
doesn't matter much: the only use of the compiler is probably to
create effective method definitions, and that doesn't happen often
enough to break anything. If you had twenty or so people telnetted
into repls and evaluating arbitrary Lisp, it's more likely you'd see
odd stuff.
-dan
--
http://web.metacircles.com/cirCLe_CD - Free Software Lisp/Linux distro
Eric Marsden wrote:
>>>>>>"am" == Alex Mizrahi <·········@xhotmail.com> writes:
>
>
> am> multiprocessing in cmucl 18e behaves very strange:
> am> - when some function is running in a process toplevel repl is locked
> am> - mp:with-timeout seems to be ignored
>
> things will run better if you have
>
> #+mp (mp::startup-idle-and-top-level-loops)
>
> in your ~/.cmucl-init file.
>
> am> am i doing something wrong or it's just broken in cmucl?
> am> where can i find documentation on cmucl mp?
>
> the multiprocessing support in CMUCL is based on user-level coroutines
> ("green threads"), and not on kernel threads. It is only implemented
> on x86 platforms, and is announced by the presence of the :mp
> *feature*. Scheduling is cooperative: processes must explicitly yield
> control, but input/output operations yield implicitly, so for many
> purposes you don't need to insert explicit yield points.
You can start process switcher by calling function
(mp::start-sigalrm-yield). After that test code runs as it should be.
(defun test ()
(mp::make-process #'testfunc2)
(mp::make-process #'testfunc1))
(defun testfunc1 ()
(mp::with-timeout (5) (loop))
(print "done"))
(defun testfunc2 ()
(dotimes (i 10)
(sleep 1)
(print i)))
--
Pozdrawiam - Rafal "nabla" Strzalinski
http://purl.org/nabla/start
>>>>> "rs" == Rafal Strzalinski <···············@o2.pl> writes:
ecm> Scheduling is cooperative: processes must explicitly yield
ecm> control, but input/output operations yield implicitly, so for
ecm> many purposes you don't need to insert explicit yield points.
rs> You can start process switcher by calling function
rs> (mp::start-sigalrm-yield). After that test code runs as it should be.
maybe your test code, but not mine!
,----
| (defun crash ()
| (setq *debugger-hook*
| (lambda (condition old-debugger-hook)
| (debug:backtrace 10)
| (unix:unix-exit 2)))
| #+live-dangerously
| (mp::start-sigalrm-yield)
| (flet ((roomy () (loop (with-output-to-string (*standard-output*) (room)))))
| (mp:make-process #'roomy)
| (mp:make-process #'roomy)))
`----
If I enable #+live-dangerously, CMUCL lives for around 3 seconds.
The sigalrm-driven scheduler is driven by interrupts, and parts of the
CMUCL runtime are not interrupt-safe. Now you know why that symbol
isn't exported from the MP package.
--
Eric Marsden <URL:http://www.laas.fr/~emarsden/>
Eric Marsden wrote:
>>>>>>"rs" == Rafal Strzalinski <···············@o2.pl> writes:
>
> If I enable #+live-dangerously, CMUCL lives for around 3 seconds.
> The sigalrm-driven scheduler is driven by interrupts, and parts of the
> CMUCL runtime are not interrupt-safe. Now you know why that symbol
> isn't exported from the MP package.
I know about that. I was hoping that docstring and comment is outdated :-).
--
Pozdrawiam - Rafal "nabla" Strzalinski
http://purl.org/nabla/start