Hi, I need to spawn a lisp process for each element in a
list (you could say it's a "map-process" function...), but
the number of processes running at any time must be limited,
let's say, to 10. When using Pthreads, I normally would use
a condition variable to signal the parent thread every time
one child thread had finished, but I can't find that
mechanism in Liquid. It seems to me that I'll have to poll a
variable, what I wouldn't like to do...
Is there a similar mechanism to condition variables in
Liquid and I'm being only unable to find it? Sorry if the
question is stupid...
I'm using Liquid Common Lisp Version 5.0.7 in Solaris 2.6.
Thanks for any help,
Luis
From: Tim Bradshaw
Subject: Re: Condition Variable in Liquid Common Lisp V5.0.7?
Date:
Message-ID: <ey3y9pfvmtj.fsf@cley.com>
* Luis Malheiro wrote:
> Hi, I need to spawn a lisp process for each element in a
> list (you could say it's a "map-process" function...), but
> the number of processes running at any time must be limited,
> let's say, to 10. When using Pthreads, I normally would use
> a condition variable to signal the parent thread every time
> one child thread had finished, but I can't find that
> mechanism in Liquid. It seems to me that I'll have to poll a
> variable, what I wouldn't like to do...
I think the trick you can do is to use a wait function for the
`parent' process which checks some variable set by a child. The trick
is that the wait function runs in the scheduler, so you don't need to
need to worry that anyone else will alter the variable there. The
worker threads then need to use something like
WITH-SCHEDULING-INHIBITED around the change of value of the variable
to prevent the scheduler running the parent's wait function.
This really is polling, but at least you get to let the system do the
polling for you. It's pretty obviously a horrible technique, but it
kind of works.
--tim
From: Luis Malheiro
Subject: Re: Condition Variable in Liquid Common Lisp V5.0.7?
Date:
Message-ID: <3B5C7C2A.B79013DA@alcatel.co.uk>
Tim Bradshaw wrote:
>
> I think the trick you can do is to use a wait function for the
> `parent' process which checks some variable set by a child. The trick
> is that the wait function runs in the scheduler, so you don't need to
> need to worry that anyone else will alter the variable there. The
> worker threads then need to use something like
> WITH-SCHEDULING-INHIBITED around the change of value of the variable
> to prevent the scheduler running the parent's wait function.
>
> This really is polling, but at least you get to let the system do the
> polling for you. It's pretty obviously a horrible technique, but it
> kind of works.
>
> --tim
OK, I came up with the following:
(defun mapprocess (the-function the-list &key (nthreads 10))
(assert (> nthreads 0))
(let ((current-threads 0)
(thread-counter 0)
(what (format nil "MAPPROCESS WAIT (~A)"
nthreads)))
(dolist (elt the-list)
(process-wait what #'(lambda ()
(< current-threads nthreads)))
(with-scheduling-inhibited
(incf current-threads))
(make-process :name (format nil "MAPPROCESS ~A SPAWNED BY ~S "
(incf thread-counter) *current-process*)
:function #'(lambda (elt)
(unwind-protect
(funcall the-function elt))
(with-scheduling-inhibited
(decf current-threads)))
:args (list elt)))))
I think this works, but it can't be used as a general
technique because (I believe) there's no guarantee that the
condition is going to stay valid from when we return from
"process-wait" until when a "with-scheduling-inhibited" or a
"process-lock" starts, in this particular case it works
because the parent thread is the only one able to violate
the condition when changing it, the children keep the
condition when updating it.
It occurred to me, that it could be
possible to build a condition variable and avoid the
overhead in the scheduler. It seems that a "process-lock"
can be done in the name of another process than the one
running (at least in Liquid), so if a condition variable is
a class which has a queue of waiting processes and a lock,
the processes can be put in inactive state when waiting on
the variable and than they release the lock associated with
the variable, as in Pthreads. Later, a process can be woke
from the queue when the condition is signalled by taking the
same lock in its name just before it is put in active
state. In that way the scheduler doesn't bother to check if
the process is ready to run or not and the process returns
from the wait with the lock acquired. Does it sound too
stupid? Maybe I'm being to influenced by Pthreads. Well,
I'll have a go on it after I have some time.
Thanks for the help.
Luis
From: Tim Bradshaw
Subject: Re: Condition Variable in Liquid Common Lisp V5.0.7?
Date:
Message-ID: <ey3d76pg0mn.fsf@cley.com>
* Luis Malheiro wrote:
> OK, I came up with the following:
[Elided]
I'd thought of doing something like this (this is untested code, I
don't have a Liquid implementation to play with!). This also doesn't
really do the same thing you did, but ...
(defun make-condition-flag (&optional (initial-state nil))
;; car is the place to lock, cdr is the value
(cons nil initial-state))
(defmacro waiting-for-condition-flag ((flag &optional (new-value nil))
&body stuff-to-do)
;; wait for FLAG to be true setting it to NEW-VALUE, and doing
;; STUFF-TO-DO. setting the new value and STUFF-TO-DO are protected
;; by a lock on FLAG.
(let ((fname (make-symbol "FLAG")))
`(let ((,fname ,flag))
(unwind-protect
(progn
(process-wait "Waiting for a flag"
#'(lambda ()
(if (cdr flag)
(progn
;; this might not work in the
;; scheduler...
(process-lock (car flag))
t)
nil)))
,@stuff-to-do)
;; unwinding: assign flag state and clear lock
(setf (cdr flag) ,new-value)
(process-unlock (car flag))))))
(defun set-condition-flag (flag &optional (value t))
(with-process-lock ((car flag))
(setf (cdr flag) value)))
;;; Silly example
(defun respawning-function (function)
;; spawn a process running FUNCTION then wait for it to exit and
;; respawn. There are better ways of doing this!
(loop with flag = (make-condition-flag)
(make-process :function #'(lambda ()
(unwind-protect
(funcall function)
(set-condition-flag flag))))
(waiting-for-condition-flag (flag)
(format *debug-io* "~&Respawning...~%"))))
[Not responding to the other part because it's 4AM and I am going to
sleep...]
--tim
From: ···@itasoftware.com
Subject: Re: Condition Variable in Liquid Common Lisp V5.0.7?
Date:
Message-ID: <g0bl2fbe.fsf@itasoftware.com>
I wouldn't try a process-lock from within the scheduler.
I *think* it will signal a deadlock (it ought to!)
What you *can* do, however, is mutate something in the wait
function.
(defvar *process-count* 0)
(process-wait "Waiting to use a process."
(lambda () (when (< *process-count* 10)
(incf *process-count*)
t)))
This (at least in Lucid) won't have a race condition
between testing the count and incrementing it.
Tim Bradshaw <···@cley.com> writes:
> * Luis Malheiro wrote:
>
> > OK, I came up with the following:
>
> [Elided]
>
> I'd thought of doing something like this (this is untested code, I
> don't have a Liquid implementation to play with!). This also doesn't
> really do the same thing you did, but ...
>
>
> (defun make-condition-flag (&optional (initial-state nil))
> ;; car is the place to lock, cdr is the value
> (cons nil initial-state))
>
> (defmacro waiting-for-condition-flag ((flag &optional (new-value nil))
> &body stuff-to-do)
> ;; wait for FLAG to be true setting it to NEW-VALUE, and doing
> ;; STUFF-TO-DO. setting the new value and STUFF-TO-DO are protected
> ;; by a lock on FLAG.
> (let ((fname (make-symbol "FLAG")))
> `(let ((,fname ,flag))
> (unwind-protect
> (progn
> (process-wait "Waiting for a flag"
> #'(lambda ()
> (if (cdr flag)
> (progn
> ;; this might not work in the
> ;; scheduler...
> (process-lock (car flag))
> t)
> nil)))
> ,@stuff-to-do)
> ;; unwinding: assign flag state and clear lock
> (setf (cdr flag) ,new-value)
> (process-unlock (car flag))))))
>
> (defun set-condition-flag (flag &optional (value t))
> (with-process-lock ((car flag))
> (setf (cdr flag) value)))
>
>
> ;;; Silly example
>
> (defun respawning-function (function)
> ;; spawn a process running FUNCTION then wait for it to exit and
> ;; respawn. There are better ways of doing this!
> (loop with flag = (make-condition-flag)
> (make-process :function #'(lambda ()
> (unwind-protect
> (funcall function)
> (set-condition-flag flag))))
> (waiting-for-condition-flag (flag)
> (format *debug-io* "~&Respawning...~%"))))
>
> [Not responding to the other part because it's 4AM and I am going to
> sleep...]
>
> --tim