I have a feeling that this is something easy to do, but my
understanding of macros doesn't go far enough for me to do it.
If i have a list of file-names: '("file1.txt" "file2.txt" "file3.txt")
is it possible to write a macro that takes this list and returns a
concatenated-stream which will read from each of the files? Even
better, is it possible to create a wrapper similar to
"with-open-file", so the files would be automatically closed at the
end of the block?
(with-read-from-file '("file1" "file2" "file3")
(do something))
Thanks for any ideas,
glauber
··········@my-deja.com (glauber) writes:
> I have a feeling that this is something easy to do, but my
> understanding of macros doesn't go far enough for me to do it.
>
> If i have a list of file-names: '("file1.txt" "file2.txt" "file3.txt")
>
> is it possible to write a macro that takes this list and returns a
> concatenated-stream which will read from each of the files? Even
> better, is it possible to create a wrapper similar to
> "with-open-file", so the files would be automatically closed at the
> end of the block?
>
> (with-read-from-file '("file1" "file2" "file3")
> (do something))
>
> Thanks for any ideas,
I imagine something like the following is what you want.
For macros like this that do a lot of work, I find the trick is to
quickly convert them to something functional and then program them
like a regular function. (You do a lot less gensym'ing this way, and
you also are able to patch this (in case of bugs) easily since you can
just patch the function in most cases, leaving the calls unaffected.)
I didn't test it. I'll leave debugging it as an exercise.
(defmacro with-open-files-concatenated ((streamvar files-list-exp
&rest open-options)
&body forms)
`(call-with-open-files-concatenated #'(lambda (,stream-var) ,@forms)
,files-list-exp
,@open-options))
(defun call-with-open-files-concatenated (function files &rest open-options)
(let ((clean-exit nil) (stream nil) (streams '()))
(unwind-protect
(progn (dolist (file files)
;; Setting this instantly to a variable helps narrow the
;; window for timing error here. We can't narrow it
;; fully, but we can get it pretty small.
(setq stream (apply #'open file open-options))
(push stream streams))
(multiple-value-prog1
(with-open-stream (joined (apply #'make-concatenated-stream
;; Use REVERSE not NREVERSE
;; to avoid dropping pointer
;; to streams that must close.
(reverse streams)))
(funcall function joined))
(setq clean-exit t)))
(when stream (close stream :abort (not clean-exit)))
(when streams
(dolist (stream streams)
(close stream :abort (not clean-exit)))))))
>>>>> "Kent" == Kent M Pitman <······@world.std.com> writes:
Kent> (defun call-with-open-files-concatenated (function files &rest open-options)
Kent> (let ((clean-exit nil) (stream nil) (streams '()))
Kent> (unwind-protect
Kent> (progn (dolist (file files)
Kent> ;; Setting this instantly to a variable helps narrow the
Kent> ;; window for timing error here. We can't narrow it
Kent> ;; fully, but we can get it pretty small.
Kent> (setq stream (apply #'open file open-options))
Kent> (push stream streams))
Kent> (multiple-value-prog1
Kent> (with-open-stream (joined (apply #'make-concatenated-stream
Kent> ;; Use REVERSE not NREVERSE
Kent> ;; to avoid dropping pointer
Kent> ;; to streams that must close.
Kent> (reverse streams)))
Can you explain why the comment holds? I'm just curious why nreverse
would drop the pointer to streams that must close.
Ray
Raymond Toy <···@rtp.ericsson.se> writes:
> Can you explain why the comment holds? I'm just curious why nreverse
> would drop the pointer to streams that must close.
Kent M Pitman <······@world.std.com> writes:
> (defun call-with-open-files-concatenated (function files &rest open-options)
> (let ((clean-exit nil) (stream nil) (streams '()))
> (unwind-protect
> (progn (dolist (file files)
> ;; Setting this instantly to a variable helps narrow the
> ;; window for timing error here. We can't narrow it
> ;; fully, but we can get it pretty small.
> (setq stream (apply #'open file open-options))
> (push stream streams))
> (multiple-value-prog1
> (with-open-stream (joined (apply #'make-concatenated-stream
> ;; Use REVERSE not NREVERSE
> ;; to avoid dropping pointer
> ;; to streams that must close.
> (reverse streams)))
^^^^^^^^^^^^^^^^^
> (funcall function joined))
> (setq clean-exit t)))
> (when stream (close stream :abort (not clean-exit)))
> (when streams
> (dolist (stream streams)
^^^^^^^^^^^^^^^^^^^^^^^^^^
> (close stream :abort (not clean-exit)))))))
If he used nreverse above, it would no longer be safe to use `streams'
below, where they're closed. Another alternative would have been to
do `(setf streams (nreverse streams))' right before the
multiple-value-prog1.
>>>>> "Thomas" == Thomas F Burdick <···@conquest.OCF.Berkeley.EDU> writes:
Thomas> If he used nreverse above, it would no longer be safe to use `streams'
Thomas> below, where they're closed. Another alternative would have been to
Thomas> do `(setf streams (nreverse streams))' right before the
Thomas> multiple-value-prog1.
Of course! I just read the comment and didn't read in detail the
following stuff. It's obvious now.
Ray
Raymond Toy <···@rtp.ericsson.se> writes:
>
> >>>>> "Thomas" == Thomas F Burdick <···@conquest.OCF.Berkeley.EDU> writes:
>
> Thomas> If he used nreverse above, it would no longer be safe to use `streams'
> Thomas> below, where they're closed. Another alternative would have been to
> Thomas> do `(setf streams (nreverse streams))' right before the
> Thomas> multiple-value-prog1.
>
> Of course! I just read the comment and didn't read in detail the
> following stuff. It's obvious now.
Except that you can't use the (setq streams (nreverse streams))
suggested above because if the nreverse operation is interrupted in the
middle, the variable streams may only point to some of the streams,
not all of them.
Kent M Pitman <······@world.std.com> writes:
> Raymond Toy <···@rtp.ericsson.se> writes:
>
> >
> > >>>>> "Thomas" == Thomas F Burdick <···@conquest.OCF.Berkeley.EDU> writes:
> >
> > Thomas> If he used nreverse above, it would no longer be safe to use `streams'
> > Thomas> below, where they're closed. Another alternative would have been to
> > Thomas> do `(setf streams (nreverse streams))' right before the
> > Thomas> multiple-value-prog1.
> >
> > Of course! I just read the comment and didn't read in detail the
> > following stuff. It's obvious now.
>
> Except that you can't use the (setq streams (nreverse streams))
> suggested above because if the nreverse operation is interrupted in the
> middle, the variable streams may only point to some of the streams,
> not all of them.
Is there any way this could happen in a non-multiprocessing
environment? I can't think of any. (I generally assume code is not
threadsafe unless explicitly designed to be).
···@conquest.OCF.Berkeley.EDU (Thomas F. Burdick) writes:
> Kent M Pitman <······@world.std.com> writes:
>
> > Raymond Toy <···@rtp.ericsson.se> writes:
> >
> > >
> > > >>>>> "Thomas" == Thomas F Burdick
> > > <···@conquest.OCF.Berkeley.EDU> writes:
> > >
> > > Thomas> If he used nreverse above, it would no longer be
> > > Thomas> safe to use `streams' below, where they're closed.
> > > Thomas> Another alternative would have been to do `(setf
> > > Thomas> streams (nreverse streams))' right before the
> > > Thomas> multiple-value-prog1.
> > >
> > > Of course! I just read the comment and didn't read in detail
> > > the following stuff. It's obvious now.
> >
> > Except that you can't use the (setq streams (nreverse streams))
> > suggested above because if the nreverse operation is interrupted
> > in the middle, the variable streams may only point to some of the
> > streams, not all of them.
>
> Is there any way this could happen in a non-multiprocessing
> environment? I can't think of any. (I generally assume code is not
> threadsafe unless explicitly designed to be).
Yes. Just about all modern computers are secretly multiprocessing.
Consider keyboard interrupts. Even a non-multiprocessing Lisp usually
lets you force either a "breakpoint" (recursive Read-Eval-Print loop)
and I think all of them allow "abort to toplevel". The latter unwinds
and one of the critical features of Lisp not mirrored in many other
so-called reasonble languaes is unwind-protect.
In fact, if I had to ask only one question to determine whether a
language I was being given was suitable for me to do real work in,
I've generally said "does it have unwind-protect?" Sure, this isn't
enough to get the work done. But its presence is the best clue I've
seen that the designers understand the concept of "data integrity".
And people who are in this right frame of mind tend to do the other
parts of the languages right, too, in my experience. Once you
sacrifice data integrity, you really are at a remarkable disadantage
for debugging.
Unwind-protect is present in Multics PL/1, ITS Teco, Maclisp, Lisp
Machine Lisp, Common Lisp, and Java. (It's conspicuously absent in
Scheme, btw, and not at all by accident.)
Kent M Pitman <······@world.std.com> writes:
> ···@conquest.OCF.Berkeley.EDU (Thomas F. Burdick) writes:
>
> > Kent M Pitman <······@world.std.com> writes:
> >
> > ...
> > > Except that you can't use the (setq streams (nreverse streams))
> > > suggested above because if the nreverse operation is interrupted
> > > in the middle, the variable streams may only point to some of the
> > > streams, not all of them.
> >
> > Is there any way this could happen in a non-multiprocessing
> > environment? I can't think of any. (I generally assume code is not
> > threadsafe unless explicitly designed to be).
>
> Yes. Just about all modern computers are secretly multiprocessing.
> Consider keyboard interrupts. Even a non-multiprocessing Lisp usually
> lets you force either a "breakpoint" (recursive Read-Eval-Print loop)
> and I think all of them allow "abort to toplevel". The latter unwinds
> and one of the critical features of Lisp not mirrored in many other
> so-called reasonble languaes is unwind-protect.
Btw, just to expand on this a bit more...
Maclisp didn't ever have WITH-OPEN-FILE but many people used a library I
wrote called IOTA. (I named it that in keeping with the use of the greek
letter LAMBDA for binding variables; IOTA was for binding "i/o" ... and
I also had a macro PI for binding "program interrupts"). IOTA was pretty
much WITH-OPEN-FILE except it took an extra level of parens around the
bindings (like with LET) so you could open more than one file in a single
form. PI was pretty much exactly WITHOUT-INTERRUPTS, just a different name.
The reason I came up with IOTA in the first place, though, was that Maclisp
(which wasn't multi-tasking, btw) didn't close open files until the GC
ran and discovered no one was pointing to them. On the ITS operating system,
there were some small number of available channels to files (like 30 or so?),
and I was locking up a dozen or so of them on a system that was supporting
20 or 30 people at once, well more than my fair share. Someone came and
complained to me that I was holding them open and I was offended that anyone
expected me to close them when the system wasn't trying very hard to close
them. Somewhere around then, either in response to that, or coincidentally
at the same time, UNWIND-PROTECT went into Maclisp and I wrote IOTA to help
keep people from yelling at me.
So these issues of what happens on an unwind to do cleanup in a
non-multi-tasking Lisp [Maclisp] do actually matter...
The Lisp Machine, which of course did have multi-tasking, added
WITH-OPEN-FILE at pretty much the same time as Maclisp got IOTA, but I
think it was independent discovery. That happened a lot. Sometimes
the time is just ripe and more than one person is bound to trip over a
good idea. (Another reason software patents are such an unfair loss.)
- - - - -
One other funny anecdote, vaguely related to this, was that my first
attempt at implementing PI was something like this [modulo the fact that
I can't recall if we'd settled on backquote yet, nor whether we had
defmacro yet]:
(defun pi macro (form) ;<-- maclisp macro got whole form as arg
`(let ((interrupts-were-off (status nointerrupt)))
(unwind-protect (progn (cond ((not interrupts-were-off)
(nointerrupt t)))
,@(cdr form))
(if (not interrupts-were-off)
(nointerrupt nil)))))
The problem was that it kept leaving interrupts turned off. Took me a while
to discover the problem was that unwind-protect was BINDING interrupts off
during the unwind clauses. When I realized it was doing that, which was
what I was already wanting, I was able to simplify it to just:
(defun pi macro (form)
`(unwind-protect nil ,@(cdr form)))
Note that this wouldn't work in CL, though, since it doesn't run the unwinds
with interrupts off.
Symbolics Genera ran unwind-protect cleanups with interrupts off UNTIL
the ivory hardware and then (perhaps due to a bug in the hardware, not sure)
it didn't. This was a major mess to clean up, since lots of code depended
on the interrupts being off and things really running to completion.
Not sure why I mention this at this point, but I think it's interesting trivia
to know...
Kent M Pitman <······@world.std.com> wrote in message news:<···············@world.std.com>...
> [snip]
> Unwind-protect is present in Multics PL/1, ITS Teco, Maclisp, Lisp
> Machine Lisp, Common Lisp, and Java. (It's conspicuously absent in
> Scheme, btw, and not at all by accident.)
>
Isn't DYNAMIC-WIND a (more general) UNWIND-PROTECT?
Will Fitzgerald
··········@inetmi.com (Will Fitzgerald) writes:
> Kent M Pitman <······@world.std.com> wrote in message news:<···············@world.std.com>...
> > [snip]
> > Unwind-protect is present in Multics PL/1, ITS Teco, Maclisp, Lisp
> > Machine Lisp, Common Lisp, and Java. (It's conspicuously absent in
> > Scheme, btw, and not at all by accident.)
> >
>
> Isn't DYNAMIC-WIND a (more general) UNWIND-PROTECT?
No, it's a totally orthogonal operator. I sometimes call it the "other"
UNWIND-PROTECT. It's a useful one, too, and it's a darned shame CL doesn't
have it. But it's no substitute for UNWIND-PROTECT.
Using the WITH-OPEN-FILE metaphor as an example: DYNAMIC-WIND is what
lets you do process-switch inside WITH-OPEN-FILE without closing the
file; UNWIND-PROTECT is what lets you close the file when you do need
to upon final exit within a process.
Kent M Pitman <······@world.std.com> wrote in message news:<···············@world.std.com>...
> ··········@inetmi.com (Will Fitzgerald) writes:
>
> > Kent M Pitman <······@world.std.com> wrote in message news:<···············@world.std.com>...
> > > [snip]
> > > Unwind-protect is present in Multics PL/1, ITS Teco, Maclisp, Lisp
> > > Machine Lisp, Common Lisp, and Java. (It's conspicuously absent in
> > > Scheme, btw, and not at all by accident.)
> > >
> >
> > Isn't DYNAMIC-WIND a (more general) UNWIND-PROTECT?
>
> No, it's a totally orthogonal operator. I sometimes call it the "other"
> UNWIND-PROTECT. It's a useful one, too, and it's a darned shame CL doesn't
> have it. But it's no substitute for UNWIND-PROTECT.
>
> Using the WITH-OPEN-FILE metaphor as an example: DYNAMIC-WIND is what
> lets you do process-switch inside WITH-OPEN-FILE without closing the
> file; UNWIND-PROTECT is what lets you close the file when you do need
> to upon final exit within a process.
OK, but can't you implement UNWIND-PROTECT in Scheme with the
following syntax and have the same semantics as Common Lisp's
UNWIND-PROTECT?
(define-syntax unwind-protect
(syntax-rules ()
((_ protected-form)
(begin protected-form))
((_ protected-form cleanup-form ...)
(dynamic-wind
(lambda () #t)
(lambda () protected-form)
(lambda () cleanup-form ...)))))
··········@inetmi.com (Will Fitzgerald) writes:
> Kent M Pitman <······@world.std.com> wrote:
>
> > ··········@inetmi.com (Will Fitzgerald) writes:
> >
> > > Kent M Pitman <······@world.std.com> wrote:
> > >
> > > > Unwind-protect is present in Multics PL/1, ITS Teco, Maclisp, Lisp
> > > > Machine Lisp, Common Lisp, and Java. (It's conspicuously absent in
> > > > Scheme, btw, and not at all by accident.)
> > > >
> > >
> > > Isn't DYNAMIC-WIND a (more general) UNWIND-PROTECT?
> >
> > No, it's a totally orthogonal operator. I sometimes call it the "other"
> > UNWIND-PROTECT. It's a useful one, too, and it's a darned shame CL doesn't
> > have it. But it's no substitute for UNWIND-PROTECT.
> >
> > Using the WITH-OPEN-FILE metaphor as an example: DYNAMIC-WIND is what
> > lets you do process-switch inside WITH-OPEN-FILE without closing the
> > file; UNWIND-PROTECT is what lets you close the file when you do need
> > to upon final exit within a process.
>
> OK, but can't you implement UNWIND-PROTECT in Scheme with the
> following syntax and have the same semantics as Common Lisp's
> UNWIND-PROTECT?
>
> (define-syntax unwind-protect
> (syntax-rules ()
> ((_ protected-form)
> (begin protected-form))
> ((_ protected-form cleanup-form ...)
> (dynamic-wind
> (lambda () #t)
> (lambda () protected-form)
> (lambda () cleanup-form ...)))))
No. Read what I wrote. The purpose of DYNAMIC-WIND is to implement
process switches. If you were in a multi-tasking system, you would
not want with-open-file to open and close your file every time you did
a process switch; worse, in the implementation above, you would close it
each time you swithed procsesses but would not even re-open the file
on re-entry to the relevant process.
Kent M Pitman <······@world.std.com> writes:
> No, it's a totally orthogonal operator. I sometimes call it the "other"
> UNWIND-PROTECT. It's a useful one, too, and it's a darned shame CL doesn't
> have it. But it's no substitute for UNWIND-PROTECT.
>
Is it useful, really? It's useful if you have call/cc, *or* if you
assume that whatever multiprocessing framework you have satisfies a
couple of criteria: (a) that it multiplexes processes on top of a
single processor so there are well-defined process switches, and (b)
that Lisp gets control on these switches. Neither of these seem like
really sensible assumptions to me.
--tim
Tim Bradshaw <···@tfeb.org> writes:
> Kent M Pitman <······@world.std.com> writes:
>
> > No, it's a totally orthogonal operator. I sometimes call it the "other"
> > UNWIND-PROTECT. It's a useful one, too, and it's a darned shame CL doesn't
> > have it. But it's no substitute for UNWIND-PROTECT.
>
> Is it useful, really? It's useful if you have call/cc, *or* if you
> assume that whatever multiprocessing framework you have satisfies a
> couple of criteria: (a) that it multiplexes processes on top of a
> single processor so there are well-defined process switches, and (b)
> that Lisp gets control on these switches. Neither of these seem like
> really sensible assumptions to me.
Let's put it this way: it's as sensible as special variables.
The same criticism as you make is true of specials. But the ability
to "bind" another kind of cell than a special variable location
is an interesting and useful one.
Kent M Pitman <······@world.std.com> writes:
> Tim Bradshaw <···@tfeb.org> writes:
>
> > Kent M Pitman <······@world.std.com> writes:
> >
> > > No, it's a totally orthogonal operator. I sometimes call it the "other"
> > > UNWIND-PROTECT. It's a useful one, too, and it's a darned shame CL doesn't
> > > have it. But it's no substitute for UNWIND-PROTECT.
> >
> > Is it useful, really? It's useful if you have call/cc, *or* if you
> > assume that whatever multiprocessing framework you have satisfies a
> > couple of criteria: (a) that it multiplexes processes on top of a
> > single processor so there are well-defined process switches, and (b)
> > that Lisp gets control on these switches. Neither of these seem like
> > really sensible assumptions to me.
>
> Let's put it this way: it's as sensible as special variables.
> The same criticism as you make is true of specials. But the ability
> to "bind" another kind of cell than a special variable location
> is an interesting and useful one.
While this might be the case, there is a clear difference, in that the
existence of special variables doesn't mandate the existence of clear
points of process/context switches, allowing the implementation to run
multiple processes/contexts at once. I'm not clear on how useful
dynamic-wind remains if the implementation is allowed to let multiple
contexts to exist at the same time, and if it isn't this seems to me
to be a clear indication that we must seek for another mechanism to
allow such things to happen, if we don't want to restrict
implementations to one particular model of multi-processing...
Regs, Pierre.
--
Pierre R. Mai <····@acm.org> http://www.pmsf.de/pmai/
The most likely way for the world to be destroyed, most experts agree,
is by accident. That's where we come in; we're computer professionals.
We cause accidents. -- Nathaniel Borenstein
"Pierre R. Mai" <····@acm.org> writes:
...
> While this might be the case, there is a clear difference, in that the
> existence of special variables doesn't mandate the existence of clear
> points of process/context switches, allowing the implementation to run
> multiple processes/contexts at once. I'm not clear on how useful
> dynamic-wind remains if the implementation is allowed to let multiple
> contexts to exist at the same time, and if it isn't this seems to me
> to be a clear indication that we must seek for another mechanism to
> allow such things to happen, if we don't want to restrict
> implementations to one particular model of multi-processing...
Yeah, I wasn't trying to get into the issue of whether it was efficient,
just whether it was expressive. I agree that as an efficiency issue, it
does seem through its full generality to create kinds of issues that
one might wish it didn't. But it definitely allows me to write useful
and interesting programs that present operators do not allow me to write.
Indeed, CLIM has some major conceptual design bugs because it thinks
you can use the LispM's LETF (bind an arbitrary location) in order to
manipulate graphics states and there is nothing like this in CL. In
the implementations I've seen, this is rewritten to use UNWIND-PROTECT
where DYNAMIC-WIND is wanted, so the changes are not undone on process
switches as they should be, and what this leads to is random lossage
if you try to use CLIM in conjunction with multiple processes.
Kent M Pitman <······@world.std.com> writes:
> Indeed, CLIM has some major conceptual design bugs because it thinks
> you can use the LispM's LETF (bind an arbitrary location) in order to
> manipulate graphics states and there is nothing like this in CL. In
> the implementations I've seen, this is rewritten to use UNWIND-PROTECT
> where DYNAMIC-WIND is wanted, so the changes are not undone on process
> switches as they should be, and what this leads to is random lossage
> if you try to use CLIM in conjunction with multiple processes.
I'm not clear how DYNAMIC-WIND would prevent this. In a Lisp which
does support multiple concurrent threads (ie no thread switches, no
stack undwinding/rewinding), I can't see how it would work. When I
started writing this I had in mind some trick using WITHOUT-PREEMPTION
but I don't think I can see how this would go now.
However I think that it's clear that anything that could work would
essentially consist in blocking out all other threads in some way so
that you convert the concurrent system into a multiplexed one. It's
one of my biggest annoyances with Lisp multiprocessing systems that
they provide operators which, with the excuse of providing some cool
functionality, are catastrophically expensive in this way
(WITHOUT-PREEMPION is another very good example). I really hope that
when/if some multiprocessing framework is agreed it is a bit less
badly designed than the existing ones.
--tim
Tim Bradshaw <···@tfeb.org> writes:
>
> Kent M Pitman <······@world.std.com> writes:
>
>
> > Indeed, CLIM has some major conceptual design bugs because it thinks
> > you can use the LispM's LETF (bind an arbitrary location) in order to
> > manipulate graphics states and there is nothing like this in CL. In
> > the implementations I've seen, this is rewritten to use UNWIND-PROTECT
> > where DYNAMIC-WIND is wanted, so the changes are not undone on process
> > switches as they should be, and what this leads to is random lossage
> > if you try to use CLIM in conjunction with multiple processes.
>
> I'm not clear how DYNAMIC-WIND would prevent this. In a Lisp which
> does support multiple concurrent threads (ie no thread switches, no
> stack undwinding/rewinding), I can't see how it would work. When I
> started writing this I had in mind some trick using WITHOUT-PREEMPTION
> but I don't think I can see how this would go now.
>
> However I think that it's clear that anything that could work would
> essentially consist in blocking out all other threads in some way so
> that you convert the concurrent system into a multiplexed one. It's
> one of my biggest annoyances with Lisp multiprocessing systems that
> they provide operators which, with the excuse of providing some cool
> functionality, are catastrophically expensive in this way
> (WITHOUT-PREEMPION is another very good example). I really hope that
> when/if some multiprocessing framework is agreed it is a bit less
> badly designed than the existing ones.
>
I was not intending to make a remark about efficiency, but rather about
correctness.
One of the things I like about Lisp is that it begins its design by people
saying what they want to do and then tries to make those things efficient.
Most other languages begin only with what is known to be efficient. The
problem with that is that mostly they have learned what is efficient by
optimizing what the language contains, and since the language will never
grow until people try to optimize other things, you have a chicken/egg
problem for growth.
Tim Bradshaw <···@tfeb.org> writes:
> It's
> one of my biggest annoyances with Lisp multiprocessing systems that
> they provide operators which, with the excuse of providing some cool
> functionality, are catastrophically expensive in this way
> (WITHOUT-PREEMPION is another very good example). I really hope that
> when/if some multiprocessing framework is agreed it is a bit less
> badly designed than the existing ones.
Ah. Yes. Right.
So supposing hypothetically that I were working on adding MP support
to SBCL (userland stack groups at present; real kernel threading for
SMP is another project) and had got to the point where the low-level
grunge appeared to be working and it was time to start thinking about
the API, would you have any thoughts on that?
WITHOUT-INTERRUPTS is one obvious (or at least, oft-repeated) no-no.
Other experiences? The Java threadoing API in earlier versions
allowed threads to start and stop each other, but iirc no longer
allows that (I forget exactly why; I have heard that it became all too
easy to deadlock, but then given that this is Java they might just
have decided it was too hard to implement on some architecture they
had to use).
Locking support? Some Lisps have recursive locks so that the same
thread can acquire the same lock more than once - anyone using that?
Java allows any object at all to be locked, but I don't see much value
in that. How about monitors? How about a facility for declaring
the required lock ordering so that attempts to acquire locks in out of
order can be reported? Or is that too difficult to implement sanely
in speed-critical code?
arrest reasons, run reasons, wait reasons: how much of this does
anyone actually use?
Sometimes you can force another process to execute a function you pass
it. (Sounds confusing where special variable bindings concerned, but
anyway). Is this a fairly normal thing to need to do, or is it for
emergency situations only?
Signals! Where do you want signals to be delivered?
with-timeout - iirc somebody was talking about with-timeout being
potentially dangerous the other day. Why?
All ideas welcome.
-dan
--
http://ww.telent.net/cliki/ - Link farm for free CL-on-Unix resources
From: Bulent Murtezaoglu
Subject: Re: MP (was Re: How to read from multiple files?)
Date:
Message-ID: <874rqoau9y.fsf@nkapi.internal>
>>>>> "DB" == Daniel Barlow <···@telent.net> writes:
[...]
DB> Signals! Where do you want signals to be delivered?
Hmm, if you are talking about unix signals, the sensible default probably
is the thread that sets up the handler. This would work for things like
USR[1,2], ALARM and overloaded stuff like WINCH etc. Things like SEGV, PIPE
etc. will probably have to be caught by the main lisp anyway.
DB> with-timeout - iirc somebody was talking about with-timeout
DB> being potentially dangerous the other day. Why?
One danger I can see is that with-timeout implies an asynchronous event
with might cause you to leave your variables etc. in an inconsistent
state. KMP's example with nreverse (AFAIR) was a lucid one: as you
are doing your black magic _within_ nreverse something happens and causes
your cleanup forms to be executed via the unwind-protect you were
thoughtful enough to use, but, OOPS, those forms rely on the integrity
of the list you just put into a messy state while executing nreverse.
I was also thinking about with-timeout, but in the non-MP context.
It seems as long as you can catch SIGALARM, something like the following
can be the guts of with-timout expansion.
;;very vague
(defun ih-sigalarm (signal code scp) ;; signal handlers are like this
(declare (ignore signal code scp)) ;; in CMUCL
(signal 'sigalarm))
;; and then
(handler-case
(progn
(sys:enable-interrupt UNIX:sigpipe #'ih-sigalarm) ;install handler
;; set the alarm here
;; bla bla, do work
)
(sigalarm (foo) (declare (ignore foo))
;;do cleanup
)))
Multiple nested timeouts (with their own lisp signals probably) can be
made to work by the way it is done with C (use setitimer/getitimer cleverly
for the nearest event and maintain a priory queue of events).
Something like the above seems to work with CMUCL and I have a report
that it works w/o MP on LW. I am not sure if there is a good reason
for the signal handler to be called from within the dynamic context of the
handler-case. Does anyone else play with such stuff? For clisp
maybe?
cheers,
BM
Daniel Barlow <···@telent.net> writes:
>
> So supposing hypothetically that I were working on adding MP support
> to SBCL (userland stack groups at present; real kernel threading for
> SMP is another project) and had got to the point where the low-level
> grunge appeared to be working and it was time to start thinking about
> the API, would you have any thoughts on that?
>
> WITHOUT-INTERRUPTS is one obvious (or at least, oft-repeated) no-no.
> Other experiences? The Java threadoing API in earlier versions
> allowed threads to start and stop each other, but iirc no longer
> allows that (I forget exactly why; I have heard that it became all too
> easy to deadlock, but then given that this is Java they might just
> have decided it was too hard to implement on some architecture they
> had to use).
>
I think that the things to avoid are things that don't have reasonable
implementations on top of the kind of thread system you may have to
live on top of. Unfortuntately there are a load of those, and even
the ones that call themselves `posix threads' have plenty of variation
I think. So this isn't really helpful, I guess.
More generally, I think you want to avoid or at least flag in red
letters operations which inevitably serialise the system while
appearing to do something else that is useful. There are at least
several of the WITHOUT-x forms that come into the category of
flagging-with-red-letters I think. You probably do want a way of
stopping all other threads, for things like dumping images, but you
definitely don't want people using it instead of locks, because it's a
*very* expensive operation on a multiprocessor - not only does it
serialise the code (which might be OK for something like assigning a
single variable), it requires some awful global broadcast to do so,
and on something like a NUMA machine that's really bad.
I think there are various varieties of WITHOUT-x which probably need
to be more carefully distinguished, too. For instance There's a
difference between `don't preempt me, but it is OK for other processes
to run on other CPUs' and `don't preempt me *and* stop all other
processes'.
And there are operations which I think have no clear semantics at all
on a multiprocessor, such as DYNAMIC-WIND, which I think should not be
there at all. (I guess DYNAMIC-WIND in particular could be given
semantics by serialising everything within any call to it (by any
thread...) and forcing context switches so it can get its fingers in
the pie, but that would be pretty horrible.)
Unfortunately a lot of existing multithreaded Lisp code seems to thing
that gratuitously serialising the code is a reasonably thing to do,
because it's easier than having to have a lock.
--tim
Tim Bradshaw <···@tfeb.org> writes:
> I think there are various varieties of WITHOUT-x which probably need
> to be more carefully distinguished, too. For instance There's a
> difference between `don't preempt me, but it is OK for other processes
> to run on other CPUs' and `don't preempt me *and* stop all other
> processes'.
Yes, but I think both are a bad idea. The first sounds at first like
a reasonable construct, but I think it's making an unreasonable
request. Imagine the following situation: there is a Lisp with 10
threads running on a 4-processor machine, and the Lisp has been given
3 of those processors. All 3 threads running on the given processors
are in the middle of some operation they all requested to be atomic,
with WITHOUT-PREEMPTING-ME, when the OS takes a processor away from
the Lisp. Now it can only run 2 threads concurrently, but which two?
It can't satisfy the requested behavior, so the construct can't be
relied on.
On 03 Sep 2001 10:49:58 +0100, Tim Bradshaw <···@tfeb.org> wrote:
>Daniel Barlow <···@telent.net> writes:
>
>>
>> So supposing hypothetically that I were working on adding MP support
>> to SBCL (userland stack groups at present; real kernel threading for
>> SMP is another project) and had got to the point where the low-level
>> grunge appeared to be working and it was time to start thinking about
>> the API, would you have any thoughts on that?
>>
>> WITHOUT-INTERRUPTS is one obvious (or at least, oft-repeated) no-no.
>> Other experiences? The Java threadoing API in earlier versions
>> allowed threads to start and stop each other, but iirc no longer
>> allows that (I forget exactly why; I have heard that it became all too
>> easy to deadlock, but then given that this is Java they might just
>> have decided it was too hard to implement on some architecture they
>> had to use).
>>
>
>I think that the things to avoid are things that don't have reasonable
>implementations on top of the kind of thread system you may have to
>live on top of. Unfortuntately there are a load of those, and even
>the ones that call themselves `posix threads' have plenty of variation
>I think. So this isn't really helpful, I guess.
>
>More generally, I think you want to avoid or at least flag in red
>letters operations which inevitably serialise the system while
>appearing to do something else that is useful. There are at least
>several of the WITHOUT-x forms that come into the category of
>flagging-with-red-letters I think. You probably do want a way of
>stopping all other threads, for things like dumping images, but you
>definitely don't want people using it instead of locks, because it's a
>*very* expensive operation on a multiprocessor - not only does it
>serialise the code (which might be OK for something like assigning a
>single variable), it requires some awful global broadcast to do so,
>and on something like a NUMA machine that's really bad.
Yes, it's all those WITHOUT macros that turned me off to the semi-standard
MP package. For example, Win32 does not support any way (to my knowledge)
of saying "don't preempt this section of code". This is by design, in that it
forces programmers of that operating system to think in terms of all threads
running in parallel rather than trying to think of a multi-tasking processor.
I fully believe this is the right approach, in that it is very scalable, as I
think threads ought to be. Therefore, I can't implement WITHOUT-INTERRUPTS
efficiently, and I think an inefficient implementation would appear to suck
(to put it bluntly). It is possible to achieve some equivalent behavior by
suspending all other processes, and resuming them after the block of code is
executed. However, this results in 2 OS calls per thread, and I hardly consider
this scalable. It is far worse than serializing, because of the overhead it
introduces. I don't trust that suspending and resuming processes is all that
efficient on Win32. It is normally not done except when you are debugging
or something like that. By the way, the garbage collector can be expected
to do this, but the garbage collector already has pretty signficant overhead
such that this is more reasonable. It doesn't run often, relatively. In any
case, I consider this to be a significant problem to do it even only for garbage
collection, and would like to figure out a way to avoid it. (Yes, there are all
kinds of papers published, I know).
The whole MP package seems to have a world view of a lisp system which
controls the scheduler. This is fundamentally opposed to a system which works
within an OS, and lets the OS do the scheduling. I think the latter is preferred
in most cases, although I could see some nice things could be done with the
former. And I certainly understand the historical context of how and why it is
the way it is.
Referring back to earlier posts, I also believe that the current Common Lisp
system can easily be adapted to support multiple processors. I am working
on such a system (with Corman Lisp) and I only say working because I know
there are still some kinks to work out. Basically, it works today.
Everybody (except of course those who read this newsgroup) seem to think that
Java supports threads quite well. I think it supports threads pretty poorly, and
that's after working with it in the trenches for some time. However, Common Lisp
can easily support threads as well as Java simply by a few standardized
functions (possibly out of the existing MP library) and some definition about
which standard lisp objects have their functions synchronized. Well, this is
actually a fair bit of work, because the standard methods that work on say,
hash-tables, is not quite as easily defined as in Java (e.g. does something
like DESCRIBE synchronize a hash-table when you pass it?).
The other important thing to declare in the standard is how special variable
bindings are handled. Presumably they are per-thread. I think this is how
everybody implements them, but it needs to be clarified in the standard.
I know there are other things, but I don't in any way buy the argument that
we should create a new language to do parallel processing. Other languages
have shoehorned it in with reasonable success.
Roger
From: ···@scieneer.com
Subject: Re: MP (was Re: How to read from multiple files?)
Date:
Message-ID: <3B94980E.8E1E8723@scieneer.com>
Roger Corman wrote:
...
> >>
> >> So supposing hypothetically that I were working on adding MP support
> >> to SBCL (userland stack groups at present; real kernel threading for
> >> SMP is another project) and had got to the point where the low-level
> >> grunge appeared to be working and it was time to start thinking about
> >> the API, would you have any thoughts on that?
> >>
> >> WITHOUT-INTERRUPTS is one obvious (or at least, oft-repeated) no-no.
> >> Other experiences? The Java threadoing API in earlier versions
> >> allowed threads to start and stop each other, but iirc no longer
> >> allows that (I forget exactly why; I have heard that it became all too
...
> Yes, it's all those WITHOUT macros that turned me off to the semi-standard
> MP package. For example, Win32 does not support any way (to my knowledge)
Here in multi-threaded (SMP) implementations for unix, the
'without-interrupts macro is only used internally within the
implementation to prevent a thread being interrupted for garbage
collection or accepting lisp level interrupts sent from other lisp
threads.
The 'with-exclusive-execution macro is used to execute a body of code
with all other threads not executing lisp code.
Non-blocking compare-swap primitives are provided, plus memory
barriers, and the support provided by the pthreads standard so you
have the pthreads locks and condition variables etc.
The use of without-interrupts in legacy code makes porting difficult.
External unix signals are handled by a new thread - so
'without-interrupts is not relevant here. If a lisp thread happens to
be within a 'with-exclusive-execution context then the new signal
handling thread will block waiting to enter lisp.
A mechanism is provided for lisp threads to send an 'interrupt' to
each other, but it is only provided for debugging purposes and there
is no guarantee that the interrupts will be handled in a timely
manner. The unix foreign libraries are in general not interrupt safe
so it is not possible to reliably interrupt foreign code; the lisp
interrupts are only handled upon return to lisp.
Attempting to use lisp interrupts to implement 'with-timeout is
obviously not going to work reliably, and 'with-timeout is not even
provided. Stream timeouts are useful.
The 'unwind-protect macro does not attempt to disable interrupts
while unwinding.
Throwing out of an interrupted context is likely to leave locks
held etc. Another reason to avoid 'with-timeout.
Some legacy code is written assuming the polled-interrupt schemes used
by the some of the vendors - allowing certain code sequences to be
effectively atomic. Obviously this is not effective in an SMP implementation.
> Referring back to earlier posts, I also believe that the current Common Lisp
> system can easily be adapted to support multiple processors. I am working
> on such a system (with Corman Lisp) and I only say working because I know
> there are still some kinks to work out. Basically, it works today.
Yes, it works fine across a range of unix systems also.
Hopefully those interested in this approach can cooperate and use
common conventions for the extensions. It should be possible to
make some of implementations developed here available for
the community to consider within a few months.
> that's after working with it in the trenches for some time. However, Common Lisp
> can easily support threads as well as Java simply by a few standardized
> functions (possibly out of the existing MP library) and some definition about
> which standard lisp objects have their functions synchronized. Well, this is
> actually a fair bit of work, because the standard methods that work on say,
> hash-tables, is not quite as easily defined as in Java (e.g. does something
> like DESCRIBE synchronize a hash-table when you pass it?).
Here, each hash table is protected by a lock. This default can be
avoided when creating the table, which can be handy when coarser
locking is used.
> The other important thing to declare in the standard is how special variable
> bindings are handled. Presumably they are per-thread. I think this is how
> everybody implements them, but it needs to be clarified in the standard.
Here each symbol has a 'global' value stored in the symbol object,
plus a 'dynamic' value for dynamic bindings. The 'dynamic' value
is local to each thread and shallow bound for speed. The 'global'
value is global to all threads. There is a primitive to atomically
compare-swap a symbol global value for implementing thread safe
non-waiting counters, lists, etc.
Areas such as object evolution under CLOS will need more thought
here, but you can certainly run a web server etc on an SMP capable
Common-Lisp implementation.
Good to see that Corman Lisp will likely support the intel/microsoft
platform as SMT rolls out, and hopefully we can coordinate
convenstions so that customers can share code with our unix
implementations.
Regards
Douglas Crosher
On 31 Aug 2001 17:53:14 +0100, Daniel Barlow <···@telent.net> wrote:
>
>So supposing hypothetically that I were working on adding MP support
>to SBCL (userland stack groups at present; real kernel threading for
>SMP is another project) and had got to the point where the low-level
>grunge appeared to be working and it was time to start thinking about
>the API, would you have any thoughts on that?
>
Once upon a time the sources for Butterfly Lisp were in the CL
archives at CMU. I always thought that their construct for MP,
"Futures", was really neat. BCL was CLtL1 with PCL, but that
shouldn't make it too hard to deal with. The real trick is that the CL
is based on Butterfly Scheme, so the real low-level stuff is in a
variant of MIT Scheme.
...arun
Kent M Pitman <······@world.std.com> writes:
...
> Indeed, CLIM has some major conceptual design bugs because it thinks
> you can use the LispM's LETF (bind an arbitrary location) in order to
> manipulate graphics states and there is nothing like this in CL.
If I remember correctly, the CLIM spec also requires a SETF* macro
which should work for 'multiple values forms'. Somethiong like
* point
#S(POINT :X 1 :Y 2)
* (coords point)
=> 1
=> 2
could be manipulated with
* (setf (coords point) 33 44)
=> 33
=> 44
* point
#S(POINT :X 33 :Y 44)
How could this macro be implemented? (Of course with all the DEFSETF*
support etc etc)
Cheers
--
Marco Antoniotti ========================================================
NYU Courant Bioinformatics Group tel. +1 - 212 - 998 3488
719 Broadway 12th Floor fax +1 - 212 - 995 4122
New York, NY 10003, USA http://bioinformatics.cat.nyu.edu
"Hello New York! We'll do what we can!"
Bill Murray in `Ghostbusters'.
On 31 Aug 2001, Marco Antoniotti wrote:
> If I remember correctly, the CLIM spec also requires a SETF* macro
> which should work for 'multiple values forms'. Somethiong like
>
> * point
> #S(POINT :X 1 :Y 2)
>
> * (coords point)
> => 1
> => 2
>
> could be manipulated with
>
> * (setf (coords point) 33 44)
> => 33
> => 44
>
> * point
> #S(POINT :X 33 :Y 44)
No, it's more like:
(defgeneric* (setf coords) (x y point))
(setf (coords point) (values 33 44))
> How could this macro be implemented? (Of course with all the DEFSETF*
> support etc etc)
From the McCLIM sources:
(defun setf-name-p (name)
(and (listp name) (eq (car name) 'setf)))
(defmacro defgeneric* (fun-name lambda-list &body options)
"Defines a SETF* generic function. FUN-NAME is a SETF function
name. The last argument is the single argument to the function in a
SETF place form; the other arguments are values collected from the
SETF new value form."
(unless (setf-name-p fun-name)
(error "~S is not a valid name for a SETF* generic function."
fun-name))
(let ((setf-name (cadr fun-name))
(args (butlast lambda-list))
(place (car (last lambda-list))))
`(progn
(defsetf ,setf-name (,place) ,args
`(funcall #',',fun-name ,,@args ,,place))
(defgeneric ,fun-name ,lambda-list ,@options))))
(defmacro defmethod* (name &body body)
"Defines a SETF* method. NAME is a SETF function name. Otherwise,
like DEFMETHOD except there must exist a corresponding DEFGENERIC* form."
(unless (setf-name-p name)
(error "~S is not a valid name for a SETF* generic function." name))
`(defmethod ,name ,@body))
I'd be interested in any criticism of this approach...
Tim
Marco Antoniotti <·······@cs.nyu.edu> writes:
> If I remember correctly, the CLIM spec also requires a SETF* macro
> which should work for 'multiple values forms'. Somethiong like
>
> * point
> #S(POINT :X 1 :Y 2)
>
> * (coords point)
> => 1
> => 2
>
> could be manipulated with
>
> * (setf (coords point) 33 44)
> => 33
> => 44
>
> * point
> #S(POINT :X 33 :Y 44)
>
> How could this macro be implemented? (Of course with all the DEFSETF*
> support etc etc)
Did you mean SETF* above? SETF doesn't work as you've indicated.
I am not familiar with how CLIM's SETF* works. It's been too long since
I looked at the sources. I don' know a syntax that used to work for some
SETF's in the early days that looked like:
(SETF (COORDS POINT) (VALUES 33 44))
[One could also use (THE (VALUES X Y) (F)) to notate that (F) will return
two values and that SETF should expect it.]
I can't remember if Zetalisp allowed this. It might have. I Know it
used the "other" arg order for setf methods, putting the values at the end.
[It's a little easier to pre-compile a SETF in this case, since it just
appends the values it gets back to whatever args it was given, but it's
harder to write the handler becuase SETF of anything with &REST has no
room for extra values after the &REST.]
The normal way this would be done would be (1) require that the VALUES
form appear in the caddr of the SETF form, and (2) rewrite this to
call a SETF method that takes n (2, in this case) arguments before the
normal ones. e.g.,
(DEFUN (SETF COORDS) (NEW-X NEW-Y POINT)
...)
Since this is a very marginal use of SETF, and could also unintentionally
"accomodate" things like:
(SETF (COORDS NEW-Y POINT) NEW-X)
I'll bet CLIM probably has a defining form for a SETF* method which declares
the number of values, so that SETF* can magically know without seeing an
explicit calls to VALUES. That would allow one to have
(SETF* (COORDS POINT1) (COORDS POINT2))
where the SETF* magically knows to reserve two arg slots for the results
of (COORDS POINT2) even though it's not lexically obvious that's how many
values will come back.
Of course, if COORDS is just a macro for
`(VALUES (XPOS ,POINT) (YPOS ,POINT))
then this all gets much simpler since even a regular SETF could figure out
what to do.
Sorry for the random walk through design space. Hope it's helpful tho.
On 31 Aug 2001 12:04:53 -0400, Marco Antoniotti <·······@cs.nyu.edu> wrote:
> If I remember correctly, the CLIM spec also requires a SETF* macro
[...]
> How could this macro be implemented? (Of course with all the DEFSETF*
I seem to remember that it has been recently added to Free CLIM (a.k.a.
McCLIM).
Paolo
--
EncyCMUCLopedia * Extensive collection of CMU Common Lisp documentation
http://web.mclink.it/amoroso/ency/README
[http://cvs2.cons.org:8000/cmucl/doc/EncyCMUCLopedia/]
From: Casper H.S. Dik - Network Security Engineer
Subject: Re: MP (was Re: How to read from multiple files?)
Date:
Message-ID: <9n25e8$mes$1@new-usenet.uk.sun.com>
[[ PLEASE DON'T SEND ME EMAIL COPIES OF POSTINGS ]]
Daniel Barlow <···@telent.net> writes:
>WITHOUT-INTERRUPTS is one obvious (or at least, oft-repeated) no-no.
>Other experiences? The Java threadoing API in earlier versions
>allowed threads to start and stop each other, but iirc no longer
>allows that (I forget exactly why; I have heard that it became all too
>easy to deadlock, but then given that this is Java they might just
>have decided it was too hard to implement on some architecture they
>had to use).
Stopping and starting threads creates serious problems with locking;
you have no idea where the other thread is, for all you know, it could
be holding a lock to a critical resource like the memory pool.
>Locking support? Some Lisps have recursive locks so that the same
>thread can acquire the same lock more than once - anyone using that?
Recursive locks are bad. If you need them, your code likely has many
problems.
>Java allows any object at all to be locked, but I don't see much value
>in that. How about monitors? How about a facility for declaring
>the required lock ordering so that attempts to acquire locks in out of
>order can be reported? Or is that too difficult to implement sanely
>in speed-critical code?
Typically, mutex code does little in the way of error checking; they
need to be cheap and fast at least when you obtain the lock; in the error
path you could conceivably do more checks as that isn't time critical.
>Signals! Where do you want signals to be delivered?
You don't :-) Some signals are thread specific (FP exceptions,
memory exceptions) the others are hard to deal with,
Casper
--
Expressed in this posting are my opinions. They are in no way related
to opinions held by my employer, Sun Microsystems.
Statements on Sun products included here are not gospel and may
be fiction rather than truth.
From: Casper H.S. Dik - Network Security Engineer
Subject: Re: MP (was Re: How to read from multiple files?)
Date:
Message-ID: <9n25pa$mh9$1@new-usenet.uk.sun.com>
[[ PLEASE DON'T SEND ME EMAIL COPIES OF POSTINGS ]]
···@conquest.OCF.Berkeley.EDU (Thomas F. Burdick) writes:
>Yes, but I think both are a bad idea. The first sounds at first like
>a reasonable construct, but I think it's making an unreasonable
>request. Imagine the following situation: there is a Lisp with 10
>threads running on a 4-processor machine, and the Lisp has been given
>3 of those processors. All 3 threads running on the given processors
>are in the middle of some operation they all requested to be atomic,
>with WITHOUT-PREEMPTING-ME, when the OS takes a processor away from
>the Lisp. Now it can only run 2 threads concurrently, but which two?
>It can't satisfy the requested behavior, so the construct can't be
>relied on.
Your typical OS will not guarantee "without preemption" anyway as it is
just too dangerous to give applications control over their own preemptive ness.
You can sometimes specify "I'd rather not be preempted now, thank you" but
the kernel will ignore that request if it takes too long to complete.
An ordinary kernel threads implementation will schedule all your Lisp threads
independently; each can be preempted at any time but the application wouldn't
really care as it shouldn't notice preemption.
And what does "WITHOUT-PREEMPTING-ME" really mean? It would seem to
indicate that you don't want other threads to run concurrently as it
would be non-sensical otherwise, but that would still not happen
(unless you decide to make this hideously expensive and stop all other
threads; but stopping other threads is very dangerous and can easily
lead to deadlocks)
Casper
--
Expressed in this posting are my opinions. They are in no way related
to opinions held by my employer, Sun Microsystems.
Statements on Sun products included here are not gospel and may
be fiction rather than truth.
From: Kent M Pitman
Subject: Re: MP (was Re: How to read from multiple files?)
Date:
Message-ID: <sfwitezjdah.fsf@world.std.com>
··········@Holland.Sun.Com (Casper H.S. Dik - Network Security Engineer) writes:
> Your typical OS will not guarantee "without preemption" anyway as it
> is just too dangerous to give applications control over their own
> preemptive ness.
I've not been following this "thread" in detail, so maybe this remark is
out of place, but:
Usually when I ask for without-preemption, I only mean
"please do not preempt this process with another LISP process sharing the
same address space; that is, I only are about preemption by other LISP
processes.
One of the only uses I use WITHOUT-PREEMPTION for is to implement
test-and-set. Yes, every MP package usually comes with locks, too, but since
everyone differs on how they are implemented, I often refuse to use their
lock package for fear it will be hard to get a compatible version working in
someone else's implementation. So I write WITHOUT-PREEMPTION around a very
short piece of code that just does my own lock management. If there were an
industry-wide lock standard, separate from MP, I would not feel a need to
do this and could eliminate a large number of my own uses of this operator.
Kent M Pitman <······@world.std.com> writes:
>
> Let's put it this way: it's as sensible as special variables.
> The same criticism as you make is true of specials. But the ability
> to "bind" another kind of cell than a special variable location
> is an interesting and useful one.
But DYNAMIC-WIND does not give you that ability. The classic example,
I guess is LETF. You can implement LETF wth DYNAMIC-WIND *iff* you
place the constraints on the multiprocessing system that I mentioned
before, namely that there are process switches and that Lisp gets to
run code during those switches. If those constraints are not true,
then you can't.
--tim
Tim Bradshaw <···@tfeb.org> writes:
> Kent M Pitman <······@world.std.com> writes:
>
> >
> > Let's put it this way: it's as sensible as special variables.
> > The same criticism as you make is true of specials. But the ability
> > to "bind" another kind of cell than a special variable location
> > is an interesting and useful one.
>
> But DYNAMIC-WIND does not give you that ability. The classic example,
> I guess is LETF. You can implement LETF wth DYNAMIC-WIND *iff* you
> place the constraints on the multiprocessing system that I mentioned
> before, namely that there are process switches and that Lisp gets to
> run code during those switches. If those constraints are not true,
> then you can't.
I take it as a given that dynamic-wind is unimportant if it is not thus
hooked into process switches. It doesn't have to run multi-processing,
it merely has to get pinged upon entry to a process and exit, and to do
the right thing for processes written in LIsp. IF it doesn't do that, and is
just a user program, anyone could write it but it's pointless to have. This
was true of the condition system as well; it's a simple user program anyone
could have written, but it takes the system calling someone in particular's
ERROR function (and other functionality) for that stuff to suddenly get raised
to the status of important.
My remarks were not intended to say "this is the right way to capture the
notion of multitasking" nor "this will have no ill effects on certain
multitasking systems". Rather, only to say "without this or something ike
this, there are certain important things you might want to write in a
multi-tasking system that you cannot"; dynamic-wind, like full continuations,
adds expressivity--it is not the answer to all the world's problems.
The original purpose of my remark was merely to acknowledge that there are
ways beyond traditional continuations in which Scheme does have expressivity
we don't. I dump on Scheme plenty, but the presence of dynamic-wind is not
one of my main irritations. unwind-protect, which is not present, IS a source
of irritation to me, and that's why I mentioned that one.
Kent M Pitman <······@world.std.com> writes:
> I take it as a given that dynamic-wind is unimportant if it is not thus
> hooked into process switches. It doesn't have to run multi-processing,
> it merely has to get pinged upon entry to a process and exit, and to do
> the right thing for processes written in LIsp. IF it doesn't do that, and is
> just a user program, anyone could write it but it's pointless to have. This
> was true of the condition system as well; it's a simple user program anyone
> could have written, but it takes the system calling someone in particular's
> ERROR function (and other functionality) for that stuff to suddenly get raised
> to the status of important.
Yes. Any my point is that multitasking systems do not need to have
process switches, and thus DYNAMIC-WIND turns out not to be a useful
abstraction in my opinion. Any Lisp multitasking system that supports
multiprocessor hardware will not always have process switches - rather
some number of processes will run simultaneously on the available
CPUs.
>
> My remarks were not intended to say "this is the right way to capture the
> notion of multitasking" nor "this will have no ill effects on certain
> multitasking systems". Rather, only to say "without this or something ike
> this, there are certain important things you might want to write in a
> multi-tasking system that you cannot"; dynamic-wind, like full continuations,
> adds expressivity--it is not the answer to all the world's problems.
>
My problem is that I can't see what DYNAMIC-WIND lets you do that is
worth the cost. If you use it to implement something like LETF (or
even special variables) then if the system is running on a
multiprocessor you either need to serialise any code that uses
DYNAMIC-WIND (and in general I think this means that if *any* thread
uses DYNAMIC-WIND, then *all threads* must be serialised), which is
really bad, or it just won't work reliably, which is also terrible.
So my take on this is that the things that DYNAMIC-WIND lets you do
are not compatible with a multiprocessing system. In particular I
think that LETF is just something that you should not expect to have
in a multiprocessing system - if you want a stack discipline for
something you actually need to have a stack, you cannot fake it by
shallow binding using DYNAMIC-WIND. So I think, for instance, that if
CLIM relies on LETF that is because CLIM's design is broken, *not*
because CL is broken.
I think (but do not have proof) that any other construct that lets you
do what DYNAMIC-WIND does will also have equally bad effects.
I definitely do not want operators in CL that will effectively forbid
a multiprocessor implementation, or make such an implementation
incredibly expensive and inefficient, any more than I want operators
in CL which require special HW support.
--tim
Tim Bradshaw <···@tfeb.org> writes:
> Kent M Pitman <······@world.std.com> writes:
>
> > I take it as a given that dynamic-wind is unimportant if it is not thus
> > hooked into process switches. It doesn't have to run multi-processing,
> > it merely has to get pinged upon entry to a process and exit, and to do
> > the right thing for processes written in LIsp. IF it doesn't do that, and is
> > just a user program, anyone could write it but it's pointless to have. This
> > was true of the condition system as well; it's a simple user program anyone
> > could have written, but it takes the system calling someone in particular's
> > ERROR function (and other functionality) for that stuff to suddenly get raised
> > to the status of important.
>
> Yes. Any my point is that multitasking systems do not need to have
> process switches, and thus DYNAMIC-WIND turns out not to be a useful
> abstraction in my opinion. Any Lisp multitasking system that supports
> multiprocessor hardware will not always have process switches - rather
> some number of processes will run simultaneously on the available
> CPUs.
I'm not talking about multi-processing, I'm talking about multitasking.
True concurrent multiprocessing is a completely different problem.
There are many other ways that CL is not protected against that.
> >
> > My remarks were not intended to say "this is the right way to capture the
> > notion of multitasking" nor "this will have no ill effects on certain
> > multitasking systems". Rather, only to say "without this or something ike
> > this, there are certain important things you might want to write in a
> > multi-tasking system that you cannot"; dynamic-wind, like full continuations,
> > adds expressivity--it is not the answer to all the world's problems.
> >
>
> My problem is that I can't see what DYNAMIC-WIND lets you do that is
> worth the cost. If you use it to implement something like LETF (or
> even special variables) then if the system is running on a
> multiprocessor you either need to serialise any code that uses
> DYNAMIC-WIND (and in general I think this means that if *any* thread
> uses DYNAMIC-WIND, then *all threads* must be serialised), which is
> really bad, or it just won't work reliably, which is also terrible.
I don't see what serializing has to do with anything. Let's talk in concrete
terms. One might write:
(LETF (((GRAPHIC-PEN *GRAPHICS-STATE*) (MAKE-GRAPHIC-PEN ...)))
...body...)
That is, we're dynamically binding something that is only a piece of
something else that is itself either globally or dynamically bound.
I don't see what serializing this does. All that's really going on is:
(LET ((.PEN. (MAKE-GRAPHIC-PEN ...)))
(FLET ((.SWAP-PENS. () (PSETF (GRAPHIC-PEN *GRAPHICS-STATE*) .PEN.
.PEN. (GRAPHIC-PEN *GRAPHICS-STATE*))))
(DYNAMIC-WIND #'.SWAP-PENS.
#'(LAMBDA ()...body...)
#'.SWAP-PENS.)))
The DYNAMIC-WIND is basically dealing with the issue that there are certain
global resources that are not themselves global variables (e.g., the pen may
be a slot in a struct somewhere) that you need to be able to assure are
used only in the process they were created in. When you move out of the
process, you put them on the stack. Now I find it *utterly* ridiculous that
Windows seems to have some model that a stream or other system resource is
good only in the process that made it and not in another, but I've been told
that's a Windows-created constraint, not a Lisp-created one. And so saying
Lisp shouldn't try to accomodate it (or any similar constraint) is what seems
to me unreasonable.
> So my take on this is that the things that DYNAMIC-WIND lets you do
> are not compatible with a multiprocessing system. In particular I
> think that LETF is just something that you should not expect to have
> in a multiprocessing system -
but so are half the operations in CL, aren't they? what do we define
in a way that is safely interlocked against real multiprocessing? get?
gethash? intern? ... i don't get it? The Lisp/Scheme community confronted
multiprocessing years ago with fights over whether we should guarantee
order of arg evaluation--some said no because they wanted to preserve
the right to multiprocess, but others said "forget it. programming in
multiprocessing is just a different thing and we need another language to
do it in". I think those people were probably right. We could solidify
CL for multiprocessing, but that won't happen by accident, and in the process
dynamic-wind might not be the only casualty. Why leave out operations
that DO make sense for mere multitasking just because someone artificially
and probably falsely envisions that programs can just be compiled and run
multiprocessed without thought?
This is not a tirade against multiprocessing. This is a question about
belief in magic... ;-)
> if you want a stack discipline for
> something you actually need to have a stack, you cannot fake it by
> shallow binding using DYNAMIC-WIND.
if you're going to use true multiprocessing.
> So I think, for instance, that if
> CLIM relies on LETF that is because CLIM's design is broken, *not*
> because CL is broken.
All I know is that the places I've seen where it relies on this, it
seemed to do so because the underlying operating system effectively
requires it to by having a global graphics state that is manipulated
by side-effect instead of an object that is passed around freely and
used as an argument. (Note that I was not the code maintainer--I was
just peeking in at the code on this so I might have been confused.
But that was what it seemed to me.)
> I think (but do not have proof) that any other construct that lets you
> do what DYNAMIC-WIND does will also have equally bad effects.
>
> I definitely do not want operators in CL that will effectively forbid
> a multiprocessor implementation, or make such an implementation
> incredibly expensive and inefficient, any more than I want operators
> in CL which require special HW support.
do you assert that the present language definition already supports
multiprocessing? if you do, then you have a valid case (and something
for me to think about). if you don't, then you're basically saying
that's not a presupposition of the system, and so your argument doesn't
hold because you are applying a false (imagined) premise.
(BTW, you have turned the argument into a feature request from me, and
I have not made such a request. So you are defending against your own
demons, not external ones. I would say the same of dynamic-wind that I
would say of full continuations: it offers power we don't have. But
I'm not asking for full continuations either. I'm merely observing
they can do things we can't. My arguments above should be taken in
that light--merely an intellectual discussion, which I am entertaining
less for the purpose of pushing DYNAMIC-WIND and more for the purpose
of understanding why and whether you think CL is already
multiprocessor ready, since I think it is probably not, though I admit
to not having done a lot of recent thought on this... it's been YEARS
since this issue came up. Last time was in the era of the connection
machine and the butterfly and probably half a dozen others, but the
issues don't seem to me, at first blush, materially different.)
[I still haven't phrased this message very well, sorry. I'm not
trying to be as nasty or combative as it reads]
Kent M Pitman <······@world.std.com> writes:
>
> The DYNAMIC-WIND is basically dealing with the issue that there are certain
> global resources that are not themselves global variables (e.g., the pen may
> be a slot in a struct somewhere) that you need to be able to assure are
> used only in the process they were created in. When you move out of the
> process, you put them on the stack. Now I find it *utterly* ridiculous that
> Windows seems to have some model that a stream or other system resource is
> good only in the process that made it and not in another, but I've been told
> that's a Windows-created constraint, not a Lisp-created one. And so saying
> Lisp shouldn't try to accomodate it (or any similar constraint) is what seems
> to me unreasonable.
No, it should try to accomodate it. But DYNAMIC-WIND relies on things
that may not happen (context switches) to do so. If you want to
insist that multitasking is different than multiprocessing, in the
sense that the interfaces to the code should be completely different,
then I guess DYNAMIC-WIND is a reasonable approach. I don't want to
have these two different approaches though - possibly because I'm
after a much hackier idea of multiprocessing than I think you are
thinking of.
> but so are half the operations in CL, aren't they? what do we define
> in a way that is safely interlocked against real multiprocessing? get?
> gethash? intern? ... i don't get it?
None of these are safe, *but* they can be made safe by a fairly
conventional approach of locking the object concerned. Other languages
have the issue that non-atomic operations are not safe and resolve it
this way: there is nothing special about Lisp which prevents this. Of
course this require people to write their programs in a careful way,
with locks and so on.
> The Lisp/Scheme community confronted
> multiprocessing years ago with fights over whether we should guarantee
> order of arg evaluation--some said no because they wanted to preserve
> the right to multiprocess, but others said "forget it. programming in
> multiprocessing is just a different thing and we need another language to
> do it in". I think those people were probably right. We could solidify
> CL for multiprocessing, but that won't happen by accident, and in the process
> dynamic-wind might not be the only casualty. Why leave out operations
> that DO make sense for mere multitasking just because someone artificially
> and probably falsely envisions that programs can just be compiled and run
> multiprocessed without thought?
>
> This is not a tirade against multiprocessing. This is a question about
> belief in magic... ;-)
I don't believe in magic. I do believe that you don't need a special
new Lisp dialect to support multiprocessors. Now I think I've just
realised what the issue is and where we're talking at cross purposes:
I'm *not* talking about some finely-grained thing, such that argument
evaluation might happen in different threads, I'm talking about
conventional boring different-processes-run-on-different-processors
stuff. The kind of thing that Posix threads supports in C. Of course
programming multithreaded applications is not trivial because you have
to be really careful about locking and so on for non-atomic operations
on shared data, but it's not something that you need a new language
for. Very fine-grained multiprocessing probably *is* something you
need a new language for, but no one is selling that kind of hardware
that I'm aware of any more.
> > if you want a stack discipline for
> > something you actually need to have a stack, you cannot fake it by
> > shallow binding using DYNAMIC-WIND.
>
> if you're going to use true multiprocessing.
Right.
>
> do you assert that the present language definition already supports
> multiprocessing? if you do, then you have a valid case (and something
> for me to think about). if you don't, then you're basically saying
> that's not a presupposition of the system, and so your argument doesn't
> hold because you are applying a false (imagined) premise.
No, I don't. However I am saying that the language + something quite
close to one of the MP systems that exist is adequate, in terms of
interface, to support the kind of multiprocessing in which I'm
interested. And again, I'm not after some fine-grained thing where
everything works by magic - those are research problems I think, quite
apart from being very unsuitable paradignms for the kind of commercial
hardware that is available where there are relatively small numbers
(less than 100 or so) of relatively fat processors - but conventional
hacky thread systems, where you have to lock things and generally be
careful. What I want to argue against is operations which have bad
global effects on such a system.
>
> (BTW, you have turned the argument into a feature request from me, and
> I have not made such a request. So you are defending against your own
> demons, not external ones. I would say the same of dynamic-wind that I
> would say of full continuations: it offers power we don't have. But
> I'm not asking for full continuations either. I'm merely observing
> they can do things we can't.
yes, I apologise for phrasing thigs this way.
> My arguments above should be taken in
> that light--merely an intellectual discussion, which I am entertaining
> less for the purpose of pushing DYNAMIC-WIND and more for the purpose
> of understanding why and whether you think CL is already
> multiprocessor ready, since I think it is probably not, though I admit
> to not having done a lot of recent thought on this... it's been YEARS
> since this issue came up. Last time was in the era of the connection
> machine and the butterfly and probably half a dozen others, but the
> issues don't seem to me, at first blush, materially different.)
Ah, I think they are very different. I'm not familiar with the
butterfly, but the CM (at least the early ones, I think later ones
were different) was a classic fine-grained system with tiny
processors, where you really did need things like pll evaluation to
take advantage of the system. I'm really thinking of big commercial
shared-memory boxes - the kind of thing you can buy from Sun or HP
today.
Really, what I'm trying to do is to raise awareness that some
operations and programming styles (and I'm obviously here thinking
about LETF / DYNAMIC-WIND, but things like WITHOUT-PREEMPTION are also
good canditates) are quite bad things to have, even in my hacky
multiprocessing world where you have to do a lot of things yourself,
and for which I think CL is already quite suitable. I'm just not
doing a very good job of expressing myself...
--tim
Tim Bradshaw <···@tfeb.org> writes:
> > but so are half the operations in CL, aren't they? what do we define
> > in a way that is safely interlocked against real multiprocessing? get?
> > gethash? intern? ... i don't get it?
>
> None of these are safe, *but* they can be made safe by a fairly
> conventional approach of locking the object concerned. Other languages
> have the issue that non-atomic operations are not safe and resolve it
> this way: there is nothing special about Lisp which prevents this. Of
> course this require people to write their programs in a careful way,
> with locks and so on.
And from a collaboration POV, the important thing is IMHO, that one
can take 3rd party code that wasn't developed with multiprocessing (or
even multitasking) in mind, and make it safe without going in and
rewriting it from the ground up. In the most trivial approach, one
has to ensure that only one process can be inside the code in question
at a time (trough locking/exclusive sections/whatever), which while
not being optimal can still allow other code to be run in parallel
without problems. And with a little more work and understanding it is
often possible to get much finer-grained locking behaviour, and hence
even parallelism inside the 3rd party code.
Contrast this with DYNAMIC-WIND, the presence of which will in all
likelyhood require the whole system to run in single-processing mode.
And since code using DYNAMIC-WIND is likely not that easily rewritten
using something else (otherwise we wouldn't really need DYNAMIC-WIND
in the first place), there is little left but to start from scratch,
rearchitecting and rewriting the 3rd party code.
This kind of non-local effect is something I'd worry about in a
construct such as DYNAMIC-WIND, and which e.g. isn't present in
special variables (which can be made to work fairly efficiently and
safely on many multiprocessing implementations, and which don't
exhibit the non-local effects of DYNAMIC-WIND).
Regs, Pierre.
--
Pierre R. Mai <····@acm.org> http://www.pmsf.de/pmai/
The most likely way for the world to be destroyed, most experts agree,
is by accident. That's where we come in; we're computer professionals.
We cause accidents. -- Nathaniel Borenstein
"Pierre R. Mai" <····@acm.org> writes:
> Contrast this with DYNAMIC-WIND, the presence of which will in all
> likelyhood require the whole system to run in single-processing mode.
> And since code using DYNAMIC-WIND is likely not that easily rewritten
> using something else (otherwise we wouldn't really need DYNAMIC-WIND
> in the first place), there is little left but to start from scratch,
> rearchitecting and rewriting the 3rd party code.
Yes, this is just what I'm worried about.
>
> This kind of non-local effect is something I'd worry about in a
> construct such as DYNAMIC-WIND, and which e.g. isn't present in
> special variables (which can be made to work fairly efficiently and
> safely on many multiprocessing implementations, and which don't
> exhibit the non-local effects of DYNAMIC-WIND).
>
It's also interesting - at least to me - that you can invent a lot of
other stack-like behaviour on top of specials, given that they have a
correctly-implemented stack. For instance I have a design (and rough
implementation of) a transaction system. In this you want it to be
the case that within a transaction changes to objects - even global
objects - are stored in the transaction and only committed to the
object when the transaction sucessfully completes (this is just part
of the usual transactional thing). This is done by making accessors
do something like this:
(defmethod crumblyness ((c cake))
(or (and *current-transaction*
(get-transaction-slot *current-transaction* c 'crumblyness)
(slot-value c 'crumblyness)))
(defmethod (setf crumblyness) (new (c cake))
(if *current-transaction*
(setf (get-transaction-slot *current-transaction* c 'crumblyness) new)
(setf (slot-value c 'crumblyness) new)))
transactions are essentially tables which map from objects ->
slotnames -> values in the obvious way, as well as having a parent
transaction, so the search recurses up throguh the parent. There's a
WITH-TRANSACTION macro which establishes a new transaction, runs its
body and then commits the results (there's obviously more hairiness
here because things must commit in order &c &c).
Anyway, in this framework you can have transaction-local (and hence
process-local) values for slots, with the whole thing being built on
the behaviour of special variables. Of course you lose performance,
and you have to define your classes with a special macro which
generates transactionally-aware accessors (I think you could also do a
MOP thing, but I didn't), but it's all quite neat.
--tim
Kent M Pitman <······@world.std.com> wrote in message news:<···············@world.std.com>...
> ··········@my-deja.com (glauber) writes:
[...]
> I imagine something like the following is what you want.
Thanks, it's perfect.
> For macros like this that do a lot of work, I find the trick is to
> quickly convert them to something functional and then program them
> like a regular function. (You do a lot less gensym'ing this way, and
> you also are able to patch this (in case of bugs) easily since you can
> just patch the function in most cases, leaving the calls unaffected.)
I agree. Functions are more powerful than we sometimes think!
g