Hi,
I'm writing because I wanted to suggest a new type of declaration that
would be useful for the implementations of CL which have
multiprocessing.
In each of these implementations we have different ways of declaring
variables to be special, but specifically within the context of a
thread, so that this data is not trampled on by other threads.
This happens all the time in multi-threaded applications. For ACL and
LispWorks, I've been able to write macros like
DEFPARAMETER-THREAD-SPECIAL and DEFVAR-THREAD-SPECIAL, but what I'd
*really* like is a declaration like this:
(declaim (thread-special *foo*))
Unfortunately, this doesn't tell Lisp enough information about the
variable. This is because upon entering a thread, you should tell
your Lisp what to initialize the variable to within that thread. For
example, if you have a nested list:
(defparameter-thread-special (*per-thread-tree* (copy-tree *per-thread-tree*))
'((foo (bar baz) (zoo rab zab)))
"A nested list of stuff.")
The second form in the first argument to DEFPARAMETER-THREAD-SPECIAL
above is an initform for each thread.
LispWorks and ACL have similar mechanisms (alist variables) for
handling this. However, I'd be much happier if there were a
declaration. For example, I might say:
(let ((*foo* *initial-foo*))
(declare (thread-special *foo*))
(dotimes (i 10)
(mp:process-run-function
(format nil "bar ~d" i)
#+lispworks nil
#'bar)))
(defun bar ()
(do-something-with *foo*))
Here I'm assuming that in each thread *foo* starts off as
*initial-foo*, whatever that is, and so in the case that *initial-foo*
were a list, you can see how it might be nice to use something like
COPY-TREE in each thread. But my primary concern is not on how to fit
the initform into the declaration. In fact, it's a bit annoying that
one has to specify an initform at all, since you can:
(defvar *my-special-var*)
or even:
(proclaim '(special *my-special-var*))
without binding it. So how are thread-special variables any different?
My primary interest here is to create a declaration for these
variables, as opposed to using things like
mp:*process-initial-bindings* (LW) or
excl:*cl-default-special-bindings* (ACL), albeit that these work, and
that they're fairly simple and straightforward to use. It would be
nice if multiprocessing had an elegant solution for thread-special
variables. The current solutions are, in my opinion, messy and in
some ways inconsistent with one another in the thread-enabled Common
Lisp implementations I'm familar with.
dave
David Bakhash <·····@alum.mit.edu> writes:
> I'm writing because I wanted to suggest a new type of declaration that
> would be useful for the implementations of CL which have
> multiprocessing.
It doesn't make as much sense as you think.
> In each of these implementations we have different ways of declaring
> variables to be special, but specifically within the context of a
> thread, so that this data is not trampled on by other threads.
Specials already get rebound per thread if already bound.
Certainly you don't want all variables of all threads to get rebound in any
thread. If they did, then the more threads you had, the more each would
create work for the others. It's bad enough having lots of threads without
having each contribute needless work to another.
Every thread that you own that cares should bind the variable itself
and then each thread will bind exactly the number of variables it
needs to.
You're perhaps also not thinking about worlds like the Lisp Machine which
are multithreaded and in which EVERY process ever run by any user is a lisp
process. Lisp machines normally ran for about 3 months without booting,
sometimes more than 6 months. If you imagine the effect accumulated of
loading junk for 6 months that forever increases the number of per-process
specials on all those processes, you can see the kind of "memory leak"
effect that makes Unix have to get booted more often. Being up a long time
doesn't have to mean accumulated garbage, but it gets that way when people
design systems that affect the "global" (which is what you're affecting when
you ask the process system to do it instead of doing it yourself just for
the processes you own) rather than the "local".
Within your own process, it's so easy to set up something where you declare
variables and then you do
(WITH-MY-VARIABLES ...)
around your own processes in a way that invades no one else's that it's hard
to really see this as a major issue.
By the way, the concept of "special" is already threadsafe, so the idea
of a thread-special is redundant, imo.
> For example, I might say:
>
> (let ((*foo* *initial-foo*))
> (declare (thread-special *foo*))
> (dotimes (i 10)
> (mp:process-run-function
> (format nil "bar ~d" i)
> #+lispworks nil
> #'bar)))
This also doesn't make any sense because the THREAD-SPECIAL declaration you're
making here would e lexical and would have zero effect on any of the other
operations. If you're going to be a language designer, you should have as a
rule of thumb that lexical declarations never affect separately compiled
functions. So nothing in this expression here could be affected by your
declaration UNLESS you made MP:PROCESS-RUN-FUNCTION be a special form instead
of a function, which seems disastrous. At best, you should ask for
(MP:PROCESS-RUN-FUNCTION "foo" `(:BIND ((*FOO* ,*INITIAL-FOO*))) #'FN)
but if you can do that why not just do
(MP:PROCESS-RUN-FUNCTION "foo" '()
#'(LAMBDA () (LET ((*FOO* *INITIAL-FOO*)) (FN))))
> So how are thread-special variables any different?
>
> My primary interest here is to create a declaration for these
> variables, as opposed to using things like
> mp:*process-initial-bindings* (LW) or
> excl:*cl-default-special-bindings* (ACL), albeit that these work, and
> that they're fairly simple and straightforward to use.
When I was in grade school, I had a teacher with a radium dial wristwatch.
She told us it was bad to have such a thing because it was radioactive but
she just didn't care. I came to wish I had something radioactive to carry
around, too. (I'm glad I never got such a thing before I learned better.)
Things can seem cool even if you don't understand them, even sometimes
especially because you don't understand them. Don't get me wrong. There
ARE sometimes reasons to add declarations. But such reasons are generally
to inform the compiler about how to compile a piece of language glue better.
Mechanisms for user programs to communicate with one another are already
present, and I don't think declarations are the right way to add them.
For example, even something so primitive as DEFVAR did not require the
addition of a new declaration. Special declarations already existed, and
DEFVAR was first added as a user-defined macro. To "declare" something
doesn't have to involve using the system-defined DECLARE facility.
You can already define a DEF-MY-VARIABLE thing which remembers stuff so
that a WITH-MY-VARIABLES macro can bind all the "declared" variables and
so you don't have to have used a declare facility.
All of the above "just my opinion", for whatever that's worth.
Kent M Pitman <······@world.std.com> writes:
> specials on all those processes, you can see the kind of "memory leak"
> effect that makes Unix have to get booted more often.
uh? more often than every 3 to 6 months? which unix have you been using
lately?
--
(espen)
* Kent M Pitman
| specials on all those processes, you can see the kind of "memory leak"
| effect that makes Unix have to get booted more often.
* Espen Vestre
| uh? more often than every 3 to 6 months? which unix have you been using
| lately?
Not Linux, apparently, where the most frequent reason to reboot is to
use a new kernel revision. However, on my production system, we still
run kernel revision 2.2.14 because I have sort of waited for today:
9:53am up 300 days, 6:00, 2 users, load average: 0.00, 0.00, 0.00
Other software that has been running exactly as long are screen 3.9.5,
Emacs 20.5, and Allegro CL 5.0.1. Allegro CL 6.0 just arrived, kernel
revision is at 2.2.18pre21, a few days old , Emacs 21.1 just got into
pretest and the new version of my own software is ready to install,
all coinciding randomly at 300 days of uptime. All we need now is for
a certain presidental election to release a final version of who won.
#:Erik
--
ALGORITHM: a procedure for solving a mathematical problem in a finite
number of steps that frequently involves repetition of an operation.
ALGOREISM: a procedure for solving an electoral problem in a finite
number of steps that frequently involves repetition of an operation.
Erik Naggum <····@naggum.net> writes:
> 9:53am up 300 days, 6:00, 2 users, load average: 0.00, 0.00, 0.00
>
> Other software that has been running exactly as long are screen 3.9.5,
> Emacs 20.5, and Allegro CL 5.0.1.
This is something I've meant to ask of Genera users and others that
keep their Lisp environments running a long time. How do you keep it
clean and in sync with the source code? I typically have to start from
a fresh image after a few days of development to check that my code
compiles without problems from source without a previously dumped
version available. Do you do stuff like calling FMAKUNBOUND on
functions you've renamed, work in a different package than
COMMON-LISP-USER and deleting it regularly?
--
Lieven Marchand <···@bewoner.dma.be>
Lambda calculus - Call us a mad club
Lieven Marchand <···@bewoner.dma.be> writes:
> This is something I've meant to ask of Genera users and others that
> keep their Lisp environments running a long time. How do you keep it
> clean and in sync with the source code? I typically have to start from
> a fresh image after a few days of development to check that my code
> compiles without problems from source without a previously dumped
> version available. Do you do stuff like calling FMAKUNBOUND on
> functions you've renamed, work in a different package than
> COMMON-LISP-USER and deleting it regularly?
>
I always found this a problem. I wasted a bunch of time in the late
80s using xerox lisp machines, which were pretty left-field machines,
and I found it far too easy to get into a state where the program I
was working on basically couldn't be loaded into a fresh image. I
found Genera better but still problematic. What I do now is to have a
`working' lisp which is connected to emacs and so on, but also to have
a way of quickly rebuilding the whole system from a shell-command
which cranks up a fresh lisp. I guess this is heretical in some way,
but Lisp compile times are so minute compared to C++ that even on a
fairly slow machine you can rebuild large systems from scratch pretty
quickly and without noticing any slowdown.
--tim
On 23 Nov 2000 19:14:49 +0000, Tim Bradshaw <···@tfeb.org> wrote:
> 80s using xerox lisp machines, which were pretty left-field machines,
What is a left-field machine?
Paolo
--
EncyCMUCLopedia * Extensive collection of CMU Common Lisp documentation
http://cvs2.cons.org:8000/cmucl/doc/EncyCMUCLopedia/
Paolo Amoroso <·······@mclink.it> writes:
> On 23 Nov 2000 19:14:49 +0000, Tim Bradshaw <···@tfeb.org> wrote:
>
> > 80s using xerox lisp machines, which were pretty left-field machines,
>
> What is a left-field machine?
I use `left-field' to mean `unconventional' or `strange' or something
like that. I guess the term comes from baseball, but I couldn't find
a definition that matches my one so may be I'm using it incorrectly.
Anyway, Xerox Lispms were pretty strange and unconventional, even by
the standards of the 80s when there was more variety in machines.
--tim
From: Will Deakin
Subject: Field [was Re: (declare (thread-special *foo* ...))]
Date:
Message-ID: <3A1E975E.5000408@pindar.com>
Tim wrote:
> I use `left-field' to mean `unconventional' or `strange' or something
> like that...but I couldn't find a definition that matches my one so
> may be I'm using it incorrectly.
FWIW this coincides with my understanding of the meaning of `left-field'
:)w
Kent M Pitman <······@world.std.com> writes:
> You're perhaps also not thinking about worlds like the Lisp Machine which
> are multithreaded and in which EVERY process ever run by any user is a lisp
> process. Lisp machines normally ran for about 3 months without booting,
> sometimes more than 6 months. If you imagine the effect accumulated of
> loading junk for 6 months that forever increases the number of per-process
> specials on all those processes, you can see the kind of "memory leak"
> effect that makes Unix have to get booted more often.
Unix isn't quite as bad as in the days of the "Suns boot real fast"
rant. After 20 years of work, they've become fairly stable. Most of my
servers only get rebooted for hardware or electricity problems.
--
Lieven Marchand <···@bewoner.dma.be>
Lambda calculus - Call us a mad club
I think you've missed some points, and I realize that some of this is
opinion. I agree that we don't need to add a declaration. Of course
-- Multiprocessing isn't even standard.
But my point was about "if it were".
But just to go through
Kent M Pitman <······@world.std.com> writes:
> David Bakhash <·····@alum.mit.edu> writes:
>
> > In each of these implementations we have different ways of declaring
> > variables to be special, but specifically within the context of a
> > thread, so that this data is not trampled on by other threads.
>
> Specials already get rebound per thread if already bound.
Here's the point. If there's a special variable:
(defvar *my-special-variable* nil)
then I might want different threads to NOT be able to hard each
other, with respect to this variable. This is EASY for threads to
do. One fine example of this is with something like *read-base*.
Applications that were NOT written to be thread-safe may have chosen
to do things like:
(setq *read-base* 2)
Unless this variable was somehow protected from one thread to another,
such a form as the one above would break code. Indeed, ACL handles
this case, but (for example) in LispWorks you just have to know that
*read-base* is unprotected, and then explicitly protect it. It's not
hard, but the point is that YOU HAVE TO KNOW.
The way I handle this is to tell LispWorks what variables I care about
protecting, and what initforms I want to use for them, on a per-thread
basis. And whereas LispWorks didn't seem to consider all the
variables that I happen to care about, ACL did, but initialized them
with initforms that I don't like (e.g. *readtable* gets bound, iirc,
to (copy-readtable nil) in each thread, so even if the caller had the
right *readtable* set, the callee ends up using the standard syntax.
It's almost as if running a thread is this
WITH-STANDARD-IO-SYNTAX-like call. It's not, and I don't know why
they implemented it that way. But at least the special variables are
protected, and I REALLY like knowing that threads don't corrupt each
other.
> Certainly you don't want all variables of all threads to get rebound in any
> thread. If they did, then the more threads you had, the more each would
> create work for the others. It's bad enough having lots of threads without
> having each contribute needless work to another.
No! One wouldn't want this for *all* of them; just some of them.
Certainly, the ones with affect the reader, printer, etc. Though this
might sound absolutely crazy to some people, if I run a thread, then
with the exception of singular data structures (which I'm _exlicitly_
sharing among threads) and the scheduling of threads, I want each
thread to operate as if it were in its own little Lisp world.
> Every thread that you own that cares should bind the variable itself
> and then each thread will bind exactly the number of variables it
> needs to.
Yeah, well this is nice in theory, but sometimes it's a lot to keep
track of.
I suppose that what you're saying is that one can write a WITH- macro
that binds the variables correctly, and this thing should enclose the
body of what you give to mp:process-run-function. I'm happier with a
thread-special version of DEFVAR and DEFPARAMETER, and having the
underlying implementation handle this issue. I realize, though, that
all you really need is this property that LET bindings on special
variables inside a thread essentially have thread-dynamic-extent (if
you don't mind the terminology). If the implementations didn't have
variables like mp:*process-initial-bindings*, then we could easily
have written our own such variable, and used it in conjunction with a
WITH- style macro enclosing the bodies of all mp:process-run-function
calls.
> You're perhaps also not thinking about worlds like the Lisp Machine which
> are multithreaded and in which EVERY process ever run by any user is a lisp
> process. Lisp machines normally ran for about 3 months without booting,
> sometimes more than 6 months. If you imagine the effect accumulated of
> loading junk for 6 months that forever increases the number of per-process
> specials on all those processes, you can see the kind of "memory leak"
> effect that makes Unix have to get booted more often. Being up a long time
> doesn't have to mean accumulated garbage, but it gets that way when people
> design systems that affect the "global" (which is what you're affecting when
> you ask the process system to do it instead of doing it yourself just for
> the processes you own) rather than the "local".
This is all completely irrelevant, as threads die just as processes
do, and their memory is allocated and then collected. What does
anything here have to do with memory leaks?
> Within your own process, it's so easy to set up something where you declare
> variables and then you do
> (WITH-MY-VARIABLES ...)
> around your own processes in a way that invades no one else's that it's hard
> to really see this as a major issue.
right. I agree (see above) just as long as bindings of special
variables within threads stay in the dynamic extent of those threads
only.
> By the way, the concept of "special" is already threadsafe, so the
> idea of a thread-special is redundant, imo.
It's a matter of definition. There is certainly different behavior
between typical special variables and special variables that have
alist entries in mp:*process-initial-bindings*. It's that difference
that I'm writing about.
> > For example, I might say:
> >
> > (let ((*foo* *initial-foo*))
> > (declare (thread-special *foo*))
> > (dotimes (i 10)
> > (mp:process-run-function
> > (format nil "bar ~d" i)
> > #+lispworks nil
> > #'bar)))
>
> This also doesn't make any sense because the THREAD-SPECIAL
> declaration you're making here would be lexical and would have zero
> effect on any of the other operations.
Okay. Let's say that the default behavior of this declaration were
such that it automatically gave any thread started in the dynamic
extent of the LET it's own value of *foo*.
Let's say *foo* and *initial-foo* are integers. Then a
(setq *foo* 99)
called in any of the 10 threads started above would not interfere,
provided that this ``thread-special'' declaration had the semantics
I'm suggesting. Without this declaration, however, I'd guess that
these threads would end up messing with the same variable, and would
those interfere with one another.
> If you're going to be a language designer, you should have as a rule
> of thumb that lexical declarations never affect separately compiled
> functions.
Point well taken, and I certainly agree. This was just an example,
and the points you're bringing up are exactly the reasons I'm asking.
> > So how are thread-special variables any different?
See above. I think it's pretty clear. Note that the LET is OUTSIDE
the call to mp:process-run-function; not inside, such as in the
#'(lambda () ...) form.
>
> [other stuff...]
>
> All of the above "just my opinion", for whatever that's worth.
I know, Kent. I don't want to add a declaration to the language. I
just want to figure out how such a declaration _would_ be added, and
also express the issues I have with multiprocessing in Common Lisp.
I'm still not convinced that the model is right, because the RHS
entries of the alist of mp:*process-initial-bindings* have no way to
specify unboundedness.
I'll do a test...
Let's say I declare a variable to be special:
(defvar *foobar*)
[note that I havn't givin it a value]
Now, I do this, say in ACL:
(push '(*foobar* . *foobar*) excl:*cl-default-special-bindings*)
to tell ACL that I want the *foobar* in any new threads to be
per-thread, and that I want their values to ``inherit'' from the
values they had before the thread was started.
(defun what-about-foobar? ()
(declare (special *foobar*))
(if (boundp '*foobar*)
(format t "~&The value of *foobar* is ~a.")
(format t "~&The variable *foobar* is unbound.")))
In ACL, I do this:
CL-USER(15): (mp:process-run-function "foobar test" #'what-about-foobar?)
==>
#<MULTIPROCESSING:PROCESS foobar test @ #x205b391a>
The variable *foobar* is unbound.
This is the behavior I expect. Now, let's check out LispWorks:
CL-USER 29 > (mp:process-run-function "foobar test" nil #'what-about-foobar?)
Error: The variable *foobar* is unbound.
1 (continue) Try evaluating *foobar* again.
2 Specify a value to use this time instead of evaluating *foobar*.
3 Specify a value to set *foobar* to.
4 (abort) Return to level 0.
5 Return to top loop level 0.
Do you see my point, Kent? I consider this to be a BUG. If something
is special, then it doesn't need to be bound.
Sure, I can complain to Xanalys about this ``minor'' problem that
probably no one else gives a damn about, and they'll surely fix it.
But I want people the Lisp community to take the whole threads issue
very seriously, because EVERY application that I find myself working
on relies heavily on threads. I love the genera-style multiprocessing
API, but I want it to make sense, and I want it to be integrated with
Common Lisp in a way that is correct and well thought-out. It is for
this reason that I posted, and I think that while you hit some key
points, you didn't really see my point, which is that the SPECIAL
declaration was not indended for multiple threads, but if they'd had
threads back then, they may likely have considered how to incorporate
them into the declaration.
dave
David Bakhash <·····@alum.mit.edu> writes:
> Here's the point. If there's a special variable:
>
> (defvar *my-special-variable* nil)
>
> then I might want different threads to NOT be able to hard each other...
"hear"? "hurt"? ...
> (setq *read-base* 2)
>
> Unless this variable was somehow protected from one thread to another,
> such a form as the one above would break code.
IMO, this is too bad and simply a lesson you have to learn. This form
will break things in a NON multiprocessing environment, too, as a rule.
e.g., you'll forget to unset it and some such. Global setqs are bad style
in general, IMO, unless you are actually creating the variable for the
first time and giving it an initial value or unless it's DEFVAR'd and
you are extending its value (e.g., pushing onto a list). Even then,
you can't just expect it to automatically be threadsafe since you have to
put without-preemption or some such around its update if you want to be
really clean.
> Indeed, ACL handles
> this case, but (for example) in LispWorks you just have to know that
> *read-base* is unprotected, and then explicitly protect it. It's not
> hard, but the point is that YOU HAVE TO KNOW.
Sorry, but my point here is that you DO have to know. There are no two
ways about this. ACL can protect *READ-BASE* because it wrote the base
system that gave you both the variable and the multitasking system, and it
can decide to protect it. But USERS can't simply decide how much overhead
other USERS can take.
Let me put it another way. Suppose there is a list called *THREADED-SPECIALS*.
The first thing I would push onto it would be *THREADED-VARIABLES* because
I don't trust someone else to be setting it.
> The way I handle this is to tell LispWorks what variables I care about
> protecting, and what initforms I want to use for them, on a per-thread
> basis.
LW is protecting you from unreasonable scheduling overhead by not
binding variables that you might not need to bind. ACL is protecting
you from being a moron. Who is right? Who knows? The market will
sort this out. But the language doesn't need to be imposing an
answer.
Curious because my memory is (and it's been a couple years, so this
info may be stale and vendors can correct me) it's LW that adds all
kinds of per-process "base" error handlers and/or restarts that I've
seen ACL not have. That is, in the handler domain, they have
a flipped sense of priorities as to whether the user needs "protection"
or can roll their own on a per-process basis. Again, though, nothing
required here (except, I think, an ABORT restart is required, and I
think once a long time ago ACL didn't have it; maybe they've added it).
> ACL did, but initialized them
> with initforms that I don't like (e.g. *readtable* gets bound, iirc,
> to (copy-readtable nil) in each thread, so even if the caller had the
> right *readtable* set, the callee ends up using the standard syntax.
> It's almost as if running a thread is this
> WITH-STANDARD-IO-SYNTAX-like call. It's not, and I don't know why
> they implemented it that way. But at least the special variables are
> protected, and I REALLY like knowing that threads don't corrupt each
> other.
And I really like knowing that if I want to communicate between
threads using a global, someone isn't going to introduce specials that
inhibit that. So what's your point? You have a personal preference and
want to impose it on others? It's so much easier to bind the variables
you care about per-process than for me to unbind them that my personal
leaning is toward saying they should not be bound without VERY careful
per-process thought.
Now you could make your own PROCESS-RUN-FUNCTION that added a wrapper
to the function to be run that made the appropriate environment in any
thread you personally were going to create. That would seem reasonable.
But having that be the global default for others? I don't think so.
Maybe it's just me, though.
> > Certainly you don't want all variables of all threads to get rebound in any
> > thread. If they did, then the more threads you had, the more each would
> > create work for the others. It's bad enough having lots of threads without
> > having each contribute needless work to another.
>
> No! One wouldn't want this for *all* of them; just some of them.
> Certainly, the ones with affect the reader, printer, etc.
This is a different matter entirely. We bind *PACKAGE* and
*READTABLE* in LOAD and COMPILE-FILE because they are system-given.
Having the standard or an implementation specify that certain
variables are bound per-process is within the realm of reason because
it's O(1) to accomodate. That is, the number doesn't change.
Once you open it to users, you slow all processes down in a way that's
O(n) where n is a quantity that grows with the ego of the designer
of each module loaded.
> Though this
> might sound absolutely crazy to some people, if I run a thread, then
> with the exception of singular data structures (which I'm _exlicitly_
> sharing among threads) and the scheduling of threads, I want each
> thread to operate as if it were in its own little Lisp world.
But only your own threads. And this is easy to arrange. You shouldn't
be assuming all threads have this desire.
> > Every thread that you own that cares should bind the variable itself
> > and then each thread will bind exactly the number of variables it
> > needs to.
>
> Yeah, well this is nice in theory, but sometimes it's a lot to keep
> track of.
>
> I suppose that what you're saying is that one can write a WITH- macro
> that binds the variables correctly, and this thing should enclose the
> body of what you give to mp:process-run-function. I'm happier with a
> thread-special version of DEFVAR and DEFPARAMETER, and having the
> underlying implementation handle this issue.
You're happier with your notion that you are the king of the world.
But in a shared, multiuser, multiprocessed lisp, you wouldn't be. You
just don't expect that to happen. And I think that's shortsighted and
sad. Global is bad. Don't be assigning global things. Including
don't be assigning global lists of what shouldn't be global. There are
better ways to implement local.
> This is all completely irrelevant, as threads die just as processes
> do, and their memory is allocated and then collected. What does
> anything here have to do with memory leaks?
If you create variables that are to be protected and even delete the package,
the symbols in the deleted package live on. If you've got them on a global
list of variables to bind per-thread, they will continue to be bound even
if no code continues to look at them.
> Let's say *foo* and *initial-foo* are integers. Then a
>
> (setq *foo* 99)
>
> called in any of the 10 threads started above would not interfere,
> provided that this ``thread-special'' declaration had the semantics
> I'm suggesting. Without this declaration, however, I'd guess that
> these threads would end up messing with the same variable, and would
> those interfere with one another.
This passion you have for using SETQ is confusing to me. SETQ is an operator
that says "assign this all the way out". In a multitasking environment,
you don't know how far "out" there is. Writing any modern code in a way
that assumes it owns the world is just bad. If you're talking legacy code
you inherited from the 1970's I can almost have sympathy for you, but if
you're saying you want to write new code in this style and you want the
language to support it, I'm saying you're doing something I don't approve
of. SETQ is to LET like RETURN is to BLOCK. I hear you saying you want to
write "RETURN" but no "BLOCK" and you want it to be "obvious" where the
BLOCK is. But I don't agree that any program can properly know the context
in which it is running.
This is like the argument we used to hear from people ages ago that Lisp
should have a QUIT or a GC function. We didn't add either but people
kept pestering us for them. No one wanted to address the fact that QUIT on
a lisp machine would halt the machine. No one wanted to address the issue
that some gc's stop the machine for milliseconds and some for hours. You
can't assume you own the world and that your invoking powerful
potentially-global operations like this can be made safe just by your say-so.
The need for QUIT was largely obviated by CLIM having an exit-application
operation (I forget the name) which exits a with-application-context (again
I forget the name) so that they work in pairs. This is not a global operation
though--now it's just dynamic.
> > All of the above "just my opinion", for whatever that's worth.
>
> I know, Kent. I don't want to add a declaration to the language. I
> just want to figure out how such a declaration _would_ be added, and
> also express the issues I have with multiprocessing in Common Lisp.
This has all been about multiprocessing?? I thought this was all about
accomodating your lack of desire to think about multiprocessing while still
having the multiprocessing environment tolerate that desire. I *really*
feel even more strongly that you're in trouble here if you think this
is a good style to advocate for people who are "thinking about
multiprocessing". You'd have to show me actual examples of the code you
want to write, not just little one-liner setqs for me to comment further
on this.
> Now, I do this, say in ACL:
>
> (push '(*foobar* . *foobar*) excl:*cl-default-special-bindings*)
It's nice they have this list, but my first impulse is not to push
onto it but to pop from it. What if someone else pops these bindings
thinking they are excessive? Of course, that would break programs.
So you couldn't do it. A push-only list seems bad, though, unless
it's manipulated only by the system and never the user.
> (defun what-about-foobar? ()
> (declare (special *foobar*))
> (if (boundp '*foobar*)
> (format t "~&The value of *foobar* is ~a.")
> (format t "~&The variable *foobar* is unbound.")))
>
> In ACL, I do this:
>
> CL-USER(15): (mp:process-run-function "foobar test" #'what-about-foobar?)
>
> ==>
> #<MULTIPROCESSING:PROCESS foobar test @ #x205b391a>
> The variable *foobar* is unbound.
>
> This is the behavior I expect. Now, let's check out LispWorks:
>
> CL-USER 29 > (mp:process-run-function "foobar test" nil #'what-about-foobar?)
> Error: The variable *foobar* is unbound.
> 1 (continue) Try evaluating *foobar* again.
> 2 Specify a value to use this time instead of evaluating *foobar*.
> 3 Specify a value to set *foobar* to.
> 4 (abort) Return to level 0.
> 5 Return to top loop level 0.
>
> Do you see my point, Kent? I consider this to be a BUG. If something
> is special, then it doesn't need to be bound.
Huh. Sounds like a bug to me, too. It doesn't blow up in the
LW Personal Edition I have on the machine I'm typing to. Did you just
report it as a bug before redesigning the language? Maybe you just need
a patch? This certainly illustrates nothing to me.
> Sure, I can complain to Xanalys about this ``minor'' problem that
> probably no one else gives a damn about, and they'll surely fix it.
I imagine they'll fix bugs. BOUNDP is supposed to work in the situation
you describe (and did work when I did exactly what you did).
> But I want people the Lisp community to take the whole threads issue
> very seriously, because EVERY application that I find myself working
> on relies heavily on threads.
To me, taking the issue seriously would involve getting the vendors to agree
on whether the process options are given as a list in the first argument to
mp:process-run-function or as a second argument. sigh... If they don't agree
on that...
> I love the genera-style multiprocessing
> API, but I want it to make sense, and I want it to be integrated with
> Common Lisp in a way that is correct and well thought-out.
Me, too. I just don't think you have the experience that the Genera people
had with multitasking CL, and I don't think you are seeing the consequences
of what you are suggesting.
> It is for
> this reason that I posted, and I think that while you hit some key
> points, you didn't really see my point, which is that the SPECIAL
> declaration was not indended for multiple threads, but if they'd had
> threads back then, they may likely have considered how to incorporate
> them into the declaration.
Honestly, I continue to believe you're confused on a lot of this.
It will take a worked program example to show me otherwise.
Kent M Pitman <······@world.std.com> writes:
> This passion you have for using SETQ is confusing to me. SETQ is an
> operator that says "assign this all the way out". In a multitasking
> environment,
I _never_ use SETQ like this. I only work with code that does, and
have seen it in two separate instances, and thought that there might
be a way to obviate the rewriting of it by making threads safer in
Common Lisp. I don't think this is asking too much.
> > (push '(*foobar* . *foobar*) excl:*cl-default-special-bindings*)
>
> It's nice they have this list, but my first impulse is not to push
> onto it but to pop from it. What if someone else pops these
> bindings thinking they are excessive? Of course, that would break
> programs. So you couldn't do it. A push-only list seems bad,
> though, unless it's manipulated only by the system and never the
> user.
I was giving an example, and keeping it short. The more likely way to
use these implementation-specific variables is either in a LET form
would have been used would use a LET (under LW) and using a special
option to mp:process-run-function on ACL.
> > I love the genera-style multiprocessing
> > API, but I want it to make sense, and I want it to be integrated with
> > Common Lisp in a way that is correct and well thought-out.
>
> Me, too. I just don't think you have the experience that the Genera people
> had with multitasking CL, and I don't think you are seeing the consequences
> of what you are suggesting.
I definitely don't. Never said I did. I just notice places where
implementations don't seem to work similarly.
As for this particular issue, I do have an opinion, which is similar
to both LW and ACL. I'd like each thread to have it's own copy of
these variables (similar to both, in that sense), but for the list of
such variables to include the standard global reader and printer
variables that are part of CL (i.e. like ACL). But where such
variables are included, I want them to inherit their state from the
caller (like LW), rather than from the default settings that CL ships
with (i.e. the ACL way).
> > It is for this reason that I posted, and I think that while you
> > hit some key points, you didn't really see my point, which is that
> > the SPECIAL declaration was not indended for multiple threads, but
> > if they'd had threads back then, they may likely have considered
> > how to incorporate them into the declaration.
>
> Honestly, I continue to believe you're confused on a lot of this.
> It will take a worked program example to show me otherwise.
instead of shipping you the quarter Meg of code that I didn't write,
but that I'm working on right now, just imagine code that SETQ's all
sorts of global variables, some of which are CL ones, and some which
are defined by the program. This program has a #'MAIN is to become
multi-threaded and thus thread-safe, and while I don't like this SETQ
style, I also have to ask myself the question "Does _bad_ style really
imply _wrong_ style?" In other words, should this code break just
because the author used SETQ, even if he knew that the code might one
day be multi-threaded? We may differ on this point, but I don't think
this should break his code. I'd rather just change the DEFVARs and
DEFPARAMETERs to slightly different ones that keep things working. If
I tried to explain all the reasons, I'd get stuck, because it's hard
to express. That's why I'm raising these points -- so that people who
might care and who understand these issues might give it some thought.
dave
David Bakhash <·····@alum.mit.edu> writes:
> Kent M Pitman <······@world.std.com> writes:
>
> > ... I just don't think you have the experience that the Genera people
> > had with multitasking CL, and I don't think you are seeing the
> > consequences of what you are suggesting.
>
> I definitely don't. Never said I did. I just notice places where
> implementations don't seem to work similarly.
I do agree it's a problem that these are issues on which there is
substantial agreement between vendors on common facilities and they
differ in ways that appear arbitrary. That can indeed be vexing when
trying to arrange easy porting. I remember that the fight over whether
rest lists would copy or not was painful, because people had to defensively
copy rest list arguments just in case they weren't copied, but then they
were risking doing extra copying. (That problem is nailed down, though
a similar problem exists for APPLY and sharing with the argument that is
still a mess, as I recall.)
This is why I said it matters whether it's provided by the system (which
is O(1)--doesn't grow with modules added by users) or provided by the users
(which creates an O(f(n)) situation for n users, where f(n) is unclear
but is probably not a constant). I'm all for nailing down the system
stuff. I think users have to come up with some other theory.
For example, I think it's good form to have facilities that work like
the following, which many of my systems have:
(define-variable-group *foo-variables*
:definer def-foo-var
:binder with-foo-vars)
(def-foo-var *foo* ...)
(def-foo-var *bar* ...)
... (with-foo-vars ...) ...
where WITH-FOO-VARS maybe turns into a PROGV with appropriate inits.
(I also think it's reasonable to go into "special form" space to add
PROGW, like the lisp machine, which takes an alist instead of two
disconnected lists like PROGV takes. The PROGV version almost always
forces gratuitous consing in my experience.) But other than the issue
of PROGW, I don't see any rason that the stuff I'm talking about now
needs any system support at all. Users can define it and if a particular
package is popular, everyone can just always use that.
> > > It is for this reason that I posted, and I think that while you
> > > hit some key points, you didn't really see my point, which is that
> > > the SPECIAL declaration was not indended for multiple threads, but
> > > if they'd had threads back then, they may likely have considered
> > > how to incorporate them into the declaration.
> >
> > Honestly, I continue to believe you're confused on a lot of this.
> > It will take a worked program example to show me otherwise.
>
> instead of shipping you the quarter Meg of code that I didn't write,
> but that I'm working on right now, just imagine code that SETQ's all
> sorts of global variables, some of which are CL ones, and some which
> are defined by the program. This program has a #'MAIN is to become
> multi-threaded and thus thread-safe, and while I don't like this SETQ
> style, I also have to ask myself the question "Does _bad_ style really
> imply _wrong_ style?"
The right way to answer this question is to look at what good style means.
Good style is not "dogma" executed without thinking. Good style is a clear
knowledge of the problems you can run into and programming with appropriate
defensiveness. If bad style can lead you to common programming errors and
you program using it and then run into those entirely predictable problems,
then indeed, it is wrong style. If you have some other solution that you
have put in place that covers you, it becomes ok style. Because the style
wasn't the problem--the risks were the problem, and you just have to cover
them somehow. But what you are asking for is to say "I want to do something
that is an inherent risk unless you change the language, and I consider it
a bug if the language doesn't protect me from myself in the absence of said
change". There are lots of ways the language does not protect you, and each
of them could in principle be "fixed" by a change to the language, at some
risk of "other problems".
You have not to me made a good case that you need the thing you are asking
for, but that's just me.
You also to me have not made a good case that there aren't just as many
problems with what you're asking for, but again that's just me.
I'm not the X3J13 committee. I'm just a crusty old guy who's lived this a
long time and thinks based on a lifetime of experience that the thing you're
asking for doesn't sound right. That doesn't mean I'm right. It could just
mean I'm being too stuck in my ways. Maybe you should try to convince someone
other than me and maybe one of them will express it in a way that sounds
more palatable than it does to me. Or maybe I'll just be outvoted in the
court of public opinion regardless of what I think. But for now, I think
I'm not the person that you should most productively be trying to convince.
> In other words, should this code break just
> because the author used SETQ, even if he knew that the code might one
> day be multi-threaded?
There are a hundred reasons why it's appropriate to build appropriate binding
abstractions even in the absence of multithreading. The mere presence of
a "breakpoint" interrupt is enough reason to do things differently, since even
in a single-processed lisp (as Maclisp was) one might recursively reinvoke
the same procedure from a breakpoint that can see the outer bindings of a
prior invocation. Someone failing to program against this is, IMO, asking
for trouble. And global setqs screw this as well.
> We may differ on this point, but I don't think
> this should break his code.
I don't think it "should". But then, I don't think pi should be an
irrational number. Sometimes even language designers just don't get
to pick. I think the fact that this is a problematic programming style
goes way beyond the one design choice you're talking about.
I do think a tool like what you're talking about might be ok for a limited
toolbox extension situation like emacs or maybe something autocad-like,
where there is a very limited notion of what the toplevel environment
might look at and where the focus was on building tools to help a specific
interaction model rather than building arbitrarily shaped applications.
I think your idea has a very "engineering" and not "linguistic" feel to it,
and is inappropriate at the language level.
> I'd rather just change the DEFVARs and
> DEFPARAMETERs to slightly different ones that keep things working. If
> I tried to explain all the reasons, I'd get stuck, because it's hard
> to express. That's why I'm raising these points -- so that people who
> might care and who understand these issues might give it some thought.
I think you just can't do that because you aren't case-marking all the
places in code that don't matter. The DEFVARs and DEFPARAMETERs do not
occur at the "other place" (which you allege is the top of every process
and I claim is more variable than that and simply cannot be reliably
inferred by so trivial an automatic means). I'm all for definining things
in a central place where there's a way of knowing that, sight unseen, the
effects of what you're creating on other code is the right thing. I just
don't think you can do that in this case.
Kent M Pitman <······@world.std.com> writes:
> David Bakhash <·····@alum.mit.edu> writes:
>
> > Kent M Pitman <······@world.std.com> writes:
> >
> > > ... I just don't think you have the experience that the Genera people
> > > had with multitasking CL, and I don't think you are seeing the
> > > consequences of what you are suggesting.
> >
> > I definitely don't. Never said I did. I just notice places where
> > implementations don't seem to work similarly.
>
> I do agree it's a problem that these are issues on which there is
> substantial agreement between vendors on common facilities and they
> differ in ways that appear arbitrary. That can indeed be vexing when
> trying to arrange easy porting. I remember that the fight over whether
> rest lists would copy or not was painful, because people had to defensively
> copy rest list arguments just in case they weren't copied, but then they
> were risking doing extra copying. (That problem is nailed down, though
> a similar problem exists for APPLY and sharing with the argument that is
> still a mess, as I recall.)
This is an interesting way of looking at it. There are a lot of ways
in which Common Lisp protects programmers. But obviously, when
there's an issue, the language designers had to determine "Should we
build this protection into the language?" That's definitely
applicable here.
> I'm not the X3J13 committee. I'm just a crusty old guy who's lived
> this a long time and thinks based on a lifetime of experience that
> the thing you're asking for doesn't sound right.
I'm not exactly sure what you mean here. What I'm asking for
functionally exists under two of the implementations that I've used
(i.e. these kinds of variables). I guess where you felt this "doesn't
sound right" thing was about the declaration. I understand that
completely. I'm starting to agree with you that if there _were_ a
declaration, it would be an awkward and scary one, and it would even
change the way declarations work. Also, I havn't thought through all
of the possible scenarios, and probably couldn't, which is why I
posted.
> That doesn't mean I'm right. It could just mean I'm being too stuck
> in my ways. Maybe you should try to convince someone other than me
> and maybe one of them will express it in a way that sounds more
> palatable than it does to me. Or maybe I'll just be outvoted in the
> court of public opinion regardless of what I think. But for now, I
> think I'm not the person that you should most productively be trying
> to convince.
Kent. I think you're too into this "my way, your way" here. I'm not
trying to convince anyone as much as I'm trying to get feedback about
why it's a good or bad idea. My model of how threads work in CL is
fine at the application level; I know how to use them, and I get by.
My knowledge at the design level is lacking if even existent.
> > In other words, should this code break just because the author
> > used SETQ, even if he knew that the code might one day be
> > multi-threaded?
>
> There are a hundred reasons why it's appropriate to build appropriate binding
> abstractions even in the absence of multithreading. The mere presence of
> a "breakpoint" interrupt is enough reason to do things differently, since even
> in a single-processed lisp (as Maclisp was) one might recursively reinvoke
> the same procedure from a breakpoint that can see the outer bindings of a
> prior invocation. Someone failing to program against this is, IMO, asking
> for trouble. And global setqs screw this as well.
Yeah. The SETQ things are a bad idea. This is where CL really
shined. As "broken" as this code was, I was very happy with the fact
that I could write two very similar macros to sub in for DEFVAR and
DEFPARAMETER and the system would work again, and that I used a
facility that the implementations all seem to have. It was in
thinking about this solution that I imagined a declaration. If one
existed, something like:
(proclaim '(thread-special *x* *y*))
Yeah. you're probably looking at this with disgust, but for my
purposes, the feedback was useful.
dave
In article <··············@cadet.dsl.speakeasy.net>, David Bakhash
<·····@alum.mit.edu> wrote:
> Hi,
>
> I'm writing because I wanted to suggest a new type of declaration that
> would be useful for the implementations of CL which have
> multiprocessing.
>
> In each of these implementations we have different ways of declaring
> variables to be special, but specifically within the context of a
> thread, so that this data is not trampled on by other threads.
Hmm, as Kent pointed out, special variables are already thread safe.
On the Symbolics Lisp Machine you have "stack groups" as a building
block for processes. Each stack group has its own dynamic binding
stack.
What you don't have is "per process global variables". Does
any Lisp offer these? Documented?
It might also be interesting to have processes that by
default cannot access any global symbol/function values -
other than those explicitly allowed. For example
one would create a new process and then give the
process a set of symbol/functions/package/classes it is allowed to
access via something like SYMBOl-FUNCTION (hey, it's all caps),
SYMBOL-VALUE, FIND-PACKAGE, FIND-CLASS, ...
Rainer Joswig
--
Rainer Joswig, Hamburg, Germany
Email: ·············@corporate-world.lisp.de
Web: http://corporate-world.lisp.de/