Let's examine some potential pitfalls first:
(let (x)
(unwind-protect
(progn
(setq x (open ...))
...)
(when x (close x))))
- If an async interrupt hits and aborts when open has finished but
before its return value is assigned to X, then a file descriptor is
leaked.
- In cl implementations that don't do anything special for
unwind-protect if an async interrupt hits and aborts in the cleanup
form then X can remain open.
A potential solution is to wrap (setq x (open ...)) in
without-interrupts (the variety that inhibits interrupts (C-c,
process-interrupt, SIGALRM) to the running process, but does not
prevent other processes/threads from running). Unfortunately, if open
takes a long time then among other things it cannot be C-c-ed and
debugged. Similarly, automatically inhibiting interrupts in cleanup
forms is too strong.
Reading the old usenet posts about unwind-protect and async interrupts
revealed a great deal of confusion about how things should behave. One
intriguing idea is that of without-aborts mentioned by kmp. From what I
gathered it prevented "asynchronous non-local exits" and cleanup forms
were automatically protected by it on lisp machines of old.
I speculate that this construct only inhibits non-local exits that are
initiated from an interrupt and would "cross" the limit marked with
without-aborts. But it could as well inhibit leaving the interrupt
function by a non-local exit.
I'm very interested in what this construct did, why, even to the detail
of what restarts were provided, where in the system was it used, was it
successful, are there better solutions ...
Thanks, Gábor Melis
····@hotpop.com writes:
> A potential solution is to wrap (setq x (open ...)) in
> without-interrupts (the variety that inhibits interrupts (C-c,
> process-interrupt, SIGALRM) to the running process, but does not
> prevent other processes/threads from running). Unfortunately, if open
> takes a long time then among other things it cannot be C-c-ed and
> debugged. Similarly, automatically inhibiting interrupts in cleanup
> forms is too strong.
You don't need to debug code that works correctly. It is more
important to have the ability to write correct code than to have the
ability to debug incorrect code.
> Reading the old usenet posts about unwind-protect and async interrupts
> revealed a great deal of confusion about how things should behave.
No, there is no confusion, but there is a clear difference of opinion.
> One intriguing idea is that of without-aborts mentioned by kmp. From
> what I gathered it prevented "asynchronous non-local exits" and
> cleanup forms were automatically protected by it on lisp machines of
> old.
Yes.
> I speculate that this construct only inhibits non-local exits that are
> initiated from an interrupt and would "cross" the limit marked with
> without-aborts. But it could as well inhibit leaving the interrupt
> function by a non-local exit.
Interrupts that are asynchronous are not handled until the form is
complete. Non-local exits must work as `normal' (or it would not
comply with Common Lisp).
> I'm very interested in what this construct did, why, even to the detail
> of what restarts were provided, where in the system was it used, was it
> successful, are there better solutions ...
This is dependent upon implementation. Lucid simply deferred the
interrupts until it was safe to handle them. I wrote one for Allegro
that kept track of how many asynchronous interrupts occurred and
offered to force an asynchronous interrupt when enough interrupts
accumulated.
Exactly what features are provided isn't as important as whether the
feature is provided at all.
~jrm
Joe Marshall <·········@alum.mit.edu> writes:
> Interrupts that are asynchronous are not handled until the form is
> complete. Non-local exits must work as `normal' (or it would not
> comply with Common Lisp).
To me it seems unreasonable to defer handling interrupts for unknown
amounts of time. How about a strategy that any non-local exit that
crosses first an asynchronous interrupt event and then a
without-asynchronous-aborts marker, the exit is queued up for
execution once the without-asynchronous-aborts exits? If a second such
exit must be enqueued, the farthest exit is given presedence (so that
"queuing up" really just means to overwrite the default continuation
of the without-asynchronous-aborts frame).
--
Frode Vatvedt Fjeld
I agree that without-interrupts is problematic, not just for debugging
but for with-timeout and process-interrupt as well.
For example
(with-timeout 10
(without-interrupts (setq x (open ...))))
is unlikely to time out.
I like the double marker approach you describe and it has the look of a
possible implementation. The farthest exit wins heuristic also sounds
reasonable, but semantic traps might be lurking here. The lisp machine
offered a restart instead of queueing or so kmp says.
G�r
····@hotpop.com writes:
> I agree that without-interrupts is problematic, not just for
> debugging but for with-timeout and process-interrupt as well.
>
> For example
>
> (with-timeout 10
> (without-interrupts (setq x (open ...))))
>
> is unlikely to time out.
Not if the file-system in question is something like a WAN filesystem
(such as one of those FTP filesystem interfaces etc.)?
> I like the double marker approach you describe and it has the look
> of a possible implementation. The farthest exit wins heuristic also
> sounds reasonable, but semantic traps might be lurking here.
I agree about there being potential traps. I'm not sure though if
these traps should be attributed to this particular scheme for
without-async-aborts, or to the introduction of asynchronous frames
(with effects visible to normal program flow) per se. I think in a
certain sense all standard semantics break down when "anything" can
happen at any time in a program, independently of what that program
is.
> The lisp machine offered a restart instead of queueing or so kmp
> says.
What kind of restart, exactly? And how would this solve/help resolve
e.g. the with-timeouts/unwind-protect problem?
--
Frode Vatvedt Fjeld
Google groups and quoting by hand sucks.
> > For example
> > (with-timeout 10
> > (without-interrupts (setq x (open ...))))
> > is unlikely to time out.
> Not if the file-system in question is something like a WAN filesystem
> (such as one of those FTP filesystem interfaces etc.)?
'Unlikely' is just the understated form of 'never' here because of the
without-interrupts around (setq x (open ...)).
> What kind of restart, exactly? And how would this solve/help resolve
> e.g. the with-timeouts/unwind-protect problem?
Oh, it does not solve it in itself. But if without-aborts is useful I'm
interesting in the whole picture. Maybe I mistakenly thought there was
a restart because what Kent says is:
"On the LispM, if you try to interactively abort an unwind, for
example, you
get an interactive query about the fact that a without-aborts is active
and
asking how serious you are about the abort. "
Gabor
On 4 Sep 2005 13:48:55 -0700, ····@hotpop.com wrote:
>Let's examine some potential pitfalls first:
>
>(let (x)
> (unwind-protect
> (progn
> (setq x (open ...))
> ...)
> (when x (close x))))
>
>- If an async interrupt hits and aborts when open has finished but
>before its return value is assigned to X, then a file descriptor is
>leaked.
>
>- In cl implementations that don't do anything special for
>unwind-protect if an async interrupt hits and aborts in the cleanup
>form then X can remain open.
>
>A potential solution is to wrap (setq x (open ...)) in
>without-interrupts (the variety that inhibits interrupts (C-c,
>process-interrupt, SIGALRM) to the running process, but does not
>prevent other processes/threads from running). Unfortunately, if open
>takes a long time then among other things it cannot be C-c-ed and
>debugged. Similarly, automatically inhibiting interrupts in cleanup
>forms is too strong.
I haven't used or worked with CL much so please be gentle as I may be
completely misinterpreting something. I have done both OS and
language runtime support implementation - just not with Lisp.
Is something like the example above *really* a problem? Perhaps I am
getting confused by terminology ... are you really talking about
"interrupts" or are you referring to conditions/signals/exceptions?
Interrupts should be transparently handled at the OS level [even in a
Lisp OS] and converted into some form of controllable [ignorable,
delayable, restartable, etc.] signal to the application.
Maybe you mean that Lisp allows conditions to be signaled between the
"open" call and the "setq". But properly, conditions which could
affect the result of open should have already been handled before the
call returns and unrelated, continuable conditions should have no
effect apart from delays due to context switches into and out of the
handlers. The return value of the function should *never* be lost.
Even if the function starts an asynchronous operation that can fail
later, the condition will be signaled either in unrelated code or
while synchronously waiting for the operation to complete. Either
way, the handler should simply return to the previous context.
Unrecoverable conditions are ... well ... unrecoverable. Who cares if
they corrupt the application's environment? Of course, a Lisp system
could be viewed as many simultaneous micro applications and the
condition might affect only one of them. But that's an environment
integrity issue for the implementation.
>Reading the old usenet posts about unwind-protect and async interrupts
>revealed a great deal of confusion about how things should behave. One
>intriguing idea is that of without-aborts mentioned by kmp. From what I
>gathered it prevented "asynchronous non-local exits" and cleanup forms
>were automatically protected by it on lisp machines of old.
I haven't read these threads but I'm assuming from your comment that
they were discussing machines where Lisp was the OS. See above.
George
--
for email reply remove "/" from address
I guess it's time to rehash this.
1. The Common Lisp standard does not address multi-processing or
multi-threading.
2. The Common Lisp standard does not address asynchronous
interrupts or events.
3. There are several implementation-dependent extensions of Common
Lisp that provide facilities such as multi-processing or
asynchronous interrupts.
> On 4 Sep 2005 13:48:55 -0700, ····@hotpop.com wrote:
>>Let's examine some potential pitfalls first:
>>
>>(let (x)
>> (unwind-protect
>> (progn
>> (setq x (open ...))
>> ...)
>> (when x (close x))))
>>
>>- If an async interrupt hits and aborts when open has finished but
>>before its return value is assigned to X, then a file descriptor is
>>leaked.
This cannot happen in some implementations.
>>- In cl implementations that don't do anything special for
>>unwind-protect if an async interrupt hits and aborts in the cleanup
>>form then X can remain open.
This cannot happen in some implementations. Other implementations
defer handling asynchronous interrupts *for only that thread* during
the protected form.
>>A potential solution is to wrap (setq x (open ...)) in
>>without-interrupts (the variety that inhibits interrupts (C-c,
>>process-interrupt, SIGALRM) to the running process, but does not
>>prevent other processes/threads from running). Unfortunately, if open
>>takes a long time then among other things it cannot be C-c-ed and
>>debugged. Similarly, automatically inhibiting interrupts in cleanup
>>forms is too strong.
>
George Neuner <·········@comcast.net> writes:
> I haven't used or worked with CL much so please be gentle as I may be
> completely misinterpreting something. I have done both OS and
> language runtime support implementation - just not with Lisp.
>
> Is something like the example above *really* a problem?
It can be. The example above may be a little *too* simple to
exhibit the problem.
> Perhaps I am getting confused by terminology ... are you really
> talking about "interrupts" or are you referring to
> conditions/signals/exceptions?
That's a good question. `Interrupts' in this case mean any sort of
asynchronous change in control flow where another chunk of arbitrary
Lisp code might be run. This can be different in different
implementations. Lucid allowed a Lisp program to be interrupted
between any two instructions, but Allegro polls for interrupts at `safe
points' within a program.
> Interrupts should be transparently handled at the OS level [even in a
> Lisp OS] and converted into some form of controllable [ignorable,
> delayable, restartable, etc.] signal to the application.
>
> Maybe you mean that Lisp allows conditions to be signaled between the
> "open" call and the "setq". But properly, conditions which could
> affect the result of open should have already been handled before the
> call returns and unrelated, continuable conditions should have no
> effect apart from delays due to context switches into and out of the
> handlers. The return value of the function should *never* be lost.
You never know what might happen between the OS call that opens a file
and the construction and return of the resulting Lisp stream object.
The situation that causes problems is something like a timeout or a
user interrupt where we want to safely abort the thread that is
performing the open.
> Unrecoverable conditions are ... well ... unrecoverable. Who cares if
> they corrupt the application's environment?
There's different levels of unrecoverable. You wouldn't want an
unrecoverable event in a single process to take down the entire OS.
Similarly, you wouldn't want an unrecoverable event in a single thread
to take out the entire Lisp process.
On Tue, 06 Sep 2005 16:10:42 -0400, Joe Marshall
<·········@alum.mit.edu> wrote:
>George Neuner <·········@comcast.net> writes:
>
>> Perhaps I am getting confused by terminology ... are you really
>> talking about "interrupts" or are you referring to
>> conditions/signals/exceptions?
>
>That's a good question. `Interrupts' in this case mean any sort of
>asynchronous change in control flow where another chunk of arbitrary
>Lisp code might be run.
Ok.
>You never know what might happen between the OS call that opens a file
>and the construction and return of the resulting Lisp stream object.
>The situation that causes problems is something like a timeout or a
>user interrupt where we want to safely abort the thread that is
>performing the open.
I'm not a fan of aborting threads in mid-execution. I much prefer
that an async handler preserve whatever data is necessary and resume
the thread using some kind of exception mechanism. That way most
stuff is done visibly in the thread code.
This isn't *terribly* difficult to do in C/C++ ... though in systems
having only per process handlers, the first problem is often
identifying the offending thread. After that it is just a matter of
tidying up a bit and changing the thread resume point to the start of
the situation error handling code.
With CL, though, I'm not yet sure how handlers really work nor how
much power handlers have to inspect and modify the state of the system
- particularly WRT the context(s) in which they can be executed and
whether they can inspect and modify unnamed temporaries and other
values not held in GC roots.
George
--
for email reply remove "/" from address
George Neuner <·········@comcast.net> writes:
> I'm not a fan of aborting threads in mid-execution.
You generally can't get much work done if your threads are aborted in
mid-execution, but sometimes you have to do that and making it work in
some sane way is good.
> I much prefer that an async handler preserve whatever data is
> necessary and resume the thread using some kind of exception
> mechanism. That way most stuff is done visibly in the thread code.
Just un-registering the thread with the OS and discarding the stack is
the least friendly way of aborting a thread, and it shouldn't be done
except as a last resort. While Lucid had the ability to interrupt
between any two machine instructions, what usually happened is that
the interrupt handler would perform a thread switch (Lucid implemented
its own `green threading'). If it was a control-c, an asynchronous
break would be handled in the thread. The user could enter the
debugger at this point and abort the thread, but this would be a
`lisp-level' abort that would appropriately unwind the stack.
A timeout could also cause a lisp abort to unwind a stack, or a
program could raise an abort in another thread.
Lucid had a notion of `interruptions' (wonderful name, eh?). An
`interruption' was a chunk of lisp code that would be run within the
context of the receiving thread just before the thread regained
control during a process switch. They appear to lisp to be
asynchronous, but the OS sees them as synchronous. When a thread was
in the cleanup of an unwind-protect, processing of `interruptions'
would be deferred until the cleanup was finished. `Interrupts', that
is, the actual clock ticks, control-C's, page-faults, etc., would be
handled immediately, but would never directly cause a thread abort.
If you `killed' a thread, it would actually push an `interruption'
that would cause the thread to kill itself at the next safe point.
On Thu, 08 Sep 2005 10:39:20 -0400, Joe Marshall
<·········@alum.mit.edu> wrote:
>George Neuner <·········@comcast.net> writes:
>
>> I'm not a fan of aborting threads in mid-execution.
>
>You generally can't get much work done if your threads are aborted in
>mid-execution, but sometimes you have to do that and making it work in
>some sane way is good.
Maybe I've been programming under a rock for 20 years, but I've never
come across a situation where I considered forcibly terminating a
thread to be the right design. OTOH, my experience has been mostly of
lousy runtime support for signaling.
When I have a situation where a thread might need to be killed, I
design it as event driven from the start. It complicates things a bit
with code that would be straightforward apart from the need to commit
suicide, but I've become quite adept at forcing code into that model.
>Lucid had a notion of `interruptions' (wonderful name, eh?). An
>`interruption' was a chunk of lisp code that would be run within the
>context of the receiving thread just before the thread regained
>control during a process switch. They appear to lisp to be
>asynchronous, but the OS sees them as synchronous.
This is more in line with my idea of language level signal handling.
>When a thread was
>in the cleanup of an unwind-protect, processing of `interruptions'
>would be deferred until the cleanup was finished. `Interrupts', that
>is, the actual clock ticks, control-C's, page-faults, etc., would be
>handled immediately, but would never directly cause a thread abort.
>If you `killed' a thread, it would actually push an `interruption'
>that would cause the thread to kill itself at the next safe point.
And it seems like Lucid had a fairly reasonable and safe way to cause
termination of a thread. The closest thing I have used is Java's
exception based thread termination model - which still has a number of
problems including dependency on the design of classes and exception
handlers to prevent leaking OS resources.
George
--
for email reply remove "/" from address
George Neuner wrote:
> Maybe I've been programming under a rock for 20 years, but I've never
> come across a situation where I considered forcibly terminating a
> thread to be the right design. OTOH, my experience has been mostly of
> lousy runtime support for signaling.
It's not good design, but otherwise, how do you kill a hanging thread?
> When I have a situation where a thread might need to be killed, I
> design it as event driven from the start. It complicates things a bit
> with code that would be straightforward apart from the need to commit
> suicide, but I've become quite adept at forcing code into that model.
I do that, too, but in theory a thread could start infinite-looping.
I'm just thinking about the possibilities here, as I've never
encountered something this ugly in my short life, but voluntary
event-handling might not always work.
>> Lucid had a notion of `interruptions' (wonderful name, eh?). An
>> `interruption' was a chunk of lisp code that would be run within the
>> context of the receiving thread just before the thread regained
>> control during a process switch. They appear to lisp to be
>> asynchronous, but the OS sees them as synchronous.
>
> This is more in line with my idea of language level signal handling.
Hm, sounds good.
>> When a thread was
>> in the cleanup of an unwind-protect, processing of `interruptions'
>> would be deferred until the cleanup was finished. `Interrupts', that
>> is, the actual clock ticks, control-C's, page-faults, etc., would be
>> handled immediately, but would never directly cause a thread abort.
>> If you `killed' a thread, it would actually push an `interruption'
>> that would cause the thread to kill itself at the next safe point.
>
> And it seems like Lucid had a fairly reasonable and safe way to cause
> termination of a thread. The closest thing I have used is Java's
> exception based thread termination model - which still has a number of
> problems including dependency on the design of classes and exception
> handlers to prevent leaking OS resources.
Don't know about that exception thing... (in case you have a pointer).
Throwing an exception should be rather safe, and free up everything
inside unwind-protect. In Java maybe using "finally" excessively would
work.
--
My mouth says the words, my brain is thinking monstertrucks.
Joey (Friends)
On Fri, 09 Sep 2005 09:44:49 +0200, Ulrich Hobelmann
<···········@web.de> wrote:
>George Neuner wrote:
>> Maybe I've been programming under a rock for 20 years, but I've never
>> come across a situation where I considered forcibly terminating a
>> thread to be the right design. OTOH, my experience has been mostly of
>> lousy runtime support for signaling.
>
>It's not good design, but otherwise, how do you kill a hanging thread?
I don't allow them to hang.
>> When I have a situation where a thread might need to be killed, I
>> design it as event driven from the start. It complicates things a bit
>> with code that would be straightforward apart from the need to commit
>> suicide, but I've become quite adept at forcing code into that model.
>
>I do that, too, but in theory a thread could start infinite-looping.
Only if you let it.
I use asynchronous I/O and a state machine driven by an event handling
loop.
A loop with uncertain limits or that contains an I/O call can be
handled by creating a separate state for the loop body, packaging the
loop variables into the shared thread environment and iterating by
remaining in the same processing state until the loop completes.
Try not to mix processing and I/O in the same state. It's ok if the
I/O is performed last, but problematic otherwise. Better to buffer
the computations and do the I/O separately.
If you make sure that each individual state function terminates, then
no matter what happens the thread will eventually return to the event
loop.
If you aren't used to event-driven programming it takes a little while
to get used to structuring your code this way, but it really isn't
very hard to do.
George
--
for email reply remove "/" from address
George Neuner wrote:
>> Lucid had a notion of `interruptions' (wonderful name, eh?). An
>> `interruption' was a chunk of lisp code that would be run within the
>> context of the receiving thread just before the thread regained
>> control during a process switch. They appear to lisp to be
>> asynchronous, but the OS sees them as synchronous.
>
> This is more in line with my idea of language level signal handling.
LtU just mentioned this (old) paper on async exceptions in Haskell.
Maybe you'll find it interesting (if a bit formal).
http://lambda-the-ultimate.org/node/view/959
--
My mouth says the words, my brain is thinking monstertrucks.
Joey (Friends)
George Neuner <·········@comcast.net> writes:
> On Thu, 08 Sep 2005 10:39:20 -0400, Joe Marshall
> <·········@alum.mit.edu> wrote:
>
>>George Neuner <·········@comcast.net> writes:
>>
>>> I'm not a fan of aborting threads in mid-execution.
>>
>>You generally can't get much work done if your threads are aborted in
>>mid-execution, but sometimes you have to do that and making it work in
>>some sane way is good.
>
> Maybe I've been programming under a rock for 20 years, but I've never
> come across a situation where I considered forcibly terminating a
> thread to be the right design. OTOH, my experience has been mostly of
> lousy runtime support for signaling.
I can't believe you've never typed control-c at a hung process.
And I'm sure that you've written code where you want to let the user
type control-c at some unpredictable point and have something
predictable happen.
If you've written a client/server system, I'm sure that you've thought
about putting timeouts on client or server responses so that at least
one of the processes can recover.
> When I have a situation where a thread might need to be killed, I
> design it as event driven from the start. It complicates things a bit
> with code that would be straightforward apart from the need to commit
> suicide, but I've become quite adept at forcing code into that model.
It's possible to get the interrupt system, multithreading, and
unwind-protect to work in concert so that you don't *have* to force
anything. It's tricky, but the advantage is that you solve the
problem once and for all rather than solving it over and over again
for each new task.
> And it seems like Lucid had a fairly reasonable and safe way to cause
> termination of a thread. The closest thing I have used is Java's
> exception based thread termination model - which still has a number of
> problems including dependency on the design of classes and exception
> handlers to prevent leaking OS resources.
The finally clause in Java doesn't defer asynchronous interrupts (at
least it didn't in the past).
The finally clause in C# *does* defer asynchronous interrupts (at
least it does in version 2 of the .NET runtime). They ran into this
very issue when trying to couple the .NET runtime with the SQL server.
On Fri, 09 Sep 2005 15:20:16 -0400, Joe Marshall
<·········@alum.mit.edu> wrote:
>>George Neuner <·········@comcast.net> writes:
>>
>> Maybe I've been programming under a rock for 20 years, but I've never
>> come across a situation where I considered forcibly terminating a
>> thread to be the right design. OTOH, my experience has been mostly of
>> lousy runtime support for signaling.
>I can't believe you've never typed control-c at a hung process.
I certainly have killed hung processes. Generally it's followed by a
curse upon the author - *especially* if the author was ME.
>And I'm sure that you've written code where you want to let the user
>type control-c at some unpredictable point and have something
>predictable happen.
Generally I want absolutely nothing to happen. Predictably.
>If you've written a client/server system, I'm sure that you've thought
>about putting timeouts on client or server responses so that at least
>one of the processes can recover.
I've spent most of my professional life writing multithread, real-time
and high availability software: industrial monitoring apps - many with
complex GUIs, network and database requirements in addition to their
real time functions. I've also done server side data processing, both
ends of middleware, bare metal embedded apps, and here and there the
odd desktop GUI application.
Many industrial systems are physically protected and use remoted
touchscreen UIs. The workers have no way to abort/restart the
application via the UI and rebooting the computer requires a set of
keys. It can take a long time to recover from a crash that occurs in
the middle of the night [when the supervisor is home in bed] and the
company may be losing a lot of money every minute the system is off
line.
For a lot of reasons I consider it a professional embarrassment to
have my software discovered to be the cause of an system outage.
>It's possible to get the interrupt system, multithreading, and
>unwind-protect to work in concert so that you don't *have* to force
>anything. It's tricky, but the advantage is that you solve the
>problem once and for all rather than solving it over and over again
>for each new task.
Sounds good. After many years of doing it manually, turning
synchronous code into interruptable code has become routine and
formulaic, but it still requires time and mental effort that could be
spent elsewhere.
George
--
for email reply remove "/" from address
From: Kent M Pitman
Subject: Re: async interrupts, without-aborts
Date:
Message-ID: <uzmqkzk6m.fsf@nhplace.com>
George Neuner <·········@comcast.net> writes:
> On Fri, 09 Sep 2005 15:20:16 -0400, Joe Marshall
> <·········@alum.mit.edu> wrote:
>
> >>George Neuner <·········@comcast.net> writes:
> >>
> >> Maybe I've been programming under a rock for 20 years, but I've never
> >> come across a situation where I considered forcibly terminating a
> >> thread to be the right design. OTOH, my experience has been mostly of
> >> lousy runtime support for signaling.
>
> >I can't believe you've never typed control-c at a hung process.
>
> I certainly have killed hung processes. Generally it's followed by a
> curse upon the author - *especially* if the author was ME.
Actually, I think a lot of this is because programs handle interrupts badly
and traditionally you don't know whether hitting c-c will leave data in a
good state.
Having worked a lot with Lisp Machines that had a really reliable condition
handling mechanism, and having known that programmers routinely put error
recovery in place, it became more natural for me to use Control-Abort (the
Lispm had a named Abort key, and Abort was a synchronous character processed
in the reader while Control-Abort generated an asynchronous interrupt at
key-press time, Meta-Abort was synchronous abort to toplevel while
Control-Meta-Abort was asynchronous abort to top-level at key-press time)
because I trusted the designers both to implement programs correctly.
Modern GUI interfaces (as opposed to shell interfaces) seem to protect
themselves by having thin GUIs that if you were to abort would not really
stop interaction, and you have to have a Stop hotbutton, which advertises
itself as "well thought out" by its mere presence. (I'll bet you've
used Stop in web browsers to abort a transfer that was going nowhere,
etc.)
> On Fri, 09 Sep 2005 15:20:16 -0400, Joe Marshall
> <·········@alum.mit.edu> wrote:
>>
>>And I'm sure that you've written code where you want to let the user
>>type control-c at some unpredictable point and have something
>>predictable happen.
George Neuner <·········@comcast.net> writes:
> Generally I want absolutely nothing to happen. Predictably.
That's easy enough. Just disable the signal.
> I've spent most of my professional life writing multithread, real-time
> and high availability software: industrial monitoring apps - many with
> complex GUIs, network and database requirements in addition to their
> real time functions. I've also done server side data processing, both
> ends of middleware, bare metal embedded apps, and here and there the
> odd desktop GUI application.
>
> Many industrial systems are physically protected and use remoted
> touchscreen UIs. The workers have no way to abort/restart the
> application via the UI and rebooting the computer requires a set of
> keys. It can take a long time to recover from a crash that occurs in
> the middle of the night [when the supervisor is home in bed] and the
> company may be losing a lot of money every minute the system is off
> line.
>
> For a lot of reasons I consider it a professional embarrassment to
> have my software discovered to be the cause of an system outage.
I agree. I've done some of the same myself. It seems that our
approaches differ (probably because of the facilities of the
language). To give an example, the change management software I've
been working on has a web-browser interface. I don't want the server
to die in the middle of a complex transaction when the user closes the
browser window. (I just now tried it so I could look at the stack
trace. Normally, the code would automatically recover, but I have it
drop into the error handler during development.)
At the point where I interrupted there are over 100 frames on the
stack. Many of these frames represent pending computations that can
simply be discarded, but a number of them represent critical actions
that *must* take place if the server is not to crash.
Frame 14 has to end the transaction. Since transactions may be
nested, a non-local exit does not necessarily return to `top level'.
The transaction end code has to determine whether to propagate the
changes visible in the nested transaction to the enclosing
transation.
Frame 18 removes persistent data from the repository record.
Persistent data cannot be used outside a transaction, and each
repository has a few persistent objects for bookkeeping that must not
be left in the image when a transaction ends.
Frame 22 closes recently opened repositories. Repository opening is
protected by the transaction, so any repositories opened *during* the
most recent transaction must be closed if the transaction aborts.
Frame 27 `finalizes' the transaction. This ensures that the
transaction is either in the commit or abort state upon exit. (You
can throw with a still-running transaction, but if you attempt to
throw past this frame, the transaction will automatically abort.)
Frame 31 closes the satellite repository. ChangeSafe repositories
can span multiple files and it so happens that this transaction was
running nested in a satellite file.
Frame 38 ends the outer transaction.
Frame 42 removes the persistent data from the master repository
record.
Frame 46 closes repositories opened during the outer transaction.
Frame 51 finalizes the outer transaction.
Frame 55 closes the master repository.
Frame 61 ends the workspace transaction. The workspace database is
separate from the project management database and I aborted in the
middle of a workspace regenerate.
Frame 65 removes the persistent data from the workspace repository
record.
Frame 69 closes repositories opened during the workspace transaction.
Frame 74 finalizes the workspace transaction.
Frame 79 closes the workspace repository.
Frame 84 shuts down the connection with the Java applet on the web
page. This instructs the Java applet to re-direct the browser to the
appropriate `success' or `failure' page.
Frame 87 turns off the timer that aborts the entire transaction if it
doesn't finish in a `reasonable' time. This prevents problems with
database interlocking if a user's process stalls for some reason.
At this point, the user's request is aborted. The thread servicing
the request would be returned to the thread pool and be eligible for
servicing future requests. However, when the server is completely
shutdown, there are a few frames to ensure that the shutdown is
graceful.
Frame 98 evicts the repository cache when the server exits. If the
server is aborted and restarted, the repository cache is reloaded.
Frame 102 closes the http log file.
Frame 109 closes the changesafe server log file.
Note that these actions have to occur in this order if the server is
to survive a simple hangup from the client. I'm not exactly sure how
one would refactor this to a message passing system (one probably
wouldn't, but would rather write it completely differently from the
get-go). The unwind-protects in the frames above ensure that the
cleanup actions occur, and since they defer interrupts during the
cleanup forms (I'm using a modified unwind-protect that guarantees
this), I can be certain that no part of the cleanup can be avoided if
the process continues to run.
On Wed, 14 Sep 2005 11:37:59 -0400, Joe Marshall
<·········@alum.mit.edu> wrote:
>... It seems that our
>approaches differ (probably because of the facilities of the
>language). To give an example, the change management software I've
>been working on has a web-browser interface. I don't want the server
>to die in the middle of a complex transaction when the user closes the
>browser window. (I just now tried it so I could look at the stack
>trace. Normally, the code would automatically recover, but I have it
>drop into the error handler during development.)
>
>At the point where I interrupted there are over 100 frames on the
>stack. Many of these frames represent pending computations that can
>simply be discarded, but a number of them represent critical actions
>that *must* take place if the server is not to crash.
>
<snip interesting long example>
>
>Note that these actions have to occur in this order if the server is
>to survive a simple hangup from the client. I'm not exactly sure how
>one would refactor this to a message passing system (one probably
>wouldn't, but would rather write it completely differently from the
>get-go). The unwind-protects in the frames above ensure that the
>cleanup actions occur, and since they defer interrupts during the
>cleanup forms (I'm using a modified unwind-protect that guarantees
>this), I can be certain that no part of the cleanup can be avoided if
>the process continues to run.
It's been a very long while since I have dealt with transaction
management at the low level. Nowadays I use OTS databases as much as
possible to avoid writing them 8-).
I assume that you are implementing transaction management functions as
part of your application. From a purely design standpoint, my opinion
is that using the frame stack to track transaction states is a mistake
- it's convenient but it can easily lead to complicated error handling
scenarios like the one you described. I would probably have chosen to
implement a state machine transaction manager walking a tree of
transaction state records and used callbacks to implement details of
individual transaction stages.
WRT language facilities driving strategy ... there's definitely truth
to that in a lot of cases. However, I think in this particular case I
probably would have taken the state machine approach regardless of
implementation language.
[Btw: are you also Joe Marshall from NEU? I learned transaction
processing and database implementation from Ken Baclawski].
George
--
for email reply remove "/" from address
George Neuner <·········@comcast.net> writes:
> It's been a very long while since I have dealt with transaction
> management at the low level. Nowadays I use OTS databases as much as
> possible to avoid writing them 8-).
I understand. We *were* using AllegroStore/ObjectStore, but the
application has non-standard database usage patterns. We almost never
delete data, and the database indexes are designed specifically for
version management. This means that a lot of the machinery in a
standard database is unnecessary, inappropriate, or poorly optimized
for our usage. Since we already had to implement versioning in lisp,
all we really needed was a very simple object-oriented persistent
store. (It turned out to need a few more features than I thought at
first, so it supports two-phase commits, multiple database extents,
and nested transactions.)
> I assume that you are implementing transaction management functions as
> part of your application. From a purely design standpoint, my opinion
> is that using the frame stack to track transaction states is a mistake
> - it's convenient but it can easily lead to complicated error handling
> scenarios like the one you described. I would probably have chosen to
> implement a state machine transaction manager walking a tree of
> transaction state records and used callbacks to implement details of
> individual transaction stages.
That's interesting.
I was going to write up something to see how my approach differs from
your approach, but on my way home I realized that the two approaches
are not very different at all. The difference is in how I am
leveraging the language.
Semantically, function calls and message passing are virtually
identical operations (message passing usually implies less
synchronization between sender and receiver than a function call). In
addition, callbacks, function returns, and non-local exits (exception
handling) are all specialized forms of continuations. When modeled in
this way, a state transition becomes a simple function call and a
callback is either a normal return, a call to a thunk, or a throw.
This is probably accounts a lot for the convenience of using the frame
stack to track transaction states.
The cleanup form of an unwind-protect is a piece of code that is
associated with a set of arcs in the transaction state machine. It
ensures that transitions out of certain states, no matter which arc is
taken, causes certain code to run. The advantage of relying on
unwind-protect is that I can take fragments of code that were not
designed to interact with the transaction manager (like, for example,
the web server) and use them within the transaction. Error handling
within the web server would add a fair amount of complexity to the
transaction manager; a number of states that had nothing to do with
transactions but nonetheless were part of the control flow would have
to be added to the transaction state machine. For example, a
recoverable error within the web page generation would not necessarily
need to abort the transaction, but a non-recoverable one obviously
would. By using unwind-protect, I can guarantee that all relevant
state transitions are covered even if I don't know what all of them
are.
Incidentally, the reason I'm so insistent that my Lisp has tail call
optimization is precisely this: to fully model message passing with
function calls you must allow for an indefinite level of message
delegation. If calls in tail position always push a stack frame, then
message delegation is limited by stack depth. You either have to
statically determine that the maximum delegation chain is short enough
or you have to rewrite your code to explicitly manage message
passing as heap objects and manually dispatch the messages.
> WRT language facilities driving strategy ... there's definitely truth
> to that in a lot of cases. However, I think in this particular case I
> probably would have taken the state machine approach regardless of
> implementation language.
Have you tried writing a database in Lisp (especially with the
tail-call optimization turned on)?
> [Btw: are you also Joe Marshall from NEU? I learned transaction
> processing and database implementation from Ken Baclawski].
Yes (although my NEU contract has just expired). Ken Baclawski's
office was just down the hall from me. We were never formally
introduced (I was in the PLT group), and we didn't really chat.
~jrm
Joe Marshall <·········@alum.mit.edu> writes:
> Incidentally, the reason I'm so insistent that my Lisp has tail call
> optimization is precisely this: to fully model message passing with
> function calls you must allow for an indefinite level of message
> delegation. If calls in tail position always push a stack frame,
> then message delegation is limited by stack depth. You either have
> to statically determine that the maximum delegation chain is short
> enough or you have to rewrite your code to explicitly manage message
> passing as heap objects and manually dispatch the messages.
Isn't this a case of abstraction inversion? It seems to me that
procedure calls is a "higher" abstraction than message passing (it
additionally provides synchronization and return values), and
therefore it'd be a mistake to model the latter in the former.
Let me play devil's advocate against tail-calls some more: Don't you
also have the problem above that you somehow (and "manually") have to
know that certain calls can in fact be tail-calls, and isn't it the
case that this can be non-trivial wrt. dynamic bindings etc?
My own gut feeling is that tail calls should be provided as a separate
operation, not as an implicit "optimization" of procedure calls. They
seem to me to be two fundamentally very different operations.
--
Frode Vatvedt Fjeld
Frode Vatvedt Fjeld <······@cs.uit.no> writes:
> Joe Marshall <·········@alum.mit.edu> writes:
>
>> Incidentally, the reason I'm so insistent that my Lisp has tail call
>> optimization is precisely this: to fully model message passing with
>> function calls you must allow for an indefinite level of message
>> delegation. If calls in tail position always push a stack frame,
>> then message delegation is limited by stack depth. You either have
>> to statically determine that the maximum delegation chain is short
>> enough or you have to rewrite your code to explicitly manage message
>> passing as heap objects and manually dispatch the messages.
>
> Isn't this a case of abstraction inversion? It seems to me that
> procedure calls is a "higher" abstraction than message passing (it
> additionally provides synchronization and return values), and
> therefore it'd be a mistake to model the latter in the former.
Messages are synchronized at the point at which they are sent
(obviously!), and returning a value is simply sending that value to
the continuation, which itself was sent as an implicit parameter.
> Let me play devil's advocate against tail-calls some more: Don't you
> also have the problem above that you somehow (and "manually") have to
> know that certain calls can in fact be tail-calls,
It depends on what you mean. In general I just count on the compiler
dealing with the problem for me. It will optimize the tail calls it
finds (provided the optimization level is set correctly). I don't
bother manually analysing each call to ensure that it is a tail call,
I just write the code.
It's usually rather easy to determine which calls are tail calls: if
the result of a call isn't being used as the argument to another call,
then it is a tail call. (This is a general rule and there are a lot
of exceptions, but it is a good first-order approximation.)
> and isn't it the case that this can be non-trivial wrt. dynamic
> bindings etc?
Yes, but I don't often run into cases where I need to establish a
dynamic binding that cuts through a loop in the control flow (this
would cause the variable to be rebound at each iteration leading to
stack overflow).
> My own gut feeling is that tail calls should be provided as a separate
> operation, not as an implicit "optimization" of procedure calls. They
> seem to me to be two fundamentally very different operations.
Steele argues that function calls are `gotos that pass arguments'. A
call in tail position is either passing the continuation of the caller
or an eta-expansion of that same continuation. Tail call optimization
simply avoids the eta-expansion. The rest of the function call, tail
or not, simply pushes arguments and jumps.
~jrm
On Thu, 15 Sep 2005 23:15:47 -0400, Joe Marshall
<·········@alum.mit.edu> wrote:
>George Neuner <·········@comcast.net> writes:
>
>> From a purely design standpoint, my opinion
>> is that using the frame stack to track transaction states is a mistake
>> - it's convenient but it can easily lead to complicated error handling
>> scenarios like the one you described. I would probably have chosen to
>> implement a state machine transaction manager walking a tree of
>> transaction state records and used callbacks to implement details of
>> individual transaction stages.
>
>That's interesting.
>
>I was going to write up something to see how my approach differs from
>your approach, but on my way home I realized that the two approaches
>are not very different at all. The difference is in how I am
>leveraging the language.
>
>Semantically, function calls and message passing are virtually
>identical operations (message passing usually implies less
>synchronization between sender and receiver than a function call). In
>addition, callbacks, function returns, and non-local exits (exception
>handling) are all specialized forms of continuations. When modeled in
>this way, a state transition becomes a simple function call and a
>callback is either a normal return, a call to a thunk, or a throw.
>This is probably accounts a lot for the convenience of using the frame
>stack to track transaction states.
Operationally, though, they are not the same.
When state is stored in the frame stack, the thread is committed to a
particular transaction for the duration. A thread per transaction,
even with parallel sub-transactions, isn't generally a problem when
the work is all done locally ... but remote sub-transactions can
quickly tie up a hierarchy of threads for indefinite periods.
Using explicit state allows thread pools to service pending operations
and avoids unnecessarily tying up resources.
Of course, this may not be an issue for any particular application.
>The advantage of relying on
>unwind-protect is that I can take fragments of code that were not
>designed to interact with the transaction manager (like, for example,
>the web server) and use them within the transaction.
I really don't see how an explicit state design would prevent using
foreign code services - processing for each transaction state
completes before advancing to the next state. Using a less than
cooperative service might impact performance, but should have no
effect otherwise.
>Have you tried writing a database in Lisp (especially with the
>tail-call optimization turned on)?
No. My interest in Lisp is mostly academic. I occasionally use
Scheme for personal hacking, but I've never used it for work and never
had any reason to learn CL.
And I've never tried to write a database engine in Scheme.
George
George Neuner <········@comcast.net> writes:
> On Thu, 15 Sep 2005 23:15:47 -0400, Joe Marshall
> <·········@alum.mit.edu> wrote:
>>
>>Semantically, function calls and message passing are virtually
>>identical operations (message passing usually implies less
>>synchronization between sender and receiver than a function call). In
>>addition, callbacks, function returns, and non-local exits (exception
>>handling) are all specialized forms of continuations. When modeled in
>>this way, a state transition becomes a simple function call and a
>>callback is either a normal return, a call to a thunk, or a throw.
>>This is probably accounts a lot for the convenience of using the frame
>>stack to track transaction states.
>
> Operationally, though, they are not the same.
>
> When state is stored in the frame stack, the thread is committed to a
> particular transaction for the duration. A thread per transaction,
> even with parallel sub-transactions, isn't generally a problem when
> the work is all done locally ... but remote sub-transactions can
> quickly tie up a hierarchy of threads for indefinite periods.
>
> Using explicit state allows thread pools to service pending operations
> and avoids unnecessarily tying up resources.
>
> Of course, this may not be an issue for any particular application.
In my particular case, I'm doing the operations locally.
In any case, though, you have to represent the continuation in some
manner. A thread is reasonable if it is lightweight (and portable),
but it would be a problem if transactions were frequent, long, and if
threads were heavyweight.
>>The advantage of relying on
>>unwind-protect is that I can take fragments of code that were not
>>designed to interact with the transaction manager (like, for example,
>>the web server) and use them within the transaction.
>
> I really don't see how an explicit state design would prevent using
> foreign code services - processing for each transaction state
> completes before advancing to the next state. Using a less than
> cooperative service might impact performance, but should have no
> effect otherwise.
I was just noticing that my approach was closer to a state design than
I had at first thought.
>>Have you tried writing a database in Lisp (especially with the
>>tail-call optimization turned on)?
>
> No. My interest in Lisp is mostly academic. I occasionally use
> Scheme for personal hacking, but I've never used it for work and never
> had any reason to learn CL.
>
> And I've never tried to write a database engine in Scheme.
You should try it (that is, if it is of interest to you). I learned
a lot from it.
Hello,
George Neuner <·········@comcast.net> writes:
> On 4 Sep 2005 13:48:55 -0700, ····@hotpop.com wrote:
>
>>Let's examine some potential pitfalls first:
>>
>>(let (x)
>> (unwind-protect
>> (progn
>> (setq x (open ...))
>> ...)
>> (when x (close x))))
>>
>>- If an async interrupt hits and aborts when open has finished but
>>before its return value is assigned to X, then a file descriptor is
>>leaked.
>>
>>- In cl implementations that don't do anything special for
>>unwind-protect if an async interrupt hits and aborts in the cleanup
>>form then X can remain open.
>>
>>A potential solution is to wrap (setq x (open ...)) in
>>without-interrupts (the variety that inhibits interrupts (C-c,
>>process-interrupt, SIGALRM) to the running process, but does not
>>prevent other processes/threads from running). Unfortunately, if
>>open takes a long time then among other things it cannot be C-c-ed
>>and debugged. Similarly, automatically inhibiting interrupts in
>>cleanup forms is too strong.
>
> I haven't used or worked with CL much so please be gentle as I may be
> completely misinterpreting something. I have done both OS and
> language runtime support implementation - just not with Lisp.
>
> Is something like the example above *really* a problem? Perhaps I am
> getting confused by terminology ... are you really talking about
> "interrupts" or are you referring to conditions/signals/exceptions?
[deletia]
IIUC, G�bor is concerned with asynchronous interrupts; those are not
caused by the currently executing code and thus can happen "anytime".
For example, there is a small window between OPEN returning and X
being set during which the user could hit some kind of <abort> key.
Such windows are an impediment to reliable systems even if they are
only a handful of instructions wide.
> George
Cheers,
Damien.
--
http://foobox.net/~dash/
I can resist everything except temptation.
--Oscar Wilde
····@hotpop.com writes:
> Reading the old usenet posts about unwind-protect and async interrupts
> revealed a great deal of confusion about how things should behave.
I have a clear proposal how it should behave, but I haven't tried to
port it to Lisp, and I don't know what existing Lisp implementations
do. My proposal in short:
There is a per-thread state of asynchronous signals: enabled,
disabled, or "synchronous mode". Actually the state also counts the
number of times signals were implicitly blocked, but these are the
three outcomes.
If signals are enabled, they may be handled at an arbitrary time.
If they are in synchronous mode, they may be allowed by certain
primitives which block the current thread indefinitely, but only
during the actual wait.
Signals are automatically blocked by certain language constructs,
e.g. by locking a mutex.
Unwind-protect is insufficient, there must be a construct with 3 parts:
acquire, main body, release. Signals are turned into synchronous mode
during the acquire part and they are blocked during the release part.
I'm not familiar enough with Common Lisp conditions to see how they
would interact with this. In my model exceptions are always aborting,
but signals are also used synchronously for what Common Lisp uses
non-aborting conditions. A thread can signal itself in a synchronous
way: the blocking state is irrelevant and the return value of the
handler is not ignored but returned.
--
__("< Marcin Kowalczyk
\__/ ······@knm.org.pl
^^ http://qrnik.knm.org.pl/~qrczak/