In the proficient CL code I frequently see something like the follwoing
(defmacro with-env (&body body)
`(invoke-with-env #'(lambda () ,@body)))
(defun invoke-with-env (continuation)
(declare (dynamic-extent continuation))
(let ((*print-right-margin* 52))
(funcall continuation)))
This is a simplified sketch for a commonly enough used idiom. The
question is about the declaration. Is it possible to see whether it
have any effect? I tried to disassemble the compiled invoke-with-env
with and without the declaration and saw no difference. Playing with
the compiler settings like
(proclaim '(optimize (speed 3) (safety 1) (space 0) (debug 0)))
also gave me nothing.
[Using ACL70]
--
Vladimir Zolotykh
Vladimir Zolotykh wrote:
> In the proficient CL code I frequently see something like the follwoing
>
> (defmacro with-env (&body body)
> `(invoke-with-env #'(lambda () ,@body)))
>
> (defun invoke-with-env (continuation)
> (declare (dynamic-extent continuation))
> (let ((*print-right-margin* 52))
> (funcall continuation)))
The trouble is that the function doesn't control the allocation of the
closure. So about the only thing the DYNAMIC-EXTENT can do here is to
make the CONTINUATION variable have dynamic extent, not the object it
holds. But that can be figured out without the declaration: the
compiler simply has to see that there are no closures being created in
this body; nowhere is the CONTINUATION variable subject to capture.
Hence it can be stack-allocated. As far as the object it refers to,
that is a done deal already.
This is a case of optimizing too far down the chain.
I suspect that DYNAMIC-EXTENT declarations can only have the desired
effect if they are placed at the site where the object is constructed.
Only at the point of construction can a storage allocation strategy be
chosen for an object.
So maybe you want:
(defmacro with-env (&body body)
`(let ((temp-closure #'(lambda () ,@body)))
(declare (dynamic-extent temp-closure))
(invoke-with-env temp-closure)))
Now maybe the compiler can stack-allocate the entire closure. You are
essentially saying: ``I promise that the closure won't escape from
here; I am only passing it down into INVOKE-WITH-ENV, which I know will
not try to stash a copy of the closure anywhere. Thus I know I only
need downward funargs and everything can be stacked.''
From: Joe Marshall
Subject: Re: To see (declare (dynamic-extent continuation)) working
Date:
Message-ID: <mznzrok1.fsf@ccs.neu.edu>
"Kaz Kylheku" <········@gmail.com> writes:
> So maybe you want:
>
> (defmacro with-env (&body body)
> `(let ((temp-closure #'(lambda () ,@body)))
> (declare (dynamic-extent temp-closure))
> (invoke-with-env temp-closure)))
>
> Now maybe the compiler can stack-allocate the entire closure. You are
> essentially saying: ``I promise that the closure won't escape from
> here; I am only passing it down into INVOKE-WITH-ENV, which I know will
> not try to stash a copy of the closure anywhere. Thus I know I only
> need downward funargs and everything can be stacked.''
Unfortunately, you cannot make that guarantee because you have no
knowledge (in general) of the contents of BODY. If BODY closes over
anything, then only the closure object itself has dynamic extent. I
suppose it is an optimization, but not much of one.
In article <············@ccs.neu.edu>, Joe Marshall <···@ccs.neu.edu>
wrote:
> "Kaz Kylheku" <········@gmail.com> writes:
>
> > So maybe you want:
> >
> > (defmacro with-env (&body body)
> > `(let ((temp-closure #'(lambda () ,@body)))
> > (declare (dynamic-extent temp-closure))
> > (invoke-with-env temp-closure)))
> >
> > Now maybe the compiler can stack-allocate the entire closure. You are
> > essentially saying: ``I promise that the closure won't escape from
> > here; I am only passing it down into INVOKE-WITH-ENV, which I know will
> > not try to stash a copy of the closure anywhere. Thus I know I only
> > need downward funargs and everything can be stacked.''
>
> Unfortunately, you cannot make that guarantee because you have no
> knowledge (in general) of the contents of BODY. If BODY closes over
> anything, then only the closure object itself has dynamic extent. I
> suppose it is an optimization, but not much of one.
I think that may be the crux of the OP's confusion, or perhaps relate to
his hoped-for optimization. What's really needed is a way to add a
declaration about INVOKE-WITH-ENV that *it* handles its argument in a
dynamic extentional manner. This information could then be used by the
caller to determine whether it can safely stack-allocate the closure.
--
Barry Margolin, ······@alum.mit.edu
Arlington, MA
*** PLEASE post questions in newsgroups, not directly to me ***
Barry Margolin wrote:
> (defmacro with-env (&body body)
> `(let ((temp-closure #'(lambda () ,@body)))
> (declare (dynamic-extent temp-closure))
> (invoke-with-env temp-closure)))
> What's really needed is a way to add a
> declaration about INVOKE-WITH-ENV that *it* handles its argument in a
> dynamic extentional manner. This information could then be used by the
> caller to determine whether it can safely stack-allocate the closure.
There is, although your favorite implementation might not take advantage
of thisinformation.
If invoke-with-env is a function, it could be defined
(defun invoke-with-env (f)
(declare (dynamic-extent f))
...)
The declaration on the argumant to invoke-with-env can be remembered by
a sufficiently-powerful compiler to optimize calls to the function.
Even if the result of a computation is assigned to a variable that has
not been declared dynamic-extent, if all uses of that variable are in
context that declare only dynamic extent use, the implenmentation can
infer that the result of the computation can be consed with dynamic
extent. The Allegro compiler does this in some circumstances. In
particular, many of the functional arguments to the standard CL
operators are known by the compiler not to be preserve beyond the
dynamic extent of the call. Examples are the functional arguments to
the various map* functions, and the various :test and :key arguments to
sequence functions.
If invoke-with-env is a macro, then the use of the argument subform in
the expansion can be decorated with dynamic-extent declarations as I
believe has been explained elsewhere in this thread.
From: Christophe Rhodes
Subject: Re: To see (declare (dynamic-extent continuation)) working
Date:
Message-ID: <sqbr4a8bug.fsf@cam.ac.uk>
"Steven M. Haflich" <···@alum.mit.edu> writes:
> Barry Margolin wrote:
>> What's really needed is a way to add a declaration about
>> INVOKE-WITH-ENV that *it* handles its argument in a dynamic
>> extentional manner. This information could then be used by the
>> caller to determine whether it can safely stack-allocate the closure.
>
> There is, although your favorite implementation might not take advantage
> of thisinformation.
>
> If invoke-with-env is a function, it could be defined
>
> (defun invoke-with-env (f)
> (declare (dynamic-extent f))
> ...)
>
> The declaration on the argumant to invoke-with-env can be remembered by
> a sufficiently-powerful compiler to optimize calls to the function.
> [...] The Allegro compiler does this in some circumstances. In
> particular, many of the functional arguments to the standard CL
> operators are known by the compiler not to be preserve beyond the
> dynamic extent of the call.
Well, yes, but the standard CL operators are protected from
redefinition. Does the Allegro compiler do this under any
circumstances for user functions of the form above?
Christophe
From: Duane Rettig
Subject: Re: To see (declare (dynamic-extent continuation)) working
Date:
Message-ID: <4oe88x4lu.fsf@franz.com>
Christophe Rhodes <·····@cam.ac.uk> writes:
> "Steven M. Haflich" <···@alum.mit.edu> writes:
>
> > If invoke-with-env is a function, it could be defined
> >
> > (defun invoke-with-env (f)
> > (declare (dynamic-extent f))
> > ...)
> >
> > The declaration on the argumant to invoke-with-env can be remembered by
> > a sufficiently-powerful compiler to optimize calls to the function.
> > [...] The Allegro compiler does this in some circumstances. In
> > particular, many of the functional arguments to the standard CL
> > operators are known by the compiler not to be preserve beyond the
> > dynamic extent of the call.
>
> Well, yes, but the standard CL operators are protected from
> redefinition.
"Protected" is such a harsh word - the openness of CL discourages
that - instead we try to inform so that programmers can avoid
unintended mistakes, but given enough persistence (and understanding of
what could go wrong) one could easily do such a thing:
CL-USER(1): (defun car (x) (foo x))
Error: Attempt to make a FUNCTION definition for the name CAR. This name is
in the COMMON-LISP package and redefining it is a violation for
portable programs. Replacing the current definition of #<Function CAR>
may be dangerous. The package COMMON-LISP has PACKAGE-DEFINITION-LOCK
set, which causes the system to signal this violation.
[condition type: PACKAGE-LOCKED-ERROR]
Restart actions (select using :continue):
0: Set the FUNCTION definition of the name CAR anyway.
1: Return to Top Level (an "abort" restart).
2: Abort entirely from this (lisp) process.
[1c] CL-USER(2): :cont 0
CAR
CL-USER(3):
> Does the Allegro compiler do this under any
> circumstances for user functions of the form above?
I take it that "this" means "protect". In some sense, yes,
we "inform" the programmer of a potential problem (note that
I have edited the file in the middle of the transcript to
demonstrate a sample change of function):
CL-USER(1): (shell "cat foo.cl")
(defun foo (f)
(declare (dynamic-extent f))
(funcall f))
0
CL-USER(2): :cf foo
;;; Compiling file foo.cl
;;; Writing fasl file foo.fasl
;;; Fasl write complete
CL-USER(3): :ld foo
; Fast loading /tmp_mnt/net/gemini/home/duane/foo.fasl
CL-USER(4): (shell "cat foo.cl")
(defun foo (f)
;; (declare (dynamic-extent f))
(funcall f))
0
CL-USER(5): :cf foo
;;; Compiling file foo.cl
; While compiling FOO:
Warning: The dynamic-extent arguments declared for function FOO have changed.
The first positional argument had dynamic extent in the former
definition but not in the new, and any previously-compiled calls may
have assumed dynamic extent.
;;; Writing fasl file foo.fasl
;;; Fasl write complete
CL-USER(6):
--
Duane Rettig ·····@franz.com Franz Inc. http://www.franz.com/
555 12th St., Suite 1450 http://www.555citycenter.com/
Oakland, Ca. 94607 Phone: (510) 452-2000; Fax: (510) 452-0182
From: Christophe Rhodes
Subject: Re: To see (declare (dynamic-extent continuation)) working
Date:
Message-ID: <sqwtmwvoxr.fsf@cam.ac.uk>
Duane Rettig <·····@franz.com> writes:
> Christophe Rhodes <·····@cam.ac.uk> writes:
>
>> Does the Allegro compiler do this under any
>> circumstances for user functions of the form above?
>
> I take it that "this" means "protect".
I actually meant "perform dynamic-extent allocation at call sites for
functions with local dynamic-extent declarations for certain of their
arguments", but you answered that, too. Thanks :-)
Christophe
Christophe Rhodes wrote:
>>The declaration on the argumant to invoke-with-env can be remembered by
>>a sufficiently-powerful compiler to optimize calls to the function.
>>[...] The Allegro compiler does this in some circumstances. In
>>particular, many of the functional arguments to the standard CL
>>operators are known by the compiler not to be preserve beyond the
>>dynamic extent of the call.
>
> Well, yes, but the standard CL operators are protected from
> redefinition.
This has nothing to do with user redefinition of the standard operators.
In any sane implementation the first functional argument to mapcar,
for example, will be used with only with dynamic-extent. (Other use is
not prohibited, such a mapcar that provides caching, but I doubt there
are any such implementations.) If the cmopiler in such an
implementation doesn't recognize this property of mapcar, then it is
missing a possible optimization.
> Does the Allegro compiler do this under any
> circumstances for user functions of the form above?
I believe the compiler always remembers d-e declarations on defun
arguments. I believe in a calling function it uses this information
only for inferring d-e-ness of closure usage, but not for inferring
autiomatically the possible d-e consing of other kinds of data (e.g.
lists and arrays). There is no fundamental reason for this irregularity
except that circumstances in real code are rare where it would be useful
to know that the list argument(s) to mapcar are also used only with
dynamic-extent. Situations where it is useful to know about us of the
functional argument are common.
I suppose it would be possible to propagate d-e argument usage
transitively. For example: A calls B and B calls C. A conses a datum
that it passes to B, and B passes it to C. If C is known to use that
argument only with d-e, and B doesn't otherwise use it in a non-d-e way,
then the compiler could infer that A could cons the datum with d-e. But
the compiler doesn't do this.
As with any optimization, the programmer has to study the circumstances
under which an optimization will and won't be triggered. (The
intelligent programmer concerns himself only in the rare code where the
difference matters, of course.) Whether a d-e declaration or inference
is trusted and therefore has any effect depends on local optimize
declarations, order of definition, and perhaps other context.
From: Christophe Rhodes
Subject: Re: To see (declare (dynamic-extent continuation)) working
Date:
Message-ID: <sqfytkv6dx.fsf@cam.ac.uk>
"Steven M. Haflich" <···@alum.mit.edu> writes:
> Christophe Rhodes wrote:
>
>>> The declaration on the argumant to invoke-with-env can be
>>> remembered by a sufficiently-powerful compiler to optimize calls to
>>> the function. [...] The Allegro compiler does this in some
>>> circumstances. In particular, many of the functional arguments to
>>> the standard CL operators are known by the compiler not to be
>>> preserve beyond the dynamic extent of the call.
>> Well, yes, but the standard CL operators are protected from
>> redefinition.
>
> This has nothing to do with user redefinition of the standard operators.
I believe that on the contrary, conforming implementations of this
optimization has something to do with the undefined consequences of
redefinition of standard operators.
Consider the three files
1.lisp
(defun invoke/dx (continuation)
(declare (dynamic-extent continuation))
(funcall continuation 2))
2.lisp
(defun invokation (x)
(flet ((frob (y) (incf x y)))
(invoke/dx #'frob)))
3.lisp
(defvar *foo*)
(defun invoke/dx (continuation)
(setq *foo* continuation)
(funcall continuation 2))
I believe that a conforming Common Lisp implementation, on compiling
and loading those three files in sequence, has no mandate to generate
undefined behaviour on calling, say, (invokation 5). It is for this
reason that the prohibition of user redefinition of the standard
operators is relevant: compilers can freely perform the optimization
that you describe on arguments of those operators, because the
consequences of their redefinition are undefined.
>> Does the Allegro compiler do this under any
>> circumstances for user functions of the form above?
>
> I believe the compiler always remembers d-e declarations on defun
> arguments.
Thanks.
Christophe
From: Paul Dietz
Subject: Re: To see (declare (dynamic-extent continuation)) working
Date:
Message-ID: <dcr893$rvq$1@avnika.corp.mot.com>
Kaz Kylheku wrote:
> So maybe you want:
>
> (defmacro with-env (&body body)
> `(let ((temp-closure #'(lambda () ,@body)))
> (declare (dynamic-extent temp-closure))
> (invoke-with-env temp-closure)))
Or:
(defmacro with-env (&body body)
(flet ((temp-closure () ,@body))
(declare (dynamic-extent (function temp-closure)))
(invoke-with-env #'temp-closure)))
(There are lisps in which this really will allocate the
closure on the stack; I don't know if the same would happen
with the LET version.)
Paul
From: Kent M Pitman
Subject: Re: To see (declare (dynamic-extent continuation)) working
Date:
Message-ID: <ubr4esk64.fsf@nhplace.com>
Paul Dietz <············@motorola.com> writes:
> Kaz Kylheku wrote:
>
> > So maybe you want:
> > (defmacro with-env (&body body)
> > `(let ((temp-closure #'(lambda () ,@body)))
> > (declare (dynamic-extent temp-closure))
> > (invoke-with-env temp-closure)))
>
> Or:
>
> (defmacro with-env (&body body)
> (flet ((temp-closure () ,@body))
> (declare (dynamic-extent (function temp-closure)))
or
(declare (dynamic-extent #'temp-closure))
which is equivalent but emphasizes the use of #'temp-closure is the
same here as in the call.
> (invoke-with-env #'temp-closure))
alternatively if you're going to write the declaration as (function
temp-closure), i suggest writing the call that way, too:
(invoke-with-env (function temp-closure))
> )
>
> (There are lisps in which this really will allocate the
> closure on the stack; I don't know if the same would happen
> with the LET version.)
I don't either. It would be nice if it did in order to promote
what are essentially equivalent stylistic choices. I also promote
the FLET form, though, because the practical effect of the LAMBDA
having a name-for-debugging in many implementations is also worth
its weight in gold in some cases where you've calld the function at
a distance from where it was created.
From: Paul F. Dietz
Subject: Re: To see (declare (dynamic-extent continuation)) working
Date:
Message-ID: <sL6dnTqVbYYr6GzfRVn-oA@dls.net>
Kent M Pitman wrote:
> I don't either. It would be nice if it did in order to promote
> what are essentially equivalent stylistic choices. I also promote
> the FLET form, though, because the practical effect of the LAMBDA
> having a name-for-debugging in many implementations is also worth
> its weight in gold in some cases where you've calld the function at
> a distance from where it was created.
I've introduced FLETs in macros just so that control flow
in the expanded macro becomes visible in backtraces and
to profilers. The FLET-ed function is declared notinline
so it isn't optimized away.
Paul
"Vladimir Zolotykh" <······@eurocom.od.ua> writes:
> In the proficient CL code I frequently see something like the follwoing
>
> (defmacro with-env (&body body)
> `(invoke-with-env #'(lambda () ,@body)))
>
> (defun invoke-with-env (continuation)
> (declare (dynamic-extent continuation))
> (let ((*print-right-margin* 52))
> (funcall continuation)))
>
> This is a simplified sketch for a commonly enough used idiom. The
> question is about the declaration. Is it possible to see whether it
> have any effect?
It is implementation dependent as to whether the declaration has any
effect. The intent is that is a way for the programmer to communicate
to the compiler that the object involved can be safely allocated on
the stack (typical means of providing dynamic extent) even though the
object would usually be given indefinite extent (typically via the
heap).
In this case, the object is a function (closure) which "clearly" won't
be accessed anywhere except at the specific call point of
invoke-with-env which results from the expansion of the macro whose
body provides the specific function's body.
/Jon
--
'j' - a n t h o n y at romeo/charley/november com
From: Joe Marshall
Subject: Re: To see (declare (dynamic-extent continuation)) working
Date:
Message-ID: <r7dbrqwh.fsf@ccs.neu.edu>
"Vladimir Zolotykh" <······@eurocom.od.ua> writes:
> In the proficient CL code I frequently see something like the follwoing
>
> (defmacro with-env (&body body)
> `(invoke-with-env #'(lambda () ,@body)))
>
> (defun invoke-with-env (continuation)
> (declare (dynamic-extent continuation))
> (let ((*print-right-margin* 52))
> (funcall continuation)))
>
> This is a simplified sketch for a commonly enough used idiom. The
> question is about the declaration. Is it possible to see whether it
> have any effect? I tried to disassemble the compiled invoke-with-env
> with and without the declaration and saw no difference. Playing with
> the compiler settings like
>
> (proclaim '(optimize (speed 3) (safety 1) (space 0) (debug 0)))
>
> also gave me nothing.
I can't see how the dynamic-extent declaration could reliably work for
downward funargs. The closure has to be created at the call site, and
if `invoke-with-env' were to be redefined to no longer declare the
continuation to be `dynamic-extent', any special action taken at the
call site would have to be re-done.
On the other hand, what difference does it make? I have a hard time
believing that you are calling this function at such a ferocious rate
that GC is an issue.
From: Christophe Rhodes
Subject: Re: To see (declare (dynamic-extent continuation)) working
Date:
Message-ID: <sqfytrnjte.fsf@cam.ac.uk>
"Vladimir Zolotykh" <······@eurocom.od.ua> writes:
> In the proficient CL code I frequently see something like the follwoing
>
> (defmacro with-env (&body body)
> `(invoke-with-env #'(lambda () ,@body)))
>
> (defun invoke-with-env (continuation)
> (declare (dynamic-extent continuation))
> (let ((*print-right-margin* 52))
> (funcall continuation)))
>
> This is a simplified sketch for a commonly enough used idiom. The
> question is about the declaration. Is it possible to see whether it
> have any effect?
The easiest way to see if it has had any effect is to define
INVOKE-WITH-ENV as
(defun invoke-with-env (continuation)
(declare (dynamic-extent continuation))
continuation)
Then if calling WITH-ENV either gives you a helpful error message or
crashes your lisp, the declaration had an effect. :-)
That said, I would not expect Common Lisp compilers to do anything
with that declaration, except possibly for uses of INVOKE-WITH-ENV in
the same file as the definition. What is more likely to work is
(defmacro with-env (&body body)
`(flet ((continuation () ,@body))
(declare (dynamic-extent #'continuation))
(invoke-with-env #'continuation)))
or something of that form, where the binding and the dynamic-extent
declaration are at the same scope. (There /are/ times when a bound
closure can be inferred to have dynamic extent without a declaration,
but this isn't one of them.)
Christophe
[ side note:
(defmacro with-env (&body body)
(let ((continuation (lambda () ,@body)))
(declare (dynamic-extent continuation))
(invoke-with-env continuation)))
which you might expect to have the same properties, in at least one
implementation does not allocation the closure on the stack. ]