Is there a way for running code to find out, based on a symbol (not
known at compile time), what the current lexical value for that symbol
is? How about for a macro to find out what lexical bindings exist in
an &environment variable, like the proposed MAP-ENVIRONMENT?
That's the short question. If you want to know why I'm asking, read
on! This request comes from me being fed up with the current
source-level debugging facilities I find in both current open-source
and commercial Lisp implementations.
Anybody who's done much Emacs programming probably knows how useful
edebug is. It's a source-level Emacs Lisp debugger that allows
stepping, evaluation, breakpoints, and other useful features. (When I
say "debugger", I mean something like what C people think of as a
debugger, which is more like a stepper in Lisp.) I generally find
CL's STEP pretty much useless (and X3J13 pretty much said as much, I
think), so I decided to write my own edebug-like facility for Common
Lisp. The goal is something similar to Jabberwocky's debug
facilities, but with a loose coupling between the Lisp and the editor
so that any extensible editor (for me, Emacs) can be used. But since
Emacs uses dynamic bindings instead of lexical, there's a big
difference in implementation strategy between edebug and my program
(tentatively called "Easel").
Obviously it'll work by instrumenting the user forms. Obviously,
it'll need a codewalker. I've got some ideas on writing a more
general codewalker than most, that can determine (sometimes) what's
user code and what isn't in macro expansions, but I'm getting ahead of
myself.
The big problem I'm having is evaluation of user-supplied forms.
Since EVAL always works in a null lexical environment, I can't
evaluate forms supplied by the user at debug time in the appropriate
lexical environment. This means that the user may have problems
finding variables' values, which is pretty much what you want to do in
a stepper.
So last night, I was thinking about this, and decided I can pass to my
breaking function all the information it needs about the current
lexical environment, and functions to modify it, based on a
macroexpansion. This basically lets me deal with the problem of
accessing lexical variables. I thought of using a macro that would
expand stuff more or less as follows:
(easel:wrap
(let ((x 3))
(foo x)))
==>
(easel:break 0
#'(lambda ()
(let ((x (easel:break 1
#'(lambda () 3)
`((x ,x ,#'(lambda (newval) (setq x newval)))))))
(easel:break 2
#'(lambda ()
(foo (easel:break 3
#'(lambda () x)
`((x ,x ,#'(lambda (newval) (setq x newval)))))))
`((x ,x ,#'(lambda (newval) (setq x newval)))))))
())
easel:break is a function that takes an identifier (so it knows where
in the source code things are), the code body to execute, and a list
of lexical variables. (I've handwaved lexical functions and lexical
macros; they're the same problem and will be solved the same way.)
That list is an alist of the form (symbol current-value setter). When
it's called, it informs the user of where it is in the source, and
waits for the user input. In the simple case-- the user steps the
form-- then it just evaluates its closure argument, displays the
result to the user, and returns that result.
If the user wants to evaluate something, say "(bar x)", then
I actually pass the following to eval:
(let ((x 3))
(bar x)
(funcall #'(lambda (newval) (setq x newval)) x))
where the "3" and the function are actually taken from the parameters
to easel:break. In other words (and this bit is important), the X
in the lambda is closed over the X binding in the original code. This
way, the user can do a (setq x 69) and it will do what they expect.
The big bugaboo in this is knowing what lexical variables exist during
the macroexpansion. I've got this great &environment variable, but
it's opaque, and there's no functions to examine it. From what I can
tell from CLtL2 and the HyperSpec, X3J13 decided to put some of the
environment introspection functions in (although not the ones I'm
looking for here), but later changed their minds.
Anybody have any thoughts?
Thanks,
joelh
--
Joel Ray Holveck - ·····@piquan.org
Fourth law of programming:
Anything that can go wrong wi
sendmail: segmentation violation - core dumped
From: Kent M Pitman
Subject: Re: Lexical environment access at runtime
Date:
Message-ID: <wkr80kfyye.fsf@nhplace.com>
Joel Ray Holveck <·····@piquan.org> writes:
> Is there a way for running code to find out, based on a symbol (not
> known at compile time), what the current lexical value for that symbol
> is?
No.
The easy proof of that is this:
(defun foo ()
(let ((x 3))
#'(lambda () x)))
=> #<FUNCTION ...>
Now you might want to poke at the function to ask what its value of
X is, but if you could do that, it would be the same as requiring
the compiler not to fold out the value of X, rewriting your function
as:
(defun foo ()
#'(lambda () 3))
which it should do because it can avoid consing in the process. This
function really returns a constant even though it wasn't written in a way
that made this apparent. There _is_ no value of X.
Some implementations do keep symbol maps and have ways of accessing
lexical variables. You can see this in the debugger. But often they
also make the variable vanish when its lifetime is expired. e.g.,
(defun foo (z)
(let* ((x 3) (y (+ x 3)))
#'(lambda () y)))
would not be a constant, and would be closed over Y but the X is done
with by the time of the closure creation and would not exist.
> How about for a macro to find out what lexical bindings exist in
> an &environment variable, like the proposed MAP-ENVIRONMENT?
Not in the standard.
>> Is there a way for running code to find out, based on a symbol (not
>> known at compile time), what the current lexical value for that symbol
>> is?
> No.
> The easy proof of that is this:
> (defun foo ()
> (let ((x 3))
> #'(lambda () x)))
Thanks for the response. Let me clarify my question. When I said
"running code", I meant "code that I write, running within the lexical
environment I'm trying to probe". So it can have variable accesses to
X that prevent it from being folded away, or whatever else.
So while I agree with you for the general case, I don't think that
proof applies for the special case when I have the luxury of running a
code-walker over the code, instrumenting it before it's submitted to
Lisp.
I apologize; my initial paragraph (the one you quoted) was unclear, or
rather less specific than it should have been. I don't have the
symbol name at compile time, but I can make adjustments to the code at
compile time, so long as the function still does what it's supposed
to. In the example I listed later in the message, it instruments the
user-supplied function to call my own code.
The good news about that case is that I can put whatever additional
code I need in there, to add variable accesses that would prevent the
X in your example from being folded away. The bad news is that, in
the only general solution I've figured out so far, it would require a
list of current lexical bindings (at compile-time), which is what led
to my second question (about MAP-ENVIRONMENT).
I was hoping to avoid making the code-walker look for forms that
create new lexical bindings. (I'll have the complete top-level form,
so I can start from a null lexical environment.) Needing to do that
makes the code walker more complex, it couldn't handle
implementation-defined special forms that create new bindings, and
would require almost a complete implementation of complex
binding-creating forms like LOOP.
>> How about for a macro to find out what lexical bindings exist in
>> an &environment variable, like the proposed MAP-ENVIRONMENT?
> Not in the standard.
Nuts. At this point, that's really what's keeping me from writing The
Debugger I Always Wanted. ANSI CL is *so* close to what I want, but
just not quite there.
Thanks for your help,
joelh
--
Joel Ray Holveck - ·····@piquan.org
Fourth law of programming:
Anything that can go wrong wi
sendmail: segmentation violation - core dumped
From: Kent M Pitman
Subject: Re: Lexical environment access at runtime
Date:
Message-ID: <wkptg4tg3t.fsf@nhplace.com>
Joel Ray Holveck <·····@piquan.org> writes:
> So while I agree with you for the general case, I don't think that
> proof applies for the special case when I have the luxury of running a
> code-walker over the code, instrumenting it before it's submitted to
> Lisp.
The problem is that if you do something to instrument it beforehand,
you are still saying "don't optimize it or I won't know what I'm doing.
Consider:
(let ((x (f)))
(+ x (let ((x 3))
(print (lexical-value 'x))
x)
(* 5 x)))
If this would get constant-folded to
(let ((x (f)))
(+ x (progn (lexical-value 'x) 3) (* 5 x)))
you are effectively saying "don't apply that optimization".
You risk that your "instrumentation" is going to change what you are
measuring, such that you may see a bug go away just for having put in
your instrumentation. Such bugs are notoriously hard to debug since
the instrumentation is making them harder to debug.
It may in some cases be better to cooperate with an implementor and
find out how to read the compiler debugging maps, etc., though by
this I mean "better for the user" not "better in that it makes it
easier for you to write".
Just my opinion.
>> So while I agree with you for the general case, I don't think that
>> proof applies for the special case when I have the luxury of running a
>> code-walker over the code, instrumenting it before it's submitted to
>> Lisp.
> The problem is that if you do something to instrument it beforehand,
> you are still saying "don't optimize it or I won't know what I'm doing.
I'm fine with that tradeoff. I'm not looking at applying this to all
code, only the code the user wants to instrument for debugging. After
the user has debugged the function, he can re-eval its defun without
the instrumentation and let the optimizer go to town. So only the
function (or functions) under scrutiny gets instrumented.
I'm basing this on Emacs's edebug. If you hit C-M-x, that evals a
defun. If you hit C-u C-M-x, it instruments the defun, and sets a
breakpoint on its entry. There's no interface to instrument lots of
defuns at once, because it is rarely useful to do so. The user (at
least, I) will typically instrument a defun that bears investigation,
invoke it (directly or indirectly), and watch the return values of the
body forms as they're executed, to make sure they agree with what I
expect. I used the term "debugger", since that's what other
languages' debuggers do, but it corresponds more to Lisp's stepper
than its debugger.
> You risk that your "instrumentation" is going to change what you are
> measuring, such that you may see a bug go away just for having put in
> your instrumentation. Such bugs are notoriously hard to debug since
> the instrumentation is making them harder to debug.
This is true, and I've come across that problem many times in my C
work, and at least once in Perl. But the same could be said for any
other mechanical debugging aid, including TRACE, STEP, or manually
inserting FORMATs / BREAKs / etc. It's practically impossible for a
debugger to not affect the program being debugged; even in an
otherwise ideal debugging environment, there may be race conditions
that disappear during the time when the user is at a debugger prompt.
But, despite that risk, I think that having the tool available will
still be handy. I expect in the majority of cases, it probably won't
affect the bug. At least, that's my experience in other languages,
and I expect it would be less of a problem in Lisp than most others.
Note that I'm not intending to replace the Lisp implementation's
debugger; more like adding a separate stepper. This is something that
will be invoked by user action, not by the code raising an error.
Most vendors' Lisp debuggers are great when code actually raises an
error; I don't intend to replace that. I just want something that I
can use to see where my code computes something I didn't expect it to,
not to examine the state of the world at a single predetermined point,
like the Lisp debugger does.
> It may in some cases be better to cooperate with an implementor and
> find out how to read the compiler debugging maps, etc., though by
> this I mean "better for the user" not "better in that it makes it
> easier for you to write".
Okay. It looks like that's where this is going, so that may be a good
route to take. I'll have to look at some implementations' debugging
interfaces so I can figure out a suitable abstraction interface, but
that is a line that can bend over time.
> Just my opinion.
And I certainly appreciate it, thanks! I may sound a bit defensive,
but mostly that's because I want people to keep picking at my ideas
until I've formed a workable plan.
Thanks,
joelh
PS: The HyperSpec's X3J13 issues description says that the environment
functions were added then rescinded, but doesn't mention why they were
rescinded. Do you recall offhand?
--
Joel Ray Holveck - ·····@piquan.org
Fourth law of programming:
Anything that can go wrong wi
sendmail: segmentation violation - core dumped
From: Kent M Pitman
Subject: Re: Lexical environment access at runtime
Date:
Message-ID: <wku15fsplq.fsf@nhplace.com>
Joel Ray Holveck <·····@piquan.org> writes:
> PS: The HyperSpec's X3J13 issues description says that the environment
> functions were added then rescinded, but doesn't mention why they were
> rescinded. Do you recall offhand?
Yes: There were too many bugs.
The basic scheme was believed sound, but the details were wrong.
We kept submitting changes to fix them and then new ones would appear.
We had no way of being sure it wasn't still buggy so we backed out.
It wasn't intended that this stuff would go away--just that vendors
be able to experiment with it privately until a consistent and reliable
interface was found and could later be standardized.
Joel Ray Holveck <·····@piquan.org> writes:
> >> How about for a macro to find out what lexical bindings exist in
> >> an &environment variable, like the proposed MAP-ENVIRONMENT?
> > Not in the standard.
>
> Nuts. At this point, that's really what's keeping me from writing The
> Debugger I Always Wanted. ANSI CL is *so* close to what I want, but
> just not quite there.
Two thoughts:
1) I'm sure any of the open/free Lisp implementations would be happy
to have you write the Ultimate Lisp Debugger for their implementation.
If it were me, I'd grab the SBCL source--since the point of that
project, as I understand it, was largely to optimize for internal
comprehensibility--and see what it would take to write such a
debugger. If along the way you come up with a coherent set of
primitive operations that you need to add to the implementation in
order to support an otherwise portarble Ulitimate Debugger, then the
existence of one instance of the debugger plus an obvious path for
other implementations to support it might get you to a place where
your debugger runs on a bunch of implementations.
2) Check out what Duanne Rettig at Franz is doing with first-class
environments. He's mentioned here a couple times that a future release
of Franz is going to bring back some of the environment stuff from
CLTL2 that didn't make it into the standard. It may be that that work
*is* the underlying primitives you need and you can write your
debugger on Franz first and then see if you can help other impls
provide similar facilities.
-Peter
--
Peter Seibel ·····@javamonkey.com
Lisp is the red pill. -- John Fraser, comp.lang.lisp
> Two thoughts:
Thanks, I'll check it out!
joelh
--
Joel Ray Holveck - ·····@piquan.org
Fourth law of programming:
Anything that can go wrong wi
sendmail: segmentation violation - core dumped
Kent M Pitman <······@nhplace.com> writes:
> Some implementations do keep symbol maps and have ways of accessing
> lexical variables. You can see this in the debugger. But often they
> also make the variable vanish when its lifetime is expired. e.g.,
>
> (defun foo (z)
> (let* ((x 3) (y (+ x 3)))
> #'(lambda () y)))
>
> would not be a constant, and would be closed over Y but the X is done
> with by the time of the closure creation and would not exist.
But there's no requirement to keep the binding of Y, so a sufficiently
smart compiler could rewrite it to
(defun foo (z)
#'(lambda () 6))
, right?
--
Lars Brinkhoff, Services for Unix, Linux, GCC, HTTP
Brinkhoff Consulting http://www.brinkhoff.se/
From: Kent M Pitman
Subject: Re: Lexical environment access at runtime
Date:
Message-ID: <wkn0b8tfvn.fsf@nhplace.com>
Lars Brinkhoff <·········@nocrew.org> writes:
> Kent M Pitman <······@nhplace.com> writes:
> > Some implementations do keep symbol maps and have ways of accessing
> > lexical variables. You can see this in the debugger. But often they
> > also make the variable vanish when its lifetime is expired. e.g.,
> >
> > (defun foo (z)
> > (let* ((x 3) (y (+ x 3)))
> > #'(lambda () y)))
> >
> > would not be a constant, and would be closed over Y but the X is done
> > with by the time of the closure creation and would not exist.
>
> But there's no requirement to keep the binding of Y, so a sufficiently
> smart compiler could rewrite it to
>
> (defun foo (z)
> #'(lambda () 6))
>
> , right?
I'd meant to do (+ x z), which is why I'd introduced z as a parameter,
so that the closure was dependent on an opaque quantity that could not
be statically folded. You're right to point out that I failed.
(I specifically didn't show calls to foo, since you might claim the
compiler could block compile under such cases. There were stories when
I was an undergrad about optimizing compilers for FORTRAN that ran
certain benchmarks in zero time by noting an absence of I/O statements
and eliminating all the computation done for the sake of eating time. :)
But either way, it makes again the original case that the compiler's
level of smartness affects what you will see.
Joel Ray Holveck <·····@piquan.org> writes:
> Anybody have any thoughts?
Well, If I strike "C-u M-C-x" rather than "M-C-x" in my lisp
environment, my lisp will create an interpreted function for me, where
every bit of source-code information is available at run-time. But I
don't need to do that very often, because the debugger is more often
than not able to provide the current value of a lexical
variable. Although it took a while for me to pick up the documentation
for the debugger to see how this was done.
--
Frode Vatvedt Fjeld
Joel Ray Holveck wrote:
> Is there a way for running code to find out, based on a symbol (not
> known at compile time), what the current lexical value for that symbol
> is? How about for a macro to find out what lexical bindings exist in
> an &environment variable, like the proposed MAP-ENVIRONMENT?
>
> That's the short question. If you want to know why I'm asking, read
> on! This request comes from me being fed up with the current
> source-level debugging facilities I find in both current open-source
> and commercial Lisp implementations.
>
> Anybody who's done much Emacs programming probably knows how useful
> edebug is. It's a source-level Emacs Lisp debugger that allows
> stepping, evaluation, breakpoints, and other useful features. (When I
> say "debugger", I mean something like what C people think of as a
> debugger, which is more like a stepper in Lisp.) I generally find
> CL's STEP pretty much useless (and X3J13 pretty much said as much, I
> think), so I decided to write my own edebug-like facility for Common
> Lisp. The goal is something similar to Jabberwocky's debug
> facilities, but with a loose coupling between the Lisp and the editor
> so that any extensible editor (for me, Emacs) can be used. But since
> Emacs uses dynamic bindings instead of lexical, there's a big
> difference in implementation strategy between edebug and my program
> (tentatively called "Easel").
Sidenote: it's dynamic vs. lexical _scoping_, not "binding".
And this points to a possible solution: you could declare the variables
that you want to inspect as special and make them dynamically scoped
that way. This is possible for local variables in Common Lisp. However,
things will get hairy if you still need to ensure that they are unique.
Maybe you can fiddle with MAKE-SYMBOL and symbol macros, but I am not
sure if you can make this work. On the other hand, if you only want to
use it for debugging purposes, you might not need a 100% perfect solution.
It might be instructive to check out
http://www.tfeb.org/lisp/hax.html#DYNAMIC-STATE
(Admittedly, these are just very rough ideas, but maybe they are helpful...)
Pascal