This is going to take a bit of explaining, so let me apologize in
advance for the length of this post.
Assume that a few months from now I've developed a facility for
tracing the execution of a system of arbitrary processes running on a
single Linux host. Ideally, these processes are "trace aware" and
contain embedded diagnostic output routines (hereafter *probes*) which
add application-specific information to the trace data collected.
But certain basic data, like the system calls made, can be collected
even from non-aware processes.
This facility is designed so that any program can readily be made
trace aware -- regardless of what programming language it happens to
be written in. There is (probably) special support for C -- because so
much of Linux is coded in C, and C (being a portable assembler) makes
certain low-level optimizations both desirable and feasible. There is
also special support for (one or more implementations of) Common Lisp
-- because the author of the facility considers Lisp one of the more
useful programming languages. In general, however, any language that
can somehow output a string can arrange to add information to an
ongoing trace.
The tracing facility is controlled using a command called trex. trex
basically behaves like a Unix shell, but processes started under trex
(and their children, and so on) have their probe output collected in a
trace[1]. trex also has various commands for interactively controlling
trace aware processes with dynamic probe support. Basically, dynamic
probe support allows trex to attach to an already running process,
selectively enable specific probes, and later (after collecting the
desired trace information) detach -- leaving the process running as
before.
[1] Trace aware processes that are *not* running under trex either
have their probe output discarded, or detect that it would be
discarded and don't produce any probe output at all.
Okay -- snap back to the present. My problem lies in how to implement
dynamic probe support. Solaris' prex has the equivalent of this
feature, but only for C programs. I suspect prex implements it by
scanning the ELF executable files for certain signature data
associated with each embedded probe. I'm pretty sure prex enables and
disables the probes by poking data into the traced processes' data
segment in memory. Not trivial, but not rocket science. But it relies
on the fact that in C it's relatively straightforward to affect the
layout of the processes' text and data segments (e.g. to embed the
probe signatures) in a predictable way.
Now, the question is how best to approach this feature given that I
also want to support dynamic probes for Common Lisp -- and in general
to come up with a technique that can be made to work for languages
that don't allow you to approximate the behavior of an assembler. (Java
would of course be another example.)
There are practical constraints which preclude a solution which is
overly intrusive to traced processes. For this reason, suggestions
involving traditional IPC techniques, specialized event-loops in trace
aware processes, etc. are likely to be dismissed out of hand. My
inclination is to limit trex's interaction with traced processes to
the following mechanisms:
-- the ptrace system call
-- POSIX signals and signal handlers
-- the facilities of a common kernel module (tnf trace buffer
support)
ptrace is necessary to the implementation of some of trex's features
anyway, and all of these mechanisms have one trait in common: they
have nominally *no* impact on a process that trex is not tracing.
I've got a few ideas in mind, but this post is definitely getting
long (and anyway the goal is to fish for suggestions) so I'll limit
myself to a few bullet points:
-- Turn probes on/off by poking into the traced processes' address
space with ptrace, but how to find where to poke? Particularly in
GC'd languages where things move around...
-- Use preprocessors to catalog the probes in an executable? One for
C, one for Common Lisp, one for Java, etc.
-- Devise a probe signature that can be embedded in some sort of
string constant, and scan for those using ptrace?
-- Require that probe-aware processes link in a special signal
handler, then use a signal to notify a traced process to read some
sort of probe control block via an ioctl to the /proc/tnf/trace
file descriptor?
Less byzantine suggestions (that still meet the constraints) are
welcome. :-)
-- Kaelin
Kaelin Colclasure <······@everest.com> writes:
> [...] My problem lies in how to implement dynamic probe
> support. Solaris' prex has the equivalent of this feature, but only
> for C programs. I suspect prex implements it by scanning the ELF
> executable files for certain signature data associated with each
> embedded probe.
You might want to look at the Paradyn research project. They're
basically dealing with exactly these issues, and have even implemented
dynamic instrumentation of OS kernels. <URL:http://www.cs.wisc.edu/~paradyn/>
I have written some code (in common lisp) for attaching to processes
and allocating buffers inside them, and like you say, it's not really
rocket science.
> Less byzantine suggestions (that still meet the constraints) are
> welcome. :-)
While hack'n slash approaches like machine-code modification via
ptrace/procfs come naturally in a C world, I'm not so sure they are
appropriate for dynamic languages like lisp. I think it'd be smarter
to have the lisp runtime system much more involved in the
instrumentation and tracing process. But this suggestion doesn't
exactly solve your problem..
--
Frode Vatvedt Fjeld
Frode Vatvedt Fjeld <······@acm.org> writes:
> Kaelin Colclasure <······@everest.com> writes:
>
> > [...] My problem lies in how to implement dynamic probe
> > support. Solaris' prex has the equivalent of this feature, but only
> > for C programs. I suspect prex implements it by scanning the ELF
> > executable files for certain signature data associated with each
> > embedded probe.
>
> You might want to look at the Paradyn research project. They're
> basically dealing with exactly these issues, and have even implemented
> dynamic instrumentation of OS kernels. <URL:http://www.cs.wisc.edu/~paradyn/>
Ah, interesting. I was aware of IBM's dProbes project for Linux.
(Thanks to another poster here on c.o.l.d.s.) I suspect these efforts
are related -- or at least collaborating at some level.
From the presentation materials available on their site, it appears
they're using symbol table information from the executables in order
to target their dynamic code modifications.
> I have written some code (in common lisp) for attaching to processes
> and allocating buffers inside them, and like you say, it's not really
> rocket science.
Did you use their dyninst API?
> > Less byzantine suggestions (that still meet the constraints) are
> > welcome. :-)
>
> While hack'n slash approaches like machine-code modification via
> ptrace/procfs come naturally in a C world, I'm not so sure they are
> appropriate for dynamic languages like lisp. I think it'd be smarter
> to have the lisp runtime system much more involved in the
> instrumentation and tracing process. But this suggestion doesn't
> exactly solve your problem..
Paradoxically, dynamic instrumentation of the product of a static
language by an external agent is relatively straightforward. Precisely
because the language itself is static, it eventually all comes down to
specific offsets into the program's text segment.
But of course, dynamic instrumentation is trivially supported in
Common Lisp if the agent itself lives in the same Lisp image. In fact,
advice, tracing and several other Common Lisp features are built on
this capability.
Hmmm, perhaps the only realistic way to adequately support such
dynamic languages is to collaborate with their native capabilities
rather than trying to second-guess them. Maybe I need to rethink my
assertion that all higher-level IPC functions are verboten. But one
motivation for that restriction was to minimize the amount of noise
incurred if system call tracing was enabled along with embedded
probes...
[Scratching head.] Maybe the whole premise of poking into the
processes' address space is flawed...
-- Kaelin
> Frode Vatvedt Fjeld <······@acm.org> writes:
>
> > I have written some code (in common lisp) for attaching to processes
> > and allocating buffers inside them, and like you say, it's not really
> > rocket science.
Kaelin Colclasure <······@everest.com> writes:
> Did you use their dyninst API?
No, it's built atop procfs for FreeBSD (differences to linux should be
minimal). I basically abandoned it as a research track when I
discovered Paradyn etc. :)
> Paradoxically, dynamic instrumentation of the product of a static
> language by an external agent is relatively
> straightforward. Precisely because the language itself is static, it
> eventually all comes down to specific offsets into the program's
> text segment.
But the potential quality of information that can be extracted from
lisp is much greated than that of C, I think.
In practical terms, I'd guess you could use the lisp's FFI to install
a gateway into the dynamic lisp world. I believe for example Allegro's
FFI will create static entries (function entries and other objects
that are guaranteed not to be moved by GC) to lisp in order to
accommodate calling lisp functions from C.
> But of course, dynamic instrumentation is trivially supported in
> Common Lisp if the agent itself lives in the same Lisp image. In
> fact, advice, tracing and several other Common Lisp features are
> built on this capability.
Yes, and I'd say you should think very carefully before you throw away
all this. The ptrace stuff is a necessity in C, but that doesn't mean
it's desirable in lisp.
> Hmmm, perhaps the only realistic way to adequately support such
> dynamic languages is to collaborate with their native capabilities
> rather than trying to second-guess them. Maybe I need to rethink my
> assertion that all higher-level IPC functions are verboten. But one
> motivation for that restriction was to minimize the amount of noise
> incurred if system call tracing was enabled along with embedded
> probes...
There are contradictory goals of low intrusion and quality of
extracted information. (Ideally, the user should be able to chose
which is preferred.) But the situation is rather different in lisp and
C. Lisp images are much more soundly constructed, and with good
introspective capabilities (and dynamism), which means you can do lots
of stuff (high-level or whatever) with only some performance intrusion
(i.e. a slow-down, but practically no functional intrusion; the
program still works exactly the same).
> [Scratching head.] Maybe the whole premise of poking into the
> processes' address space is flawed...
Most any instrumentation scheme can be seen as "poking in processes'
address space", but this activity must be based on some set of
assumptions about the image. These assumptions typically stem from the
CPU architecture, OS conventions and the compiler that produced the
code (function calling conventions, for example). The nature of these
assumptions is quite different for C and lisp images.
--
Frode Vatvedt Fjeld
Frode Vatvedt Fjeld <······@acm.org> writes:
[...]
> Kaelin Colclasure <······@everest.com> writes:
[...]
> > Paradoxically, dynamic instrumentation of the product of a static
> > language by an external agent is relatively
> > straightforward. Precisely because the language itself is static, it
> > eventually all comes down to specific offsets into the program's
> > text segment.
[...]
> > But of course, dynamic instrumentation is trivially supported in
> > Common Lisp if the agent itself lives in the same Lisp image. In
> > fact, advice, tracing and several other Common Lisp features are
> > built on this capability.
>
> Yes, and I'd say you should think very carefully before you throw away
> all this. The ptrace stuff is a necessity in C, but that doesn't mean
> it's desirable in lisp.
I find this argument pretty compelling.
> > Hmmm, perhaps the only realistic way to adequately support such
> > dynamic languages is to collaborate with their native capabilities
> > rather than trying to second-guess them. Maybe I need to rethink my
> > assertion that all higher-level IPC functions are verboten. But one
> > motivation for that restriction was to minimize the amount of noise
> > incurred if system call tracing was enabled along with embedded
> > probes...
>
> There are contradictory goals of low intrusion and quality of
> extracted information. (Ideally, the user should be able to chose
> which is preferred.) But the situation is rather different in lisp and
> C. Lisp images are much more soundly constructed, and with good
> introspective capabilities (and dynamism), which means you can do lots
> of stuff (high-level or whatever) with only some performance intrusion
> (i.e. a slow-down, but practically no functional intrusion; the
> program still works exactly the same).
[...]
You've encouraged me to think about the problem from a new angle.
Particularly your parenthetical statement above about allowing users
to trade-off between intrusiveness and quality of information.
I'm now thinking that, rather than employing esoteric methods for
locating probes in the variety of possible executable formats, it
makes more sense to defer the cataloging of probes to a separate
tool. Thus, the trex program would simply consult a file with
definitions for each of the various probes a process has available.
In the event that the definitions are not available (perhaps because
this executable came from a third party), trex can be instructed to
simply enable all probes in the process -- but dump them in "raw" form
since it doesn't have the information necessary to "cook" the data.
From the raw data it would no doubt be possible to reverse-engineer
suitable definitions for at least the most interesting probes.
In addition to better meeting your suggested goals, this approach
considerably simplifies the implementation of trex, is more portable
across OS/hardware platforms, and also simplifies the extension of the
system to support new programming languages.
The trade-off, of course, is the introduction of ancillary "probe
definition" files, and the necessity of a tool chain to support their
construction and maintenance. But I'm thinking this is probably a fine
compromise...
-- Kaelin
Kaelin Colclasure <······@everest.com> writes:
> My problem lies in how to implement
> dynamic probe support. Solaris' prex has the equivalent of this
> feature, but only for C programs. I suspect prex implements it by
> scanning the ELF executable files for certain signature data
> associated with each embedded probe.
The problem is one of how to identify the probe. The common type of
probe is a breakpoint in any of the debuggers that use ptrace (e.g.
dbx, gdb ...) and these are usually specified either by address or by
symbol-name and offset. (Higher-level specifications include line
numbers, which count on such line number information being available
and up-to-date, but these usually translate to an offset from a symbol.)
An address breakpoint may become invalidated over more than one invocation
of the executable, because a shared-library might have moved between the
end of one program and its next invocation. I have seen debuggers on
some operating systems not bother to re-establish by-address breakpoints
at the start of a new run of the program under debug, but others only
invalidate those breakpoints if the shared-library did in fact move.
It is also possible to dynamically load and unload shared-libraries
during normal program execution, but I am not sure what ptrace
implementations do with these dynamics.
The above C-style scenario is predicated on a shared-library being
a module which, if it moves, moves as a unit, and not very often.
The problem/feature of lisp is that it is the _function_'s code
which is a dynamic unit which might move all on its own.
Breakpoints or probes must be identified within the lisp by
function and offset, and not by address. Also, the code is much
more likely to move, since it is in most cases simply lisp data
in some form of vector. So the probe mechanism must consult the
lisp under test for the exact identity of the probe, because
otherwise the address it sees is arbitrary.
Allegro CL has a mechanism called "lldb" (lisp low-level debugger)
which sets breakpoints at the instruction level and responds to
them to provide an instruction stepper. The mechanism existed in
5.0.1 but has been improved in 6.0. Breakpoints are set via the
:br[eak] top-level command, and enabled via the :ldb command. A
breakpoint is identified by function and pc-offset.
Unfortunately, the lldb facility as of yet has no hooks to provide
for general probing. I suppose such hooks could be added in the
future.
> I'm pretty sure prex enables and
> disables the probes by poking data into the traced processes' data
> segment in memory. Not trivial, but not rocket science. But it relies
> on the fact that in C it's relatively straightforward to affect the
> layout of the processes' text and data segments (e.g. to embed the
> probe signatures) in a predictable way.
> Now, the question is how best to approach this feature given that I
> also want to support dynamic probes for Common Lisp -- and in general
> to come up with a technique that can be made to work for languages
> that don't allow you to approximate the behavior of an assembler. (Java
> would of course be another example.)
If you also want to hook an interpreter, you would need support for this.
I don't know what Java provides. CLtL1 defined hook variables *evalhook*,
*applyhook*, and some other functionality to to this. X3J13 removed these,
so CL doesn't have them. However, some implementations may retain this
functionality as backward compatibility. (Allegro CL provides for this)
--
Duane Rettig Franz Inc. http://www.franz.com/ (www)
1995 University Ave Suite 275 Berkeley, CA 94704
Phone: (510) 548-3600; FAX: (510) 548-8253 ·····@Franz.COM (internet)
Duane Rettig <·····@franz.com> writes:
> Kaelin Colclasure <······@everest.com> writes:
>
> > My problem lies in how to implement
> > dynamic probe support. Solaris' prex has the equivalent of this
> > feature, but only for C programs. I suspect prex implements it by
> > scanning the ELF executable files for certain signature data
> > associated with each embedded probe.
>
> The problem is one of how to identify the probe. The common type of
> probe is a breakpoint in any of the debuggers that use ptrace (e.g.
> dbx, gdb ...) and these are usually specified either by address or by
> symbol-name and offset. (Higher-level specifications include line
> numbers, which count on such line number information being available
> and up-to-date, but these usually translate to an offset from a symbol.)
> An address breakpoint may become invalidated over more than one invocation
> of the executable, because a shared-library might have moved between the
> end of one program and its next invocation. I have seen debuggers on
> some operating systems not bother to re-establish by-address breakpoints
> at the start of a new run of the program under debug, but others only
> invalidate those breakpoints if the shared-library did in fact move.
> It is also possible to dynamically load and unload shared-libraries
> during normal program execution, but I am not sure what ptrace
> implementations do with these dynamics.
That's a good point. I'm pretty sure prex scans the ELF executable
for .so dependencies and recursively loads and scans all of them for
probes as well -- but I don't know *what* it does when the code
dlopen's a .so that contains probes. Probably nothing. :-/
Hmmm, let's hypothesize for a bit that I've become convinced that in
order to do dynamic probe support properly, some responsibility is
going to have to be delegated to trace aware processes. For instance
that they have to provide a catalog of their probes (somehow) and that
they have to implement some primitive mechanism for enabling and
disabling them under trex's control. What primitive mechanisms will
interact well (or at least predictably) with Common Lisp? For
instance, if I implement a signal handler in a supporting .so, can it
safely call out to Lisp code exposed by the FFI layer?
[...]
> > Now, the question is how best to approach this feature given that I
> > also want to support dynamic probes for Common Lisp -- and in general
> > to come up with a technique that can be made to work for languages
> > that don't allow you to approximate the behavior of an assembler. (Java
> > would of course be another example.)
>
> If you also want to hook an interpreter, you would need support for this.
> I don't know what Java provides. CLtL1 defined hook variables *evalhook*,
> *applyhook*, and some other functionality to to this. X3J13 removed these,
> so CL doesn't have them. However, some implementations may retain this
> functionality as backward compatibility. (Allegro CL provides for this)
At the point interpreted code comes into the picture, I see little
benefit in making heroic efforts to enable dynamic probe support at
the level of the traced process. It's much simpler (and possibly even
more efficient) to eat the system call and then let the driver decide
whether to keep the probe data or not. I call this style of probe a
"Compatible String Probe" or CSP. The (perceived, by me at least)
problem with CSP's is that they're relatively expensive. They incur
the expense of a system call and parsing at the driver level whether
enabled or not -- e.g. you pay for them even when you're not using
them. In general that's something I'm trying hard to avoid.
-- Kaelin
"Kaelin Colclasure" <······@everest.com> wrote in message
···················@vanguard.arslogica.lan...
> Duane Rettig <·····@franz.com> writes:
> > It is also possible to dynamically load and unload shared-libraries
> > during normal program execution, but I am not sure what ptrace
> > implementations do with these dynamics.
>
> That's a good point. I'm pretty sure prex scans the ELF executable
> for .so dependencies and recursively loads and scans all of them for
> probes as well -- but I don't know *what* it does when the code
> dlopen's a .so that contains probes. Probably nothing. :-/
>
I have not tried this, but prex could easily get notified about .so
load/unload
events on Solaris via the runtime linker-debugger interfaces:
·······································@Ab2PageView/10995?Ab2Lang=C&Ab2Enc=i
so-8859-1
I'd be surprized if it actually did nothing.
"Paul Pluzhnikov" <········@gte.net> writes:
> "Kaelin Colclasure" <······@everest.com> wrote in message
> ···················@vanguard.arslogica.lan...
> > Duane Rettig <·····@franz.com> writes:
> > > It is also possible to dynamically load and unload shared-libraries
> > > during normal program execution, but I am not sure what ptrace
> > > implementations do with these dynamics.
> >
> > That's a good point. I'm pretty sure prex scans the ELF executable
> > for .so dependencies and recursively loads and scans all of them for
> > probes as well -- but I don't know *what* it does when the code
> > dlopen's a .so that contains probes. Probably nothing. :-/
> >
>
> I have not tried this, but prex could easily get notified about .so
> load/unload
> events on Solaris via the runtime linker-debugger interfaces:
> ·······································@Ab2PageView/10995?Ab2Lang=C&Ab2Enc=i
> so-8859-1
>
> I'd be surprized if it actually did nothing.
I believe you are both correct. Evidence for this is shown below.
Having said this, though, I cannot do this in all debug systems; I
am not sure whether the problems are in dlopen, gdb/dbx, or ptrace.
So I usually only do this on systems which I have learned to trust
as having gotten it right.
The big idea is that when a breakpoint is set that doesn't yet exist,
it is deferred. The loading of the first library which contains
that entry point will establish an address for the breakpoint, and
insert the actual breakpont at that address. This must occur with
some hook at dlopen time, or else the breakpoint would never have been
set in the example below.
I have never had occasion to try dlclosing a library with a breakpoint
in it, so I don't know whether dlclose works as it intuitively should
(i.e. hooking the deletion/re-deferral of breakpoints upon unloading
of a library).
% gdb lisp
Detected 64-bit executable.
Invoking /opt/langtools/bin/gdb64.
HP gdb 2.0
Copyright 1986 - 1999 Free Software Foundation, Inc.
Hewlett-Packard Wildebeest 2.0 (based on GDB 4.17-hpwdb-980821)
Wildebeest is free software, covered by the GNU General Public License, and
you are welcome to change it and/or distribute copies of it under certain
conditions. Type "show copying" to see the conditions. There is
absolutely no warranty for Wildebeest. Type "show warranty" for details.
Wildebeest was built for PA-RISC 2.0 (wide), HP-UX 11.00.
..
(gdb) break lisp_main
Breakpoint 1 (deferred) at "lisp_main" ("lisp_main" was not found).
Breakpoint deferred until a shared library containing "lisp_main" is loaded.
(gdb) info break
Num Type Disp Enb Address What
1 breakpoint keep n (deferred) in lisp_main
(gdb) run -I dcl
Starting program: /acl/duane/acl61/src/lisp -I dcl
Breakpoint 1, lisp_main (argc=3, argv=0x80000001000248c0,
envp=0x800003ffefff05a8) at c/startup.c:373
373 int startup_debug = (char *)getenv("ACL_STARTUP_DEBUG") != NULL;
Current language: auto; currently c++
(gdb) info break
Num Type Disp Enb Address What
1 breakpoint keep y 0x800003ffefcce478 in lisp_main at c/startup.c:373
breakpoint already hit 1 time
(gdb)
--
Duane Rettig Franz Inc. http://www.franz.com/ (www)
1995 University Ave Suite 275 Berkeley, CA 94704
Phone: (510) 548-3600; FAX: (510) 548-8253 ·····@Franz.COM (internet)