I've searched the archives (a treasure trove with decades of gems).
I _think_ I understand packages...
I'm sure I don't understand their relationship to the reader...(or visa
versa)...
I want to write a program that analyzes code.
Aha, I can use 'read' instead of writing yet another Lisp parser...
...except that... any code I read that names other packages explicitly
breaks the reader. If only the author(s) of the code-to-be-read had used
use-package, or import... (then the symbols would be "unadorned"...) but
they didn't and in general, can't be made to.
What are my options?
a) write my own reader function (uck, lots of redundant work).
b) catch/handle the read error and try to continue.
c) tweak the reader to do my own symbol reading.
d) redefine intern or find-package?
e) some other options???
Any insight or pointers to specific archive entries appreciated! (there
are a lot of package related messages in the archives... info overload!)
<D\'gou
Douglas Philips <····@mac.com> writes:
> I want to write a program that analyzes code.
> Aha, I can use 'read' instead of writing yet another Lisp parser...
> ...except that... any code I read that names other packages explicitly
> breaks the reader. If only the author(s) of the code-to-be-read had
> used use-package, or import... (then the symbols would be
> "unadorned"...) but they didn't and in general, can't be made to.
>
> What are my options?
> a) write my own reader function (uck, lots of redundant work).
> b) catch/handle the read error and try to continue.
> c) tweak the reader to do my own symbol reading.
> d) redefine intern or find-package?
> e) some other options???
>
It's a cheesy hack, but you could tweak the readtable so that `:'
wasn't a package marker. You'd end up with symbols with embedded
colons, and they wouldn't `alias' correctly (suppose they referred to
an exported symbol, but the programmer used : and :: inconsistently),
but it might get things working enough for your purposes.
Joe Marshall <···@ccs.neu.edu> writes:
> Douglas Philips <····@mac.com> writes:
>
> > I want to write a program that analyzes code.
> > Aha, I can use 'read' instead of writing yet another Lisp parser...
> > ...except that... any code I read that names other packages explicitly
> > breaks the reader. If only the author(s) of the code-to-be-read had
> > used use-package, or import... (then the symbols would be
> > "unadorned"...) but they didn't and in general, can't be made to.
> >
> > What are my options?
> > a) write my own reader function (uck, lots of redundant work).
> > b) catch/handle the read error and try to continue.
> > c) tweak the reader to do my own symbol reading.
> > d) redefine intern or find-package?
> > e) some other options???
> >
>
> It's a cheesy hack, but you could tweak the readtable so that `:'
> wasn't a package marker. You'd end up with symbols with embedded
> colons, and they wouldn't `alias' correctly (suppose they referred to
> an exported symbol, but the programmer used : and :: inconsistently),
> but it might get things working enough for your purposes.
I don't think that works. ':' is special because of its constituent
trait, not it's character syntax. And there "no mechanism is provided
for changing the constituent trait of a character." (CLHS, 2.1.4.2)
-Peter
--
Peter Seibel ·····@javamonkey.com
Lisp is the red pill. -- John Fraser, comp.lang.lisp
Peter Seibel <·····@javamonkey.com> writes:
> Joe Marshall <···@ccs.neu.edu> writes:
>
>> Douglas Philips <····@mac.com> writes:
>>
>> > I want to write a program that analyzes code.
>> > Aha, I can use 'read' instead of writing yet another Lisp parser...
>> > ...except that... any code I read that names other packages explicitly
>> > breaks the reader. If only the author(s) of the code-to-be-read had
>> > used use-package, or import... (then the symbols would be
>> > "unadorned"...) but they didn't and in general, can't be made to.
>> >
>> > What are my options?
>> > a) write my own reader function (uck, lots of redundant work).
>> > b) catch/handle the read error and try to continue.
>> > c) tweak the reader to do my own symbol reading.
>> > d) redefine intern or find-package?
>> > e) some other options???
>> >
>>
>> It's a cheesy hack, but you could tweak the readtable so that `:'
>> wasn't a package marker. You'd end up with symbols with embedded
>> colons, and they wouldn't `alias' correctly (suppose they referred to
>> an exported symbol, but the programmer used : and :: inconsistently),
>> but it might get things working enough for your purposes.
>
> I don't think that works. ':' is special because of its constituent
> trait, not it's character syntax. And there "no mechanism is provided
> for changing the constituent trait of a character." (CLHS, 2.1.4.2)
While that's true, I think the reader macro test happens while
accumulating the token, but the constituent trait test happens after
the token is accumulated. So you ought to be able to assign the colon
to be a terminating macro character. I said it was cheesy.
In article <················@mac.com>, Douglas Philips <····@mac.com> wrote:
>I've searched the archives (a treasure trove with decades of gems).
>I _think_ I understand packages...
>I'm sure I don't understand their relationship to the reader...(or visa
>versa)...
>
>I want to write a program that analyzes code.
>Aha, I can use 'read' instead of writing yet another Lisp parser...
>...except that... any code I read that names other packages explicitly
>breaks the reader. If only the author(s) of the code-to-be-read had used
>use-package, or import... (then the symbols would be "unadorned"...) but
>they didn't and in general, can't be made to.
How does it "break the reader"?
>What are my options?
> a) write my own reader function (uck, lots of redundant work).
> b) catch/handle the read error and try to continue.
> c) tweak the reader to do my own symbol reading.
> d) redefine intern or find-package?
> e) some other options???
>
>Any insight or pointers to specific archive entries appreciated! (there
>are a lot of package related messages in the archives... info overload!)
Most programs that analyze code have similar requirements to the compiler.
In particular, any packages that are used by the code need to be defined at
analysis time, just as the compiler requires them to be defined at compile
time. What this probably means is that you'll have to process EVAL-WHEN
expressions like the compiler does: if it appears at toplevel (perhaps
after macro expansion) and contains the :COMPILE-TOPLEVEL keyword, you'll
have to evaluate it.
Then you just need to document the fact that they'll need to present the
file containing the DEFPACKAGE form to your analyzer before telling it to
analyze the file containing the functions.
--
Barry Margolin, ··············@level3.com
Level(3), Woburn, MA
*** DON'T SEND TECHNICAL QUESTIONS DIRECTLY TO ME, post them to newsgroups.
Please DON'T copy followups to me -- I'll assume it wasn't posted to the group.
Barry Margolin wrote:
> In article <················@mac.com>, Douglas Philips <····@mac.com> wrote:
>>I want to write a program that analyzes code.
>>Aha, I can use 'read' instead of writing yet another Lisp parser...
>>...except that... any code I read that names other packages explicitly
>>breaks the reader. If only the author(s) of the code-to-be-read had used
>>use-package, or import... (then the symbols would be "unadorned"...) but
>>they didn't and in general, can't be made to.
>
>
> How does it "break the reader"?
It causes the reader to signal a condition that lands me in the
debugger. Syntacticly, "foo:bar", "bar", "zaphod:sym1" are all names of
symbols. The "breakage" occurs because the reader attempts a semantic
action, namely looking for (and failing to find) a package on what is a
perfectly syntacticly valid form. I understand why the reader normally
does this. Given that Lisp has been around for so long, and supports so
much 'reflective'-style programming, I had expected to find something in
the archives that would be a decent answer. (For example, I can control
how the reader handles upper/lower case letters, etc.)
> Then you just need to document the fact that they'll need to present the
> file containing the DEFPACKAGE form to your analyzer before telling it to
> analyze the file containing the functions.
I really would like to avoid that. My analyzer wants to look at
structure. Once I start evaluating code, I really don't know (turing
complete problem) what it might do. I want to look at the code itself
without evaluating anything. I suppose "way back in the day" this wasn't
an issue because there weren't packages to blur the syntax and semantics
of "just" ;-) reading code.
Thanks for the tips on eval-when evaluation... I hope I can find a way
to avoid having to use them. ;-)
<D\'gou
Douglas Philips <····@mac.com> writes:
> It causes the reader to signal a condition that lands me in the
> debugger. Syntacticly, "foo:bar", "bar", "zaphod:sym1" are all names of
> symbols. The "breakage" occurs because the reader attempts a semantic
> action, namely looking for (and failing to find) a package on what is a
> perfectly syntacticly valid form.
Perhaps you could explain what you wish to have happen. If you had a reader
with the magical properties that you're hoping for, what would you like it to
return if given the form
(foo:bar bar zaphod:sym1)
when neither FOO nor ZAPHOD packages exist yet?
The real reader "breaks" because there's no obvious solution about what to do
next. If you have a particular solution in mind, perhaps you can share it and
then people can advise you on how to achieve that.
(For example: do you just want to ignore all the package prefixes? Create
them automatically? Rename foo:bar to foo_bar? Etc.)
-- Don
_______________________________________________________________________________
Don Geddis http://don.geddis.org/ ···@geddis.org
It is not from the benevolence of the butcher, the brewer, or the baker, that
we expect our dinner, but from their regard to their own self-interest. We
address ourselves, not to their humanity but to their self-love, and never talk
to them of our own necessities but of their advantages. -- Adam Smith
Don Geddis wrote:
> Perhaps you could explain what you wish to have happen. If you had a reader
> with the magical properties that you're hoping for, what would you like it to
> return if given the form
> (foo:bar bar zaphod:sym1)
> when neither FOO nor ZAPHOD packages exist yet?
Just as I can tweak what happens with the case of letters, I had hoped
that there was some way to relieve ":" of its package delimitting
properties. In essence, wrap a pair of vertical bars around the symbol.
At least I think Section 2.1.4.5 says that will work... (Section 2.1.4
and Figure 2-7 there in indicate : is a constituent character, just as
the alphabetics are), though it doesn't talk about the traits of the
characters, just that they are alphabetic.
> The real reader "breaks" because there's no obvious solution about what to do
> next. If you have a particular solution in mind, perhaps you can share it and
> then people can advise you on how to achieve that.
>
> (For example: do you just want to ignore all the package prefixes? Create
> them automatically? Rename foo:bar to foo_bar? Etc.)
I didn't have any one particular solution in mind other than "being able
to read forms and 'sort out' what happened to packaging characters
later." ;-) I was hoping to be able to choose from a couple of different
options. I'm guessing that even if I could, I wouldn't wan't to make : a
reader macro, wouldn't that end the previous symbol (the package name)
and even if I return (values) from the macro, the unqualified symbol
name would be read as a separate symbol, right
Thanks,
<D\'gou
In article <················@mac.com>, Douglas Philips <····@mac.com> wrote:
>Barry Margolin wrote:
>> Then you just need to document the fact that they'll need to present the
>> file containing the DEFPACKAGE form to your analyzer before telling it to
>> analyze the file containing the functions.
>
>I really would like to avoid that. My analyzer wants to look at
>structure. Once I start evaluating code, I really don't know (turing
>complete problem) what it might do. I want to look at the code itself
>without evaluating anything. I suppose "way back in the day" this wasn't
>an issue because there weren't packages to blur the syntax and semantics
>of "just" ;-) reading code.
I don't think you can just "look at structure". Lisp allows the programmer
to change the syntax of the language, and you have to take that into
account. That means expanding macros, reader macros, and dealing with
packages.
There's no magical solution. A program analyzer needs to recognize all the
ways that the programmer can modify the lexical and syntactic components of
the language. That means recognizing macros, reader macros, and package
definitions.
--
Barry Margolin, ··············@level3.com
Level(3), Woburn, MA
*** DON'T SEND TECHNICAL QUESTIONS DIRECTLY TO ME, post them to newsgroups.
Please DON'T copy followups to me -- I'll assume it wasn't posted to the group.
Barry Margolin wrote:
> I don't think you can just "look at structure". Lisp allows the programmer
> to change the syntax of the language, and you have to take that into
> account. That means expanding macros, reader macros, and dealing with
> packages.
>
> There's no magical solution. A program analyzer needs to recognize all the
> ways that the programmer can modify the lexical and syntactic components of
> the language. That means recognizing macros, reader macros, and package
> definitions.
Ah, I wondered if/when that'd pop up. ;-)
Actually, I want to look at the structure of the source. I don't _want_
macros expanded, I want to analyze how they're called. Reader macros are
a stickier wicket. I should have been more explicit in my original
message that I want to analyze ANSI Common Lisp. Yes, ACL allows for
reader macros, but the only ones I want to deal with are the ones
defined by the standard. In any event, "user" reader macros are
orthogonal to the built in ACL read's handling of packages...
Packages are (if I may summarize) name spaces, they are semantic, not
syntactic, entities, so I would argue that by using packages alone one
cannot alter either the lexical or syntatic structure of ACL. Yes,
reader macros can do both. Regular macros still have to live within "the
system", they are invoked either as symbols or as functions,
syntacticly. Packages don't get their own read-tables, so I can't "jump"
into a different syntax by just qualifying/prefixing a package name...
foo:()are-now-regular-letters-in-the-foo-package
isn't possible, at least, in my understanding of packages its not. ;-)
So, forgoing "user" reader macros, ACL's read function gets me 99.99% of
what I want to look at the structure of source code. If I cared to
preserve comments, or the distinction between FOO |FOO| \F\O\O I would
have to use another mechanism. Fortunately read lets me ignore all of
that. I just wish I could tell read to let me "ignore" packages (well,
no, not ignore, just handle dynamicly).
I suppose I could using a handler that would attempt a read, if it got a
package error, make the package, and then re-attempt the read. I would
disable #. of course. ;-) Hmmm, hadn't thought of that one before. Its
really messy since some streams are not seekable. Hmmm...
There is a difference between "the code" and "the representation of the
code presented to read" and it is the former, more than the latter,
which I'm interesting in analyzing.
Thanks,
<D\'gou
On Tue, Aug 19, 2003 at 11:51:09AM -0400, Douglas Philips wrote:
> Packages are (if I may summarize) name spaces, they are semantic, not
> syntactic, entities, so I would argue that by using packages alone one
> cannot alter either the lexical or syntatic structure of ACL.
What is the difference between DEFUN and CL:DEFUN?
> Yes, reader macros can do both. Regular macros still have to live
> within "the system", they are invoked either as symbols or as
> functions, syntacticly.
Regular macros are syntactic extensions, but at a different level than
reader macros. (And they aren't invoked, but rather, expanded at some
point before evaluation).
--
; Matthew Danish <·······@andrew.cmu.edu>
; OpenPGP public key: C24B6010 on keyring.debian.org
; Signed or encrypted mail welcome.
; "There is no dark side of the moon really; matter of fact, it's all dark."
Matthew Danish wrote:
>Packages are (if I may summarize) name spaces, they are semantic, not
>>syntactic, entities, so I would argue that by using packages alone one
>>cannot alter either the lexical or syntatic structure of ACL.
>
>
> What is the difference between DEFUN and CL:DEFUN?
If any, semantic. They are not parsed any differently (they both match
the pattern for a symbol), (Unless the trailing ? is part of the
symbol). Depending on packaging, you might get a different symbol, but
you're gonna get a symbol (at read time) one way or the other. ;-) I
haven't run any experiments to see if one really can change ':', but
then I'd rather go from the spec. than any particular implemenations
interpretation of it.
>>Yes, reader macros can do both. Regular macros still have to live
>>within "the system", they are invoked either as symbols or as
>>functions, syntacticly.
>
>
> Regular macros are syntactic extensions, but at a different level than
> reader macros. (And they aren't invoked, but rather, expanded at some
> point before evaluation).
Invoked in the same sense that you "invoke" a function. You name it and
it does what it does (well, the system does it for you, but that's true
for function calls too). But this a point petite.
<D\'gou
On Tue, Aug 19, 2003 at 06:09:42PM -0400, Douglas Philips wrote:
> Matthew Danish wrote:
> >What is the difference between DEFUN and CL:DEFUN?
> If any, semantic.
And that's the incorrect reasoning I am trying to point out.
DEFUN, if it is not CL:DEFUN, could have any arbitrary syntax. How is
your analyzer supposed to understand the syntax which follows?
> >Regular macros are syntactic extensions, but at a different level than
> >reader macros. (And they aren't invoked, but rather, expanded at some
> >point before evaluation).
> Invoked in the same sense that you "invoke" a function. You name it and
> it does what it does (well, the system does it for you, but that's true
> for function calls too). But this a point petite.
Not really: the syntactic position of a macro is important. Macros
aren't first-class values like functions--there is no MACROCALL--they
are processed in a step of compilation completely before evaluation can
proceed. Macros can have their own second-level syntax, which your
analyzer must be able to contend with.
Without dealing with packages properly, your analyzer cannot understand
the meaning of a symbol such as DEFUN in the source code. What kind of
analysis are you doing that does not require differentiating between
symbols of different packages? Remember: Lisp code is built from
lists and atoms (including symbols), and not from character strings.
--
; Matthew Danish <·······@andrew.cmu.edu>
; OpenPGP public key: C24B6010 on keyring.debian.org
; Signed or encrypted mail welcome.
; "There is no dark side of the moon really; matter of fact, it's all dark."
Matthew Danish wrote:
> On Tue, Aug 19, 2003 at 06:09:42PM -0400, Douglas Philips wrote:
>
>>Matthew Danish wrote:
>>
>>>What is the difference between DEFUN and CL:DEFUN?
>>
>>If any, semantic.
> And that's the incorrect reasoning I am trying to point out.
>
> DEFUN, if it is not CL:DEFUN, could have any arbitrary syntax. How is
> your analyzer supposed to understand the syntax which follows?
Ahhhhh, because its a list? Whatever DEFUN "means" in this context is
the same everytime its spelt the same. Since I am analyzing code that is
not necessarily in context (the program(mer) might not have all the
pieces necessary to even execute a defpackage, human guidence is a
factor. a moot point if due to explicit package prefixes I cannot even
'read' the code in the first place.
I fully understand the point you are making. I think you are
overestimating the scope of my problem. I could always "preprocess" the
input to eliminte the package qualifiers (the more automated that
preprocessing, the more chance of changing a ":" at the wrong point. Of
course I could do it in lisp, oh, wait, no, because READ won't let me.
;-). Oh, but it might, reduce this to the original problem I had. ;-)
> Not really: the syntactic position of a macro is important. Macros
> aren't first-class values like functions--there is no MACROCALL--they
> are processed in a step of compilation completely before evaluation can
> proceed. Macros can have their own second-level syntax, which your
> analyzer must be able to contend with.
Which is part of the point of the human part of the analyzer. In any
event the defmacro for a macro may not be available to either the
program or the human, and so the human can decide how to cope with it.
> Without dealing with packages properly, your analyzer cannot understand
> the meaning of a symbol such as DEFUN in the source code. What kind of
> analysis are you doing that does not require differentiating between
> symbols of different packages? Remember: Lisp code is built from
> lists and atoms (including symbols), and not from character strings.
An analysis that has a significant human directed learning process.
OTOH, I can easily get away with DEFUN knowing that no sane programmer
is going to define their own in another package and then shadow it into
read code. But as a human directing the process, either the syntax will
remain the same because the redefined DEFUN is mucky semantics, or the
syntax will be different and either a program or a human can detect
this, or it doesn't matter.
As Barry pointed out in another message, getting the packaging
information right programmaticly could involve arbitrary eval-when
forms, and rather than fret over solving an NP complete problem (or
spending the time proving that it is or isn't NP complete), I'd rather
get something that is 80-95% automated and useful, the productivity
bonus is way worth it.
Hmmm, time for Zzzzzzzz, I hope this makes as much sense "out there" as
it does "in here in my head". ;-)
<D\'gou
Douglas Philips wrote:
> ...
> Actually, I want to look at the structure of the source. I don't _want_
> macros expanded, I want to analyze how they're called. Reader macros are
> a stickier wicket. I should have been more explicit in my original
> message that I want to analyze ANSI Common Lisp. Yes, ACL allows for
> reader macros, but the only ones I want to deal with are the ones
> defined by the standard. In any event, "user" reader macros are
> orthogonal to the built in ACL read's handling of packages...
if the goal is, given a lexical expression, to produce the same values in the
runtime's concrete model as the reader would, then one has to conform to the
semantics of the reader. isn't them more or less definitional?
>
> Packages are (if I may summarize) name spaces, they are semantic [entities],
for the reader
> [even if they remain]
> syntactic entities,
for the program read.
> so I would argue that by using packages alone one
> cannot alter either the lexical or syntatic structure of ACL.
no, not "acl" as such, and no, not the concrete syntax of the program, but
yes, one can alter the concrete model. that's the point of in-package.
> Yes,
> reader macros can do both. Regular macros still have to live within "the
> system", they are invoked either as symbols or as functions,
> syntacticly. Packages don't get their own read-tables, so I can't "jump"
> into a different syntax by just qualifying/prefixing a package name...
> foo:()are-now-regular-letters-in-the-foo-package
> isn't possible, at least, in my understanding of packages its not. ;-)
>
> So, forgoing "user" reader macros, ACL's read function gets me 99.99% of
> what I want to look at the structure of source code.
so long as you manage package state analogous to the reader.
> ...
>
> I suppose I could using a handler that would attempt a read, if it got a
> package error, make the package, and then re-attempt the read.
why would it not be sufficient to simply observe in-package operations?
> I would
> disable #. of course. ;-) Hmmm, hadn't thought of that one before. Its
> really messy since some streams are not seekable. Hmmm...
>
> There is a difference between "the code" and "the representation of the
> code presented to read" and it is the former, more than the latter,
> which I'm interesting in analyzing.
>
then, by problem definition, you have to produce the same results as the
reader. how can one do this without reproducing the reader's semantics?
...
james anderson wrote:
> if the goal is, given a lexical expression, to produce the same values in the
> runtime's concrete model as the reader would, then one has to conform to the
> semantics of the reader. isn't them more or less definitional?
Ah, it seems a slip of the keyboard/brain... I want to analyze the
source... I'm not trying to simulate or interpret it...
> why would it not be sufficient to simply observe in-package operations?
The issue is not in-package... its explicit reference to another named
package for which I might not even have source to analyze.
>>There is a difference between "the code" and "the representation of the
>>code presented to read" and it is the former, more than the latter,
>>which I'm interesting in analyzing.
I got that backwards... I am interested in analyzing the source
structure of the code... sigh... I hate when I make those kinds of
inversion mistakes. Sorry for the confusion!
<D\'gou
Douglas Philips wrote:
>
> james anderson wrote:
>
> > if the goal is, given a lexical expression, to produce the same values in the
> > runtime's concrete model as the reader would, then one has to conform to the
> > semantics of the reader. isn't them more or less definitional?
>
> Ah, it seems a slip of the keyboard/brain... I want to analyze the
> source... I'm not trying to simulate or interpret it...
that is, you are not concerned with cases where two lexically distinct package
markers were intended to name the same package. sorry, i missed that.
>
> > why would it not be sufficient to simply observe in-package operations?
>
> The issue is not in-package... its explicit reference to another named
> package for which I might not even have source to analyze.
that is, it does not matter that symbols are interned in the intended
packages. sorry, i missed that.
>
> >>There is a difference between "the code" and "the representation of the
> >>code presented to read" and it is the former, more than the latter,
> >>which I'm interesting in analyzing.
>
> I got that backwards... I am interested in analyzing the source
> structure of the code... sigh... I hate when I make those kinds of
> inversion mistakes. Sorry for the confusion!
>
oh.
i wonder if some of the difficulties arise from this perspective:
Douglas Philips wrote:
>
> Barry Margolin wrote:
> ...
> >
> > How does it "break the reader"?
>
> It causes the reader to signal a condition that lands me in the
> debugger. Syntacticly, "foo:bar", "bar", "zaphod:sym1" are all names of
> symbols. The "breakage" occurs because the reader attempts a semantic
> action, namely looking for (and failing to find) a package on what is a
> perfectly syntacticly valid form. ....
should one understood the toplevel/file-loader's handling of in-package to be
any more a "semantic act" that the printer's generation of package prefixes
with reference to the package currently bound to *package* or even - in
general - the recognition and invocation of reader macros. ', `, (, and " are
all macro characters in the standard syntax, but i'd never have thought of
them as having semantic implications beyond the transformation of a lexical
expression into a value in the runtime's concrete model. the way the reader
handles an expression for a symbol, such as "foo:bar", seems no different.
?
is the philips perspective an instance of tripping over the metacircularity in
the reader, or does the form 'foo:bar in the absence of a package definition
(nick)named foo with the external symbol bar really constitute a sematic error?
...
james anderson wrote:
> should one understood the toplevel/file-loader's handling of in-package to be
> any more a "semantic act" that the printer's generation of package prefixes
> with reference to the package currently bound to *package*
Indeed. Its really too bad the printer can't (he says ducking from
looking it up :-) ) be told to qualify _all_ symbols...
or even - in
> general - the recognition and invocation of reader macros.
Ah, now that _is_ a different thing... (User) Reader macros can
drasticly alter the syntax, and I admit/concede that point...
', `, (, and " are
> all macro characters in the standard syntax, but i'd never have thought of
> them as having semantic implications beyond the transformation of a lexical
> expression into a value in the runtime's concrete model. the way the reader
> handles an expression for a symbol, such as "foo:bar", seems no different.
> ?
Hmmm... ' versus QUOTE is something you can't "see" once the form has
been read. A kind of short hand...
( and " are structure indicating tokens (for lists and string literals).
` is, at read time, just another shortcut like QUOTE.
However "just" read-ing those has no semantic action. Yes actions are
taken but they alter the appearance of the form without regard to its
content. dee:aaa and qwerty:keys may or may not parse (by ACL's read)
depending on the content even though the form/syntax is valid. None of
the reader macros you mentioned, nor (I think) the rest of the syntax,
has this property.
> is the philips perspective an instance of tripping over the metacircularity in
> the reader, or does the form 'foo:bar in the absence of a package definition
> (nick)named foo with the external symbol bar really constitute a sematic error?
How could it not be semantic? The presence (or absence) of the package
"foo" doesn't alter the syntax of the expression... Just as the
presence/absence of "bar" within that package... The token foo:bar is
valid, how else could read get far enough to start looking it up?
(actually... 2.3.5 of the HyperSpec uses the term pattern, but its
pretty clear that it is syntax... ) Hmmmm... I also just noticed that it
says "the reader signals a correctable error"... I'm not sure if that is
sufficient for programmatic use, or if implementations are free to vary
in how the resumption works...
<D\'gou
Douglas Philips wrote:
> I've searched the archives (a treasure trove with decades of gems).
> I _think_ I understand packages...
> I'm sure I don't understand their relationship to the reader...(or visa
> versa)...
>
> I want to write a program that analyzes code.
> Aha, I can use 'read' instead of writing yet another Lisp parser...
> ....except that... any code I read that names other packages explicitly
> breaks the reader. If only the author(s) of the code-to-be-read had used
> use-package, or import... (then the symbols would be "unadorned"...) but
> they didn't and in general, can't be made to.
>
> What are my options?
> a) write my own reader function (uck, lots of redundant work).
> b) catch/handle the read error and try to continue.
> c) tweak the reader to do my own symbol reading.
> d) redefine intern or find-package?
> e) some other options???
>
(handler-case (read-from-string "'yabba::dabba-do")
(reader-error (c)
(make-package (caadr (simple-condition-format-arguments c)))))
kenny
[looking forward to the education he is about to get from the experts]
> (handler-case (read-from-string "'yabba::dabba-do")
> (reader-error (c)
> (make-package (caadr (simple-condition-format-arguments c)))))
I think I'd rather want to use package-error-package... but the real rub
is that the standard doesn't guarantee that I can "return back into the
reader" so that it can retry and resume parsing.... though I admit I
haven't memorized the standard yet... :-)
...D\'gou
Douglas Philips wrote:
> What are my options?
> a) write my own reader function (uck, lots of redundant work).
> b) catch/handle the read error and try to continue.
> c) tweak the reader to do my own symbol reading.
> d) redefine intern or find-package?
> e) some other options???
Belatedly, various respondents have already thrashed through the
problems. Dependnig on your needs, you might or might not need to
duplicate the forms read by the compiler during compilation. If
the kind of analysis you want to do depends only on the surface
syntax of the forms, you might get by with some of the proposed
hacks. They would, for instance, allow you to gather statistics
about extreme levels of parens nesting in lisp code. But they
would not be sufficient to study the frequency of non-null closures
vs. null closures.
But understanding code requires preserving the identity of symbols.
You need to know whether the expression "foo::bar" and "boo::bar"
resolve to the same operator. The simple attempts to make-package
foo on a reader error won't resolve this.
A different solution might use some system of advice on read inside
compile-file. Most systems provide smoe way to advise standard
functions, and if you can determine that compile-file calls read
and open or some similar identifiable functions, and if you actually
have enough of the target system's source code actually to build it,
you can build it and collect the read forms as a side effect of
building. Then you can analyze and walk forms to your heart's content.
This approach isn't portable, but I suspect it is implementable in
most real implementations.
Steven M. Haflich wrote:
> Douglas Philips wrote:
>> What are my options?
>> a) write my own reader function (uck, lots of redundant work).
>> b) catch/handle the read error and try to continue.
>> c) tweak the reader to do my own symbol reading.
>> d) redefine intern or find-package?
>> e) some other options???
>
> Belatedly, various respondents have already thrashed through the
> problems. Dependnig on your needs, you might or might not need to
> duplicate the forms read by the compiler during compilation. If
> the kind of analysis you want to do depends only on the surface
> syntax of the forms, you might get by with some of the proposed
> hacks. They would, for instance, allow you to gather statistics
> about extreme levels of parens nesting in lisp code. But they
> would not be sufficient to study the frequency of non-null closures
> vs. null closures.
>
> But understanding code requires preserving the identity of symbols.
Ah. Yes, there is the code, and there is the representation of the code
in ASCII/Unicode/etc. What I want to analyze is the structure in how the
code is represented. I was able to use the ACL "read" as a cheat until I
hit some code with explicit package qualifiers.
> You need to know whether the expression "foo::bar" and "boo::bar"
> resolve to the same operator. The simple attempts to make-package
> foo on a reader error won't resolve this.
Indeed. And the risk polluting/corrupting the system which is trying to
do the analysis. In addition to wanting to be able to analyze pieces of
the system, I also wish that the process of scanning/reading the code,
in and of itself, cannot risk affecting the analyzer (see the thread(s)
on how to signal a truly unique EOF for other cases of this).
> A different solution might use some system of advice on read inside
> compile-file....
> This approach isn't portable, but I suspect it is implementable in
> most real implementations.
Agreed. I could nab the sources to an open source ACL implementation and
change it myself, but I had hoped to be able to write this in ACl itself.
Its interesting to me that given all of Lisps reflection/reflexive
abilities, it (ACL at least) doesn't seem to be able to create a
"package jail" in which to load/read/analyze not-necessarily-trusted
code. I don't really fault ACL in particular, but it is to me a
fascinating thing.
Thanks for the additional ideas.
<D\'gou
P.S. Loading of code at run time to fix bugs aside, perhaps one reason
Java Applets took off was that issue of trust and isolation. Too bad we
can't get that portably with Lisp (I'm sure vendor extensions could be
made, even if (eq Vendor OpenSourceImplementation) ;-)
Douglas Philips wrote:
> Ah. Yes, there is the code, and there is the representation of the code
> in ASCII/Unicode/etc. What I want to analyze is the structure in how the
> code is represented. I was able to use the ACL "read" as a cheat until I
> hit some code with explicit package qualifiers.
If you care about the syntac of characters by which the program is
expressed -- including comments and failing #+ #- constructs -- then
you'd best write a grammar for S-expressions in yacc (for C programs) or
Zeru (for Lisp programs) and analyze away.
>> You need to know whether the expression "foo::bar" and "boo::bar"
>> resolve to the same operator. The simple attempts to make-package
>> foo on a reader error won't resolve this.
>
> Indeed. And the risk polluting/corrupting the system which is trying to
> do the analysis. In addition to wanting to be able to analyze pieces of
> the system, I also wish that the process of scanning/reading the code,
> in and of itself, cannot risk affecting the analyzer (see the thread(s)
> on how to signal a truly unique EOF for other cases of this).
If you put the analyzer in its own package, and delete the package before
starting the analysis, nothing will affect it. It is atypical for a
completed Lisp application to need to intern symbols on its own behalf --
the mapping of source code onto symbols is done by the reader when the
application (i.e. the analyzer itself) is compiled.
>> A different solution might use some system of advice on read inside
>> compile-file....
>
>> This approach isn't portable, but I suspect it is implementable in
>> most real implementations.
>
> Agreed. I could nab the sources to an open source ACL implementation and
> change it myself, but I had hoped to be able to write this in ACl itself.
ACL is not open source, although licensed customers can sign an additional
agreement to get most of the sources. But Franz doesn't encourage customers
messing with the sources because it makes support almost impossible.
However, having the sources would not solve the package problems you face.
You could indeed probably cause an unknown package error to give you the
information you need to identify the package, and to ptrovide a restart, but
unless you have successfully analyzed all preceding package mangling forms
(in this and earlier files) you won't know how to construct the missing
package.
> Its interesting to me that given all of Lisps reflection/reflexive
> abilities, it (ACL at least) doesn't seem to be able to create a
> "package jail" in which to load/read/analyze not-necessarily-trusted
> code. I don't really fault ACL in particular, but it is to me a
> fascinating thing.
If you want to control the interning of symbols by the reader, there is a
simple and portable way to do so.
The lisp reader has two phases. The first phase is under control of the
readtable and implements the syntax of the language, including lists and
strings and collecting tokens (ANS 2.2). The second phase is the
interpretation of tokens (ANS 2.3). A programmer cannot in any way alter
the syntax of interpretation of tokens, but the programmer _can_ prevent
the standard CL token interpretation from ever taking place.
What you do is construct your own readtable in which every constituent
character is given a macro dispatch that is your token collecting function.
It needs to duplicate the token-collecting behavior of ANS 2.2, but when a
complete token has been collected, it can examine the token and decide
whether there is anything interesting in it. In particular, it would be
interesting in any unexcaped #\: character. If it decides it doesn't need
to deal with the token specially, it can simply rebind *readtable* to the
standard readtable, and delegate to read-from-string of the collected
token. This hack gets tricky if the Lisp source being read is using a
nonstandard readtable, but you have already maintained elsewhere that you
need handle only standard Lisp readtable syntax.
To pop up to the global view, perhaps you would be more useful advice if
you explained what you are actually trying to do. Aside from claiming the
you want to analyze the expression of Lisp program source, you have kept
your intentions secret. This thread may therefore be beating around the
wrong bush.
Steven M. Haflich wrote:
> If you care about the syntac of characters by which the program is
> expressed -- including comments and failing #+ #- constructs -- then
> you'd best write a grammar for S-expressions in yacc (for C programs) or
> Zeru (for Lisp programs) and analyze away.
That is seeming more and more like the only way to go. Actually, I still
want to cheat and just have reader macros, but then I'm just using
readtables as my parsing control system. ;-)
> If you put the analyzer in its own package, and delete the package before
> starting the analysis, nothing will affect it.
Ah! So cool! I shoulda thought of that. Mustn't post when I'm on a
caffiene/sugar crash. (1/2 ;-) ).
Actually, (tangentially to my current project), could one do that to all
packages? Hmm, maybe I should start another thread about this,
if/when I would ever play with it. Creating an sandbox set of packages
with the "right names" so that code could be loaded, but having them be
"proxies" for the real packages (which have been deleted to protect
them). Well, deletion doesn't really mean to make the package invalid,
it more means to break the mapping from string name to package object,
right?
> ACL is not open source, although licensed customers can sign an additional
> agreement to get most of the sources.
Sorry, I was referring to ANSI Common Lisp, not the Franz product (nor
the Graham Book with the same name).
> The lisp reader has two phases. The first phase is under control of the
> readtable and implements the syntax of the language, including lists and
> strings and collecting tokens (ANS 2.2). The second phase is the
> interpretation of tokens (ANS 2.3). A programmer cannot in any way alter
> the syntax of interpretation of tokens, but the programmer _can_ prevent
> the standard CL token interpretation from ever taking place.
Agreed, though I would assert, pedanticly, that the second thing is a
semantic operation.
> What you do is construct your own readtable in which every constituent
> character is given a macro dispatch that is your token collecting function.
...
> This hack gets tricky if the Lisp source being read is using a
> nonstandard readtable, but you have already maintained elsewhere that you
> need handle only standard Lisp readtable syntax.
Agreed.
> To pop up to the global view, perhaps you would be more useful advice if
> you explained what you are actually trying to do. Aside from claiming the
> you want to analyze the expression of Lisp program source, you have kept
> your intentions secret. This thread may therefore be beating around the
> wrong bush.
My intentions aren't so much secret as boring. I want to analyze the
structure of ANSI Common Lisp source code. I have been, for lack of a
better word, smitten by Paul Graham's On Lisp and the power of
simplicity and terseness. Following in many footsteps (starting with
eXtreme Programming before i reacquainted with Lisp), I want to look at
how source changes as understanding (of the source and of the problem
domain) changes. I can't figure out how to articulate more about my
goals until I've started playing with the system. Meta-understanding, I
would suppose, is still embryonic (at least for me).
<D\'gou
Douglas Philips <····@mac.com> writes:
> Steven M. Haflich wrote:
>
> > ACL is not open source, although licensed customers can sign an additional
> > agreement to get most of the sources.
>
> Sorry, I was referring to ANSI Common Lisp, not the Franz product (nor
> the Graham Book with the same name).
I think in general it is a bad idea to use the term "ACL" in this
newsgroup, (except of course self-referentially, as I'm doing here),
since it means so many different things. Sometimes the context of
its usage makes it clear what is being referred to, but in this
particular case the context you had intended was not clear, and
many times one must read an article or its references several
times to get that context, which I'm sure the authors of those
articles had not intended.
If you want to refer to the various things that ACL means, use the
terms and abbreviations either established by practice or by
definition instead, and stay away from those abbreviations that
are ambiguous:
Use the following instead of ACL to mean:
- The product from Franz Inc: Use Allegro CL; that is its name. In
this newsgroup you can probably use just Allegro, although we were
not able to copyright that name because it is too general.
- The language: Use Common Lisp or CL, for short - the ANSI is
redundant. The only other uses for these terms are for pre-ansi
versions of CL, which can easily be referred to as CLtL1, CLtL2,
or simply "pre-ansi CL". Do not refer to Common Lisp as clisp,
since clisp is a specific vendor implementation of CL.
- The book: This one is hardest, because it has the longest
non-ambiguous titles. Also, the tendency is to shorten the
name in order to reduce typing (why do people do that? :-)
and we end up with the ambiguous form again later in the threads.
I would assume that "Graham's ACL book" or even "the ACL book" would
be sufficient, if those using the term would not shorten the term
and make it ambiguous again.
Thank you for your cooperation :-)
--
Duane Rettig ·····@franz.com Franz Inc. http://www.franz.com/
555 12th St., Suite 1450 http://www.555citycenter.com/
Oakland, Ca. 94607 Phone: (510) 452-2000; Fax: (510) 452-0182
Duane Rettig wrote:
> Douglas Philips <····@mac.com> writes:
>
>
>>Steven M. Haflich wrote:
>>
>>
>>>ACL is not open source, although licensed customers can sign an additional
>>>agreement to get most of the sources.
>>
>>Sorry, I was referring to ANSI Common Lisp, not the Franz product (nor
>>the Graham Book with the same name).
>
>
> I think in general it is a bad idea to use the term "ACL" in this
> newsgroup, (except of course self-referentially, as I'm doing here),
What is this, the stoning sketch from "The Life of Brian"?
> since it means so many different things.
Is there a non-Ansii Common Lisp!? Where is the Squad Squad when we need
them? (obscure William Safire reference)
Tell you what, fly out here and type out "llegro " in the middle every
time I want to say how great is ACL and you have a deal.
:)
--
kenny tilton
clinisys, inc
http://www.tilton-technology.com/
---------------------------------------------------------------
"Career highlights? I had two. I got an intentional walk from
Sandy Koufax and I got out of a rundown against the Mets."
-- Bob Uecker
Kenny Tilton <·······@nyc.rr.com> writes:
> >>Sorry, I was referring to ANSI Common Lisp, not the Franz product (nor
> >>the Graham Book with the same name).
>
> Is there a non-Ansii Common Lisp!? Where is the Squad Squad when we need
> them? (obscure William Safire reference)
>
> Tell you what, fly out here and type out "llegro " in the middle every
> time I want to say how great is ACL and you have a deal.
How about a comprimise aCL for the franz stuff ACL for the rest ;-)
cor
--
(setq reply-to (concatenate 'string "Cor Gest "" <cor"'(··@)"clsnet.nl>"))
OPEN-SOURCE BAR-B-QUE
OS-BBQ 6 september http://nl.linux.org/~nlobbq
On 25 Aug 2003, Cor Gest wrote:
> Kenny Tilton <·······@nyc.rr.com> writes:
>
> > >>Sorry, I was referring to ANSI Common Lisp, not the Franz product (nor
> > >>the Graham Book with the same name).
> >
> > Is there a non-Ansii Common Lisp!? Where is the Squad Squad when we need
> > them? (obscure William Safire reference)
> >
> > Tell you what, fly out here and type out "llegro " in the middle every
> > time I want to say how great is ACL and you have a deal.
>
> How about a comprimise aCL for the franz stuff ACL for the rest ;-)
alisp? Not to be confused with a generic lisp request + bad space bar, as
in "I want to use a lisp/alisp for the following project....what do you
recommend?" :)
Cheers,
Bijan Parsia.
Bijan Parsia <·······@email.unc.edu> writes:
> On 25 Aug 2003, Cor Gest wrote:
>
> > Kenny Tilton <·······@nyc.rr.com> writes:
> >
> > > >>Sorry, I was referring to ANSI Common Lisp, not the Franz product (nor
> > > >>the Graham Book with the same name).
> > >
> > > Is there a non-Ansii Common Lisp!? Where is the Squad Squad when we need
> > > them? (obscure William Safire reference)
> > >
> > > Tell you what, fly out here and type out "llegro " in the middle every
> > > time I want to say how great is ACL and you have a deal.
> >
> > How about a comprimise aCL for the franz stuff ACL for the rest ;-)
>
> alisp? Not to be confused with a generic lisp request + bad space bar, as
> in "I want to use a lisp/alisp for the following project....what do you
> recommend?" :)
OK, guys. I've always realized that when I bring this up I'm picking nits
that I have no hope of getting out, and that c.l.l. readers will all have
to always take the trouble to disambiguate, sometimes incorrectly. I'm
only hoping that newbies (or not) that hadn't realized the ambiguities
will be more careful to know what potential misunderstandings they might
create.
--
Duane Rettig ·····@franz.com Franz Inc. http://www.franz.com/
555 12th St., Suite 1450 http://www.555citycenter.com/
Oakland, Ca. 94607 Phone: (510) 452-2000; Fax: (510) 452-0182
Duane Rettig <·····@franz.com> wrote in
··················@beta.franz.com:
> OK, guys. I've always realized that when I bring this up I'm picking
> nits that I have no hope of getting out, and that c.l.l. readers will
> all have to always take the trouble to disambiguate, sometimes
> incorrectly. I'm only hoping that newbies (or not) that hadn't
> realized the ambiguities will be more careful to know what potential
> misunderstandings they might create.
>
As a newbie, I've always thought of ACL as the Allegro product (if that has
any influence). I just use CL to refer to Common Lisp (as the ANSI is
assumed, IMHO).
--
Best regards,
Jeff ··········@mfire.com
http://www.simforth.com
On 25 Aug 2003, Duane Rettig wrote:
[snip]
> OK, guys. I've always realized that when I bring this up I'm picking nits
> that I have no hope of getting out, and that c.l.l. readers will all have
> to always take the trouble to disambiguate, sometimes incorrectly.
Well, my alisp suggestion was at least semiserious. The worry that it'd
get confused with "a lisp" wasn't :)
> I'm
> only hoping that newbies (or not) that hadn't realized the ambiguities
> will be more careful to know what potential misunderstandings they might
> create.
It's a reasonable hope.
Cheers,
Bijan Parisa.
Duane Rettig wrote:
> Bijan Parsia <·······@email.unc.edu> writes:
>
>
>>On 25 Aug 2003, Cor Gest wrote:
>>
>>
>>>Kenny Tilton <·······@nyc.rr.com> writes:
>>>
>>>
>>>>>>Sorry, I was referring to ANSI Common Lisp, not the Franz product (nor
>>>>>>the Graham Book with the same name).
>>>>>
>>>>Is there a non-Ansii Common Lisp!? Where is the Squad Squad when we need
>>>>them? (obscure William Safire reference)
>>>>
>>>>Tell you what, fly out here and type out "llegro " in the middle every
>>>>time I want to say how great is ACL and you have a deal.
>>>
>>>How about a comprimise aCL for the franz stuff ACL for the rest ;-)
>>
>>alisp? Not to be confused with a generic lisp request + bad space bar, as
>>in "I want to use a lisp/alisp for the following project....what do you
>>recommend?" :)
>
>
> OK, guys. I've always realized that when I bring this up I'm picking nits
> that I have no hope of getting out, and that c.l.l. readers will all have
> to always take the trouble to disambiguate, sometimes incorrectly. I'm
> only hoping that newbies (or not) that hadn't realized the ambiguities
> will be more careful to know what potential misunderstandings they might
> create.
>
Hopefully my regular pronouncement of Franz, Inc.s Allegro Common Lisp:
http://www.franz.com/
... as having the best by far Common Lisp IDE, and hence the best IDE
period in the universe, makes up in some small way for my not doing my
part to build A L L E G R O mindshare every time I refer to the best
Common Lisp IDE in this or any parallel universe.
:)
PS: http://www.franz.com/
--
kenny tilton
clinisys, inc
http://www.tilton-technology.com/
---------------------------------------------------------------
"Career highlights? I had two. I got an intentional walk from
Sandy Koufax and I got out of a rundown against the Mets."
-- Bob Uecker
Kenny Tilton <·······@nyc.rr.com> writes:
> Duane Rettig wrote:
> > Bijan Parsia <·······@email.unc.edu> writes:
> >
>
> >>On 25 Aug 2003, Cor Gest wrote:
> >>
> >>
> >>>Kenny Tilton <·······@nyc.rr.com> writes:
> >>>
> >>>
> >>>>>>Sorry, I was referring to ANSI Common Lisp, not the Franz product (nor
> >>>>>>the Graham Book with the same name).
> >>>>>
> >>>>Is there a non-Ansii Common Lisp!? Where is the Squad Squad when we need
> >>>>them? (obscure William Safire reference)
> >>>>
> >>>>Tell you what, fly out here and type out "llegro " in the middle every
> >>>>time I want to say how great is ACL and you have a deal.
> >>>
> >>>How about a comprimise aCL for the franz stuff ACL for the rest ;-)
> >>
> >>alisp? Not to be confused with a generic lisp request + bad space bar, as
> >>in "I want to use a lisp/alisp for the following project....what do you
> >>recommend?" :)
> > OK, guys. I've always realized that when I bring this up I'm
> > picking nits
>
> > that I have no hope of getting out, and that c.l.l. readers will all have
> > to always take the trouble to disambiguate, sometimes incorrectly. I'm
> > only hoping that newbies (or not) that hadn't realized the ambiguities
> > will be more careful to know what potential misunderstandings they might
> > create.
> >
>
>
> Hopefully my regular pronouncement of Franz, Inc.s Allegro Common Lisp:
>
> http://www.franz.com/
>
> ... as having the best by far Common Lisp IDE, and hence the best IDE
> period in the universe, makes up in some small way for my not doing my
> part to build A L L E G R O mindshare every time I refer to the best
> Common Lisp IDE in this or any parallel universe.
Yes, Kenny, we do indeed appriciate your verbal support...
--
Duane Rettig ·····@franz.com Franz Inc. http://www.franz.com/
555 12th St., Suite 1450 http://www.555citycenter.com/
Oakland, Ca. 94607 Phone: (510) 452-2000; Fax: (510) 452-0182
"Kenny Tilton" <·······@nyc.rr.com> wrote in message
·····················@nyc.rr.com...
> Hopefully my regular pronouncement of Franz, Inc.s Allegro Common Lisp:
>
> http://www.franz.com/
>
> ... as having the best by far Common Lisp IDE, and hence the best IDE
> period in the universe, makes up in some small way for my not doing my
> part to build A L L E G R O mindshare every time I refer to the best
> Common Lisp IDE in this or any parallel universe.
>
> :)
>
> PS: http://www.franz.com/
I dunno, sounds like you're crossing the line to "paid shill," and thus
losing all credibility...
From: Christopher C. Stacy
Subject: Re: Term usage: ACL (was Re: read/packages....)
Date:
Message-ID: <uy8xhcsym.fsf@dtpq.com>
>>>>> On 25 Aug 2003 08:06:25 -0700, Duane Rettig ("Duane") writes:
Duane> OK, guys. I've always realized that when I bring this up I'm picking nits
Duane> that I have no hope of getting out, and that c.l.l. readers will all have
Duane> to always take the trouble to disambiguate, sometimes incorrectly. I'm
Duane> only hoping that newbies (or not) that hadn't realized the ambiguities
Duane> will be more careful to know what potential misunderstandings they might create.
The nomenclature should be in the various FAQs, at the top somewhere,
and should include just the few terms that Duane mentioned.
Well, maybe also mention the HyperSpec - so as to get that in
front of them right away. (More entries, or a complete glossary,
would be overwhelming up front. Just getting people to say CL and
instead of "clisp", and giving them the hint that CLtl is not the
ANSI spec, would be enough of a mission.)
Douglas Philips wrote:
> Ah. Yes, there is the code, and there is the representation of the code
> in ASCII/Unicode/etc. What I want to analyze is the structure in how the
> code is represented. I was able to use the ACL "read" as a cheat until I
> hit some code with explicit package qualifiers.
If you care about the syntac of characters by which the program is
expressed -- including comments and failing #+ #- constructs -- then
you'd best write a grammar for S-expressions in yacc (for C programs) or
Zeru (for Lisp programs) and analyze away.
>> You need to know whether the expression "foo::bar" and "boo::bar"
>> resolve to the same operator. The simple attempts to make-package
>> foo on a reader error won't resolve this.
>
> Indeed. And the risk polluting/corrupting the system which is trying to
> do the analysis. In addition to wanting to be able to analyze pieces of
> the system, I also wish that the process of scanning/reading the code,
> in and of itself, cannot risk affecting the analyzer (see the thread(s)
> on how to signal a truly unique EOF for other cases of this).
If you put the analyzer in its own package, and delete the package before
starting the analysis, nothing will affect it. It is atypical for a
completed Lisp application to need to intern symbols on its own behalf --
the mapping of source code onto symbols is done by the reader when the
application (i.e. the analyzer itself) is compiled.
>> A different solution might use some system of advice on read inside
>> compile-file....
>
>> This approach isn't portable, but I suspect it is implementable in
>> most real implementations.
>
> Agreed. I could nab the sources to an open source ACL implementation and
> change it myself, but I had hoped to be able to write this in ACl itself.
ACL is not open source, although licensed customers can sign an additional
agreement to get most of the sources. But Franz doesn't encourage customers
messing with the sources because it makes support almost impossible.
However, having the sources would not solve the package problems you face.
You could indeed probably cause an unknown package error to give you the
information you need to identify the package, and to ptrovide a restart, but
unless you have successfully analyzed all preceding package mangling forms
(in this and earlier files) you won't know how to construct the missing
package.
> Its interesting to me that given all of Lisps reflection/reflexive
> abilities, it (ACL at least) doesn't seem to be able to create a
> "package jail" in which to load/read/analyze not-necessarily-trusted
> code. I don't really fault ACL in particular, but it is to me a
> fascinating thing.
If you want to control the interning of symbols by the reader, there is a
simple and portable way to do so.
The lisp reader has two phases. The first phase is under control of the
readtable and implements the syntax of the language, including lists and
strings and collecting tokens (ANS 2.2). The second phase is the
interpretation of tokens (ANS 2.3). A programmer cannot in any way alter
the syntax of interpretation of tokens, but the programmer _can_ prevent
the standard CL token interpretation from ever taking place.
What you do is construct your own readtable in which every constituent
character is given a macro dispatch that is your token collecting function.
It needs to duplicate the token-collecting behavior of ANS 2.2, but when a
complete token has been collected, it can examine the token and decide
whether there is anything interesting in it. In particular, it would be
interesting in any unexcaped #\: character. If it decides it doesn't need
to deal with the token specially, it can simply rebind *readtable* to the
standard readtable, and delegate to read-from-string of the collected
token. This hack gets tricky if the Lisp source being read is using a
nonstandard readtable, but you have already maintained elsewhere that you
need handle only standard Lisp readtable syntax.
To pop up to the global view, perhaps you would be more useful advice if
you explained what you are actually trying to do. Aside from claiming the
you want to analyze the expression of Lisp program source, you have kept
your intentions secret. This thread may therefore be beating around the
wrong bush.
Douglas Philips <····@mac.com> writes:
> What are my options?
> a) write my own reader function (uck, lots of redundant work).
> b) catch/handle the read error and try to continue.
> c) tweak the reader to do my own symbol reading.
> d) redefine intern or find-package?
> e) some other options???
>
A nasty, but effective one:
1. Fork the lisp process with the PTRACE option. This is the
`slave' image.
2. Read the file into the slave image, let it modify what it
wants.
3. Once it is read, have the master probe the slave to analyze
it.
This is pretty gross, but the point is that you have a model of the
package system (in the slave) that works just like the real one
(because it is a real one), but doesn't pollute the real one (because
it isn't in this image!)
A variation on this is to launch a slave lisp to read the file and
have it print the contents in such a way that the symbol qualifiers
don't have colons in them.
Joe Marshall wrote:
> A nasty, but effective one:
...
> This is pretty gross, but the point is that you have a model of the
> package system (in the slave) that works just like the real one
> (because it is a real one), but doesn't pollute the real one (because
> it isn't in this image!)
Yes, it is rather, ah, brutish, but would be nearly bullet proof. He.
> A variation on this is to launch a slave lisp to read the file and
> have it print the contents in such a way that the symbol qualifiers
> don't have colons in them.
Well, I'd still have the same problem in the slave lisp, needing to make
reads of symbols with explicit package qualifiers work.
Given the other issues that have come up with analyzing the source
representation of the code, I think this is interesting but not tenable.
--- Not a reply to Joe, but sort of a reply to "why do this, why look at
'source' _code_ and not just _code?)
Why are macros interesting? If you look "at the code" you see billions
of PROGNs, LETs/LET*s, BLOCKs, and all kinds of other macro expanded
clutter. Its looking at the macro calls which create the higher level of
abstractions and increased code density which make Lisp a joy to program
in. Similarly for reader macros. What makes them powerful is how they
let you abstract, compress and simplify the code you look at, the code
you write. I don't care how "with-open-file' works under the cover, I
care that it means I have to write and read less code. I don't have to
remember to close the file, it happens. I don't have to see the file
being closed. I don't have to see the unwind-protect, all that is detail
which doesn't help me understand what the meat of the code is doing.
And, I want to write code that helps me look at the code (source code),
not the clutter and bustle/hustle of the behind-the-scenes stuff.
Yes, yes, I know, one needs to understand that to understand how to
write good code, good macros, that just isn't the focus of what I'm
looking into. ;-)
<D\'gou