From: Kent M Pitman
Subject: Re: Package: symbol reading woes
Date: 
Message-ID: <ufxwtgdhr.fsf@nhplace.com>
szergling <···············@gmail.com> writes:

> I have gradually evolved my thinking, trying out various method (shown
> below) with various degrees of flexibility/robustness until I gave up
> and settled on delayed-eval, completely inelegant, and hard to edit,
> since the code passed to it is not a sexp, but raw text.
> 
> ;; funcall a symbol
> (funcall
>  (eval
>   '(intern "MCOMMAND" :matlab-ffi-extensions))
>  "1 + 1")
> 
> (defun dfuncall (package name &rest args)
>   "Delayed symbol interning funcall. Useful for calling symbols whose
> package hasn't been loaded at read time."
>   (apply (intern name package) args))
> 
> (defun delayed-eval (code)
>   (eval (read-from-string code)))
> 
> ;; Example usage
> (lisp-fork
>  :sexp
>  '(progn
>    (blah blah)
> 
>    (use-package :matlab-ffi-extensions)
> 
>    (blah blah)
> 
>    (dfuncall
>     :matlab-ffi-extensions "MCOMMAND"
>     "~{addpath('~a');~}" *matlab-paths*)
> 
>    (blah blah)
> 
>    (delayed-eval
>     "(progn
>       (matlab-ffi-extensions:close-matlab-engine *matlab-engine*)
>       (setf *matlab-engine* nil)
>       )")
> 
>    (ext:saveinitmem "default.mem")))
> 
> I don't want to return to bash scripts to create Lisp images. Any
> ideas? Maybe I'll need to settle for eval?

I don't know if I understand your question.  Does this last not work?
You're asking for ideas about how to get it to work or how to do it 
better?

Given the constraints of what you're describing, if it all has to work
in one form, then this may be what you have to do. 

Your problem, it looks to me (I didn't execute your code and don't
have whatever Lisp version you're using, I suspect), is caused by
wanting to load a package definition in the same expression as you use
the symbols you'll get back.  The Lisp package system is not directly
intended to do this and you have to do a kind of dance like this to
protect the expression long enough to be able to read it and execute
the first forms.

The problem would vanish if you could send several forms.  [Or evaluate
them before the fork, if that's what's going on.  Are you trying not to
pollute your pre-fork environment?]

I'm not sure by what means you're getting dfuncall loaded; you show it
in a separate form in the example above [although it would work to
have it in the progn, too].  So apparently you can tolerate some pollution
of the pre-fork environment. If so, maybe you can just load the library.

Or maybe you're trying to get the latest version of the indicated
package when you start up.  In that case, maybe you can load a
compatible dummy version that just has enough symbols to get you
going.

So if you can execute forms before the lisp-fork that will fix a lot of
things.  If you can't, then I bet you can also do this, though I didn't
test it:

 (lisp-fork :sexp '(eval (read-from-string "...")))

For something like this I don't see any issue with using eval.  This is
the kind of gluish thing for which eval is needed.  (The only reason for
not using EVAL here would be if doing so was going to cause Lisp somehow
cause your lisp to think eval was now "in play" and could not be "gc'd out"
from some situation where it wanted to do that.  That's quite an 
implementation-specific issue, and might or might not come up, but I think
it's likely that it won't.)

You're almost surely using eval implicitly anyway to execute the
lisp-fork command anyway.  (You're not compiling the above code and then
executing it as a fasl file, are you?  Or if you are, you don't have to,
do you?  I'd be surprised if so.)

Does any of this help?

From: szergling
Subject: Re: Package: symbol reading woes
Date: 
Message-ID: <44aac699-5f20-4943-878a-fc5766b0a21a@1g2000hsl.googlegroups.com>
On Jan 20, 8:30 pm, Kent M Pitman <······@nhplace.com> wrote:
> szergling <···············@gmail.com> writes:
> > I have gradually evolved my thinking, trying out various method (shown
> > below) with various degrees of flexibility/robustness until I gave up
> > and settled on delayed-eval, completely inelegant, and hard to edit,
> > since the code passed to it is not a sexp, but raw text.
>
> > ;; funcall a symbol
> > (funcall
> >  (eval
> >   '(intern "MCOMMAND" :matlab-ffi-extensions))
> >  "1 + 1")
>
> > (defun dfuncall (package name &rest args)
> >   "Delayed symbol interning funcall. Useful for calling symbols whose
> > package hasn't been loaded at read time."
> >   (apply (intern name package) args))
>
> > (defun delayed-eval (code)
> >   (eval (read-from-string code)))
>
> > ;; Example usage
> > (lisp-fork
> >  :sexp
> >  '(progn
> >    (blah blah)
>
> >    (use-package :matlab-ffi-extensions)
>
> >    (blah blah)
>
> >    (dfuncall
> >     :matlab-ffi-extensions "MCOMMAND"
> >     "~{addpath('~a');~}" *matlab-paths*)
>
> >    (blah blah)
>
> >    (delayed-eval
> >     "(progn
> >       (matlab-ffi-extensions:close-matlab-engine *matlab-engine*)
> >       (setf *matlab-engine* nil)
> >       )")
>
> >    (ext:saveinitmem "default.mem")))
>
> > I don't want to return to bash scripts to create Lisp images. Any
> > ideas? Maybe I'll need to settle for eval?
>
> I don't know if I understand your question.  Does this last not work?
> You're asking for ideas about how to get it to work or how to do it
> better?
>
> Given the constraints of what you're describing, if it all has to work
> in one form, then this may be what you have to do.
>
> Your problem, it looks to me (I didn't execute your code and don't
> have whatever Lisp version you're using, I suspect), is caused by
> wanting to load a package definition in the same expression as you use
> the symbols you'll get back.  The Lisp package system is not directly
> intended to do this and you have to do a kind of dance like this to
> protect the expression long enough to be able to read it and execute
> the first forms.
>
> The problem would vanish if you could send several forms.  [Or evaluate
> them before the fork, if that's what's going on.  Are you trying not to
> pollute your pre-fork environment?]
>
> I'm not sure by what means you're getting dfuncall loaded; you show it
> in a separate form in the example above [although it would work to
> have it in the progn, too].  So apparently you can tolerate some pollution
> of the pre-fork environment. If so, maybe you can just load the library.
>
> Or maybe you're trying to get the latest version of the indicated
> package when you start up.  In that case, maybe you can load a
> compatible dummy version that just has enough symbols to get you
> going.
>
> So if you can execute forms before the lisp-fork that will fix a lot of
> things.  If you can't, then I bet you can also do this, though I didn't
> test it:
>
>  (lisp-fork :sexp '(eval (read-from-string "...")))
>
> For something like this I don't see any issue with using eval.  This is
> the kind of gluish thing for which eval is needed.  (The only reason for
> not using EVAL here would be if doing so was going to cause Lisp somehow
> cause your lisp to think eval was now "in play" and could not be "gc'd out"
> from some situation where it wanted to do that.  That's quite an
> implementation-specific issue, and might or might not come up, but I think
> it's likely that it won't.)
>
> You're almost surely using eval implicitly anyway to execute the
> lisp-fork command anyway.  (You're not compiling the above code and then
> executing it as a fasl file, are you?  Or if you are, you don't have to,
> do you?  I'd be surprised if so.)
>
> Does any of this help?
From: szergling
Subject: Re: Package: symbol reading woes
Date: 
Message-ID: <6a8d5754-ada7-4c04-bd9b-00a1ef9f7cb8@u10g2000prn.googlegroups.com>
Oopsies, please ignore previous post, it was posted in error.

> On Jan 20, 8:30 pm, Kent M Pitman <······@nhplace.com> wrote:
> > szergling <···············@gmail.com> writes:

[[ ... ]]

> > I don't know if I understand your question.  Does this last not work?
> > You're asking for ideas about how to get it to work or how to do it
> > better?

Apologies again for how vague the question was. It was more a
brainstorm request. Yes, it works, I just want something better. Eg,
can we arrange for

(package1:fun 1 2 3)

to read into a list of (forward-referenced-symbol number number
number), so that if package1 exists, the forward-reference works out
fine at evaluation time. Otherwise, an error is thrown. I guess it
could be called a "late-binding symbol-intern".

Maybe I am thinking about a macro to do:

(with-late-intern
 ("USE-PACKAGE" :package1)
 ("USE-PACKAGE" ":PACKAGE2") ;; can also do keywords
 ("PACKAGE1:FUN" 1 2 3)
 ("+" 1 "PACKAGE1:*VAR*"))

(may need to escape string data separately)

> Your problem, it looks to me (I didn't execute your code and don't
> have whatever Lisp version you're using, I suspect), is caused by
> wanting to load a package definition in the same expression as you
> use the symbols you'll get back.  The Lisp package system is not
> directly intended to do this and you have to do a kind of dance
> like this to protect the expression long enough to be able to read
> it and execute the first forms.

Yes, that is exactly the problem. Don't some Schemes 'solve' this
'transparently' with their module systems? I don't know.

> The problem would vanish if you could send several forms.  [Or
> evaluate them before the fork, if that's what's going on.  Are you
> trying not to pollute your pre-fork environment?]

I'm not too worried about pollution. I don't think I can send several
forms successively -- the only way to do that might be to use LOAD.

> I'm not sure by what means you're getting dfuncall loaded; you
> show it in a separate form in the example above [although it would
> work to have it in the progn, too].  So apparently you can
> tolerate some pollution of the pre-fork environment. If so, maybe
> you can just load the library.

I can use load, but if you imagine what I was trying to do, compare
typing everything at the REPL vs saving into file then LOAD. I was
looking for the least effort when typing out code. However, there
certainly is room to play with loading files, which currently looks to
be the most promising avenue to try.

> Or maybe you're trying to get the latest version of the indicated
> package when you start up.  In that case, maybe you can load a
> compatible dummy version that just has enough symbols to get you
> going.

Don't know, this sounds too involved for REPL work. The sexp sent to
lisp-fork is a way to document how an image was created, so I can make
new slightly different images when required. Ultimately, I was trying
to run code on a cluster at some useful level of abstraction. Eg, the
cluster can be pretty congested, so some way to have fine grained
job-submission is useful. I thought some generic way to deal with
multiple Lisps and module dependency tracking would be nice here.

> So if you can execute forms before the lisp-fork that will fix a
> lot of things.  If you can't, then I bet you can also do this,
> though I didn't test it:
>
>  (lisp-fork :sexp '(eval (read-from-string "...")))

Don't think I can do a single, whole, read-then-eval like this. It has
to contain multiple selectively-delayed eval-reads. Otherwise, it's no
different from the problem with "wanting to load a package definition
in the same expression as you use the symbols" (above).

Eg:

'(eval
  (read-from-string
   "(progn
     (use-package :package1)
     (package1:fun1 2 3))"))

won't work.

> For something like this I don't see any issue with using eval.
> This is the kind of gluish thing for which eval is needed.  (The
> only reason for not using EVAL here would be if doing so was going
> to cause Lisp somehow cause your lisp to think eval was now "in
> play" and could not be "gc'd out" from some situation where it
> wanted to do that.  That's quite an implementation-specific issue,
> and might or might not come up, but I think it's likely that it
> won't.)

Is this a tree-shaking issue? I'm not too worried about
executable/image size.

> You're almost surely using eval implicitly anyway to execute the
> lisp-fork command anyway.  (You're not compiling the above code
> and then executing it as a fasl file, are you?  Or if you are, you
> don't have to, do you?  I'd be surprised if so.)

In case it wasn't clear, lisp-fork is a "run-shell-command" hack, with
"shell arguments" passed in -- one of which is an input sexp to be
evaluated. I think you mean that eval is used implicitly in that the
forms are evaluated in a REPL: sbcl --eval, or clisp -x, etc.

> Does any of this help?

I'll try them, keep exploring, and see what's nicest. I'll probably
not have too much time to work my way out of this mess this time. It
will have to stay messy.

Thank you for trying :)
From: Pascal Bourguignon
Subject: Re: Package: symbol reading woes
Date: 
Message-ID: <87wsq44qlx.fsf@thalassa.informatimago.com>
szergling <···············@gmail.com> writes:
> Maybe I am thinking about a macro to do:
>
> (with-late-intern
>  ("USE-PACKAGE" :package1)
>  ("USE-PACKAGE" ":PACKAGE2") ;; can also do keywords
>  ("PACKAGE1:FUN" 1 2 3)
>  ("+" 1 "PACKAGE1:*VAR*"))


(defpackage "PACKAGE1" (:export "FUN" "*VAR*"))
(defpackage "PACKAGE2")
(progn
    (asdf:oos 'asdf:load-op :system1)
    (asdf:oos 'asdf:load-op :system2)   
    (package1:fun 1 2 3)
    (+ 1 package1:*var*))

But the sanest thing to do is still

(asdf:oos 'asdf:load-op :system1)
(asdf:oos 'asdf:load-op :system2)   
(progn
    (package1:fun 1 2 3)
    (+ 1 package1:*var*))

Otherwise, read this: http://www.nhplace.com/kent/PS/Ambitious.html

-- 
__Pascal Bourguignon__                     http://www.informatimago.com/
The mighty hunter
Returns with gifts of plump birds,
Your foot just squashed one.
From: Rainer Joswig
Subject: Re: Package: symbol reading woes
Date: 
Message-ID: <joswig-FA9247.14104120012008@news-europe.giganews.com>
In article <··············@thalassa.informatimago.com>,
 Pascal Bourguignon <···@informatimago.com> wrote:

> szergling <···············@gmail.com> writes:
> > Maybe I am thinking about a macro to do:
> >
> > (with-late-intern
> >  ("USE-PACKAGE" :package1)
> >  ("USE-PACKAGE" ":PACKAGE2") ;; can also do keywords
> >  ("PACKAGE1:FUN" 1 2 3)
> >  ("+" 1 "PACKAGE1:*VAR*"))
> 
> 
> (defpackage "PACKAGE1" (:export "FUN" "*VAR*"))
> (defpackage "PACKAGE2")
> (progn
>     (asdf:oos 'asdf:load-op :system1)
>     (asdf:oos 'asdf:load-op :system2)   
>     (package1:fun 1 2 3)
>     (+ 1 package1:*var*))
> 
> But the sanest thing to do is still
> 
> (asdf:oos 'asdf:load-op :system1)
> (asdf:oos 'asdf:load-op :system2)   
> (progn
>     (package1:fun 1 2 3)
>     (+ 1 package1:*var*))
> 
> Otherwise, read this: http://www.nhplace.com/kent/PS/Ambitious.html


I also use calling the function via finding the symbol based
on symbol and package.

But sometimes I wonder if it had been possible to
design Common Lisp, such that 'late binding' is also
possible where one has to name a symbol in a not-yet-existing
package.


Symbol and Package undefined:

? (compile-file "/Users/joswig/test.lisp")
Read error between positions 0 and 27 in /Users/joswig/test.lisp.
> Error: Reader error on #<BASIC-FILE-CHARACTER-INPUT-STREAM ("/Users/joswig/test.lisp"/9 ISO-8859-1) #x300041B1D80D>, near position 27, within "baz:f-baz ":
>        Reference to unknown package "P-BAZ".
> While executing: CCL::SIGNAL-READER-ERROR, in process Listener(79).
> Type :POP to abort, :R for a list of available restarts.
> Type :? for other options.
1> 

vs.   (symbol is just undefined, no package referenced)

? (compile-file "/Users/joswig/test.lisp")
;Compiler warnings for "/Users/joswig/test.lisp" :
;   Undefined function F-BAZ, in FOO.


If I stay in an existing package, I can write calls to functions
in my code, where the functions don't exist yet - even though there
was no symbol for them - just using the symbol introduces it.
But when the function is
in another package, I can't write package:symbol. Sometimes
I wish I could and those symbols get automagically 'linked' when
the package and the symbol are defined. 

Instead of (fboundp 'cool-software:do-magic) I have to write:

(and (find-package "COOL-SOFTWARE")
   (fboundp (find-symbol "DO-MAGIC" "COOL-SOFTWARE")))

Or define:

(defun sfboundp (symbol-name &optional (package *package*))
  (and (find-package package)
       (fboundp (find-symbol symbol-name package))))

(defun sfuncall (package symbol-name &rest args)
  (apply (find-symbol symbol-name package) args))


? (sfuncall "CL" "+" 1 2 3 4)
10

? (sfboundp "+" "CL")
#<Compiled-function + #x3000400DFD5F>
From: Kent M Pitman
Subject: Re: Package: symbol reading woes
Date: 
Message-ID: <uzlv0cp88.fsf@nhplace.com>
szergling <···············@gmail.com> writes:

> > Your problem, it looks to me (I didn't execute your code and don't
> > have whatever Lisp version you're using, I suspect), is caused by
> > wanting to load a package definition in the same expression as you
> > use the symbols you'll get back.  The Lisp package system is not
> > directly intended to do this and you have to do a kind of dance
> > like this to protect the expression long enough to be able to read
> > it and execute the first forms.
> 
> Yes, that is exactly the problem. Don't some Schemes 'solve' this
> 'transparently' with their module systems? I don't know.

Yes and no.

It does in the sense that it doesn't use symbols to separate environments,
it uses lexical environments.  So symbols are not an issue because the 
symbols used by different modules are not different symbols, and this means
there is no reader problem.

This approach however, does nothing to separate symbols as data.  If
you want a symbol FOO in your package to be different as data, not
program, from the symbol in another module, then you can't do that.
This turns out to be important if, for example, you want to do things
like attach properties to named symbols in mixable namespaces.

For example, consider that in Common Lisp you could have two packages, 
SMITH and JONES, each with a symbol called JOHN.  In CL, you can do
(SETF (GET 'SMITH::JOHN 'AGE) 7)
(SETF (GET 'JONES::JOHN 'AGE) 77)
You cannot do this in Scheme.

This was especially relevant to CL because a great many early AI
applications used this programming technique.  There are ways to avoid it,
but they give up some of the "symbolic programming" aspect of Lisp.

Even things like hygienic macros, which address this problem from
another angle, have the problem that you can't _just_ use symbols--you
have to make sure to have an elaborate namespacing component in the
form of modules and some form of painting technology or other way of
keeping the namespaces separate.  In effect, these implement packages,
but in a different way, and one that has far less easy-debugging
support for just naming things.  CL is very name-oriented; Scheme is
very binding-oriented.  This is a fundamental difference between the
two languages.

> > So if you can execute forms before the lisp-fork that will fix a
> > lot of things.  If you can't, then I bet you can also do this,
> > though I didn't test it:
> >
> >  (lisp-fork :sexp '(eval (read-from-string "...")))
> 
> Don't think I can do a single, whole, read-then-eval like this. It has
> to contain multiple selectively-delayed eval-reads. Otherwise, it's no
> different from the problem with "wanting to load a package definition
> in the same expression as you use the symbols" (above).

'(with-input-from-string (str "...")
   (loop for form = (read str nil nil)
         while form
         do (eval form)))


> > For something like this I don't see any issue with using eval.
> > This is the kind of gluish thing for which eval is needed.  (The
> > only reason for not using EVAL here would be if doing so was going
> > to cause Lisp somehow cause your lisp to think eval was now "in
> > play" and could not be "gc'd out" from some situation where it
> > wanted to do that.  That's quite an implementation-specific issue,
> > and might or might not come up, but I think it's likely that it
> > won't.)
> 
> Is this a tree-shaking issue? I'm not too worried about
> executable/image size.

Yes, tree-shaking.  I don't think it's relevant here anyway.  I mentioned
the issue only for completeness, so you'd know to think about it briefly in
other cases where it might matter.
From: George Neuner
Subject: Re: Package: symbol reading woes
Date: 
Message-ID: <vn88p31e7r6vr6ca0926ald3qkl9nduji9@4ax.com>
On 20 Jan 2008 13:43:19 -0500, Kent M Pitman <······@nhplace.com>
wrote:

>For example, consider that in Common Lisp you could have two packages, 
>SMITH and JONES, each with a symbol called JOHN.  In CL, you can do
>(SETF (GET 'SMITH::JOHN 'AGE) 7)
>(SETF (GET 'JONES::JOHN 'AGE) 77)
>You cannot do this in Scheme.

More specifically, you do it portably in Scheme unless you roll your
own namespace/environment support.

Native support for multiple environments isn't standard in Scheme, but
quite a number of implementations provide it in some form.  I'm
actually surprised that there isn't an SRFI for environments, but
Schemes with module systems would want to integrate them with
environments and since there is no agreement on module systems
(despite the R6RS library spec) I guess there is also no agreement on
evaluation environments.

To be clear: Scheme environments are NOT like CL packages - they have
very different semantics.  But simple namespace uses like the one
above can be easily supported, and using generalized setters (SRFI 17)
with some additional macrology, could even look much like the CL code.

George
--
for email reply remove "/" from address
From: Kent M Pitman
Subject: Re: Package: symbol reading woes
Date: 
Message-ID: <u8x2j4nd0.fsf@nhplace.com>
George Neuner <·········@/comcast.net> writes:

> On 20 Jan 2008 13:43:19 -0500, Kent M Pitman <······@nhplace.com>
> wrote:
> 
> >For example, consider that in Common Lisp you could have two packages, 
> >SMITH and JONES, each with a symbol called JOHN.  In CL, you can do
> >(SETF (GET 'SMITH::JOHN 'AGE) 7)
> >(SETF (GET 'JONES::JOHN 'AGE) 77)
> >You cannot do this in Scheme.
> 
> More specifically, you do it portably in Scheme unless you roll your
> own namespace/environment support.

The "this" I referred to was "use a native datatype in that way".
That is, the capability Lisp provides is "built-in support for attaching
properties on symbols in a way such that competing applications do not
clobber each other's data".  Scheme doesn't provide that capability.

Scheme is a Turing machine and is certainly capable of programming
around it.  But so is C, Fortran, and assembler.  So that's not an
interesting claim once you permit the writing of a program to solve
the problem.

> Native support for multiple environments isn't standard in Scheme, but
> quite a number of implementations provide it in some form.

There's a quote before the relevant symbols in (GET 'SMITH::JOHN
'AGE).  That means "environments" are not in play.  But packages are.
This is a FEATURE of Lisp, not an error in usage that everyone
(certainly me) would just as soon change.  It is an intentional and
important capability.

Scheme does not INTEND to solve this problem.  It is not an environment
problem.  It is a problem of separating symbol data in a way that is
orthogonal to environments.  Scheme symbols do not contain complex enough
information to even represent this problem.

> I'm
> actually surprised that there isn't an SRFI for environments, but
> Schemes with module systems would want to integrate them with
> environments and since there is no agreement on module systems
> (despite the R6RS library spec) I guess there is also no agreement on
> evaluation environments.

This isn't an issue of add-ons.  It's an issue of primitive
capability.  If I say "C doesn't have arbitrary precision arithmetic",
it does not fix the problem to say "someone should make a library that
offers such capability".  It would be nice and useful, but it would
still be true that anyone using ints did not get the capability--only
people who chose to use it did, and so if you gave the data to a
program that didn't realize it was supposed to overflow arithmetic
into your extra library, the loading of that library would still lead
to wrong effects.

The problem here with Scheme is that if you give symbols--the type that
is in the system already--to programs that want to make data associations
with those symbols and have them still be the data.  Scheme has made a
choice to make symbols not that rich.  They get certain advantages for 
that, but there are also costs.  It is not fair to tout the advantages
and handwave away the costs.

The analogy for CL users would be to say that in Lisp if we just added
modules, we could all program in the keyword package and would no longer
need or want other packages at all.  That simply isn't so.

> To be clear: Scheme environments are NOT like CL packages - they have
> very different semantics.  But simple namespace uses like the one
> above can be easily supported, and using generalized setters (SRFI 17)
> with some additional macrology, could even look much like the CL code.

To be clear: I wasn't talking about modules.  We apparently agree that
modules and packages are simply different, and I'm glad for that.  We
apparently agree that Scheme does not have packages.  My point was that
while modules may suffice for you to do what you think packages are for,
they do not for me.  Symbols are primitive to CL programmers in a way 
that Scheme thinks it can handwave away by substituting modules.  But
modules do not do what I want and would not be a substitute as an 
expressional capability.  Consequently, I underscore my previous point:
that this is one of the significant differences in mindset/intent between
Scheme and Lisp.

Among other things, it is the presence of packages and Lisp2, taken
together, that enable CL not to need a hygienic macro system.  If
Scheme had these two features, it could probably have a simpler macro
system.  But I think it prefers a more complicated macro system and a
simpler set of symbols.  I don't begrudge them that.  I don't
criticize it.  That's their choice.  It's a fine self-consistent
theory.  But it's different than the CL one.

Sorry if I sound a little heated.  I'm not intending to really, but
I'm not a good judge of that at the moment--I have a cold, actually,
and am not sleeping well, and it probably makes it harder for me to
have the patience to sit and edit clearly.  Please take the
firm/emphatic nature of my remarks here not as any kind of anger, but
rather as a desire to be clear on the fact that this issue is not
casual or insignificant to me.

If it helps, I will add that although I have programmed in Lisp for
now over 30 years, I only within the last 5 or so years came to a new
understanding of what symbols were really about and why they were
important.  I think they have cognitive significance that transcends
their status as a casual datatype.  It's something I hope to have a
few minutes to sit and write more about sometime.  But for now I'll
just say that the ability to name things has philosophical
significance that is not something I am willing to bargain away in
favor of something that "merely implements" equivalent behavior local
to a particular program under particular known and controlled
circumstances.  To repeat myself again from my prior post, CL is
name-oriented and Scheme is not.  This matters to me.
From: George Neuner
Subject: Re: Package: symbol reading woes
Date: 
Message-ID: <0rl8p3p10dv9jkkqjh8aneu6ognqhgf52t@4ax.com>
On 21 Jan 2008 03:03:07 -0500, Kent M Pitman <······@nhplace.com>
wrote:

>George Neuner <·········@/comcast.net> writes:
>
>> On 20 Jan 2008 13:43:19 -0500, Kent M Pitman <······@nhplace.com>
>> wrote:
>> 
>> >For example, consider that in Common Lisp you could have two packages, 
>> >SMITH and JONES, each with a symbol called JOHN.  In CL, you can do
>> >(SETF (GET 'SMITH::JOHN 'AGE) 7)
>> >(SETF (GET 'JONES::JOHN 'AGE) 77)
>> >You cannot do this in Scheme.
>> 
>> More specifically, you do it portably in Scheme unless you roll your
>> own namespace/environment support.

Obviously I meant to say "_can't_ do it portably".


>The "this" I referred to was "use a native datatype in that way".
>That is, the capability Lisp provides is "built-in support for attaching
>properties on symbols in a way such that competing applications do not
>clobber each other's data".  Scheme doesn't provide that capability.
>
>Scheme is a Turing machine and is certainly capable of programming
>around it.  But so is C, Fortran, and assembler.  So that's not an
>interesting claim once you permit the writing of a program to solve
>the problem.
>
>> Native support for multiple environments isn't standard in Scheme, but
>> quite a number of implementations provide it in some form.
>
>There's a quote before the relevant symbols in (GET 'SMITH::JOHN
>'AGE).  That means "environments" are not in play.  But packages are.
>This is a FEATURE of Lisp, not an error in usage that everyone
>(certainly me) would just as soon change.  It is an intentional and
>important capability.

I understand how the symbols were being used.  My point is simply that
you example is only of the same symbol having different attributes in
different namespaces.  That particular usage can be solved in Scheme
using environments (though other uses of packages maybe cannot be).

I'm sure you can quickly find another example that would be difficult
to do in Scheme just as I can find something in Scheme that is hard to
do in CL.  This particular example is not particularly hard to
replicate in Scheme.


>Scheme does not INTEND to solve this problem.  It is not an environment
>problem.  It is a problem of separating symbol data in a way that is
>orthogonal to environments.  Scheme symbols do not contain complex enough
>information to even represent this problem.

We're using different definitions of "environment" (see below).


>The problem here with Scheme is that if you give symbols--the type that
>is in the system already--to programs that want to make data associations
>with those symbols and have them still be the data.  Scheme has made a
>choice to make symbols not that rich.  They get certain advantages for 
>that, but there are also costs.  It is not fair to tout the advantages
>and handwave away the costs.


Not sure what you're trying to say here.  Scheme symbols are both data
themselves and references to data ... the only real difference between
CL and Scheme symbols is that CL symbols can bind multiple thingies
simultaneously.

I've never been persuaded that that is any kind of insurmountable
advantage.  Symbols as in either CL or Scheme simply don't exist in
most languages and, AFAIK, only Lisp derivatives offer multiple
binding symbols.  I personally dislike program punning, and for most
uses of binding multiple thingies to a single name, I would need a
more complex object than a CL symbol.  So Scheme symbols generally
serve me fine.


>The analogy for CL users would be to say that in Lisp if we just added
>modules, we could all program in the keyword package and would no longer
>need or want other packages at all.  That simply isn't so.

Now you've lost me.  Modules and packages are not interchangeable.
Scheme environments are similar to packages (see below) but modules
are distinctly different - and intended for a different purpose.


>> To be clear: Scheme environments are NOT like CL packages - they have
>> very different semantics.  But simple namespace uses like the one
>> above can be easily supported, and using generalized setters (SRFI 17)
>> with some additional macrology, could even look much like the CL code.
>
>To be clear: I wasn't talking about modules.  We apparently agree that
>modules and packages are simply different, and I'm glad for that.  We
>apparently agree that Scheme does not have packages.  My point was that
>while modules may suffice for you to do what you think packages are for,
>they do not for me.  Symbols are primitive to CL programmers in a way 
>that Scheme thinks it can handwave away by substituting modules.  But
>modules do not do what I want and would not be a substitute as an 
>expressional capability.  Consequently, I underscore my previous point:
>that this is one of the significant differences in mindset/intent between
>Scheme and Lisp.

Modules and packages have very little in common.

Scheme environments are first class lexical namespaces, more general
in application than CL's packages.  While there is no standard API for
environments, Schemes that offer them typically provide a complete set
of functions for manipulating them and their bindings.

Modules are independent compilation units.  They are dependent on, but
separate from, the notion of environments.  Only a few Schemes
currently have a module system because of thorny issues with macros
and separate compilation.  However, many Schemes offer environments in
some form.


>Among other things, it is the presence of packages and Lisp2, taken
>together, that enable CL not to need a hygienic macro system.  If
>Scheme had these two features, it could probably have a simpler macro
>system.  But I think it prefers a more complicated macro system and a
>simpler set of symbols.  I don't begrudge them that.  I don't
>criticize it.  That's their choice.  It's a fine self-consistent
>theory.  But it's different than the CL one.
>
>Sorry if I sound a little heated.  I'm not intending to really, but
>I'm not a good judge of that at the moment--I have a cold, actually,
>and am not sleeping well, and it probably makes it harder for me to
>have the patience to sit and edit clearly.  Please take the
>firm/emphatic nature of my remarks here not as any kind of anger, but
>rather as a desire to be clear on the fact that this issue is not
>casual or insignificant to me.

Sorry to hear you're not feeling well.


>If it helps, I will add that although I have programmed in Lisp for
>now over 30 years, I only within the last 5 or so years came to a new
>understanding of what symbols were really about and why they were
>important.  I think they have cognitive significance that transcends
>their status as a casual datatype.  It's something I hope to have a
>few minutes to sit and write more about sometime.  But for now I'll
>just say that the ability to name things has philosophical
>significance that is not something I am willing to bargain away in
>favor of something that "merely implements" equivalent behavior local
>to a particular program under particular known and controlled
>circumstances.  To repeat myself again from my prior post, CL is
>name-oriented and Scheme is not.  This matters to me.

That's fine.  I'm certainly not trying to convert you ... I'm just
injecting a little bit of information you may be unaware of.

George
--
for email reply remove "/" from address
From: Rainer Joswig
Subject: Re: Package: symbol reading woes
Date: 
Message-ID: <joswig-7D754B.12103321012008@news-europe.giganews.com>
In article <··································@4ax.com>,
 George Neuner <·········@/comcast.net> wrote:

...

> Not sure what you're trying to say here.  Scheme symbols are both data
> themselves and references to data ... the only real difference between
> CL and Scheme symbols is that CL symbols can bind multiple thingies
> simultaneously.

I'm not sure if that is (still) true. Variables in Scheme are
not really symbols and symbols are nothing more than
representation of strings.

http://www.r6rs.org/final/html/r6rs/r6rs-Z-H-4.html#node_sec_Temp_10

  A symbol is an object representing a string, the symbol's name.
  Unlike strings, two symbols whose names are spelled the same way are
  never distinguishable. Symbols are useful for many applications;
  for instance, they may be used the way enumerated values are used
  in other languages.

No word about references to data.



Compare that with:

http://lispm.dyndns.org/documentation/HyperSpec-7-0/HyperSpec/Body/t_symbol.htm

or

http://www.cs.cmu.edu/Groups/AI/html/cltl/clm/node27.html#SECTION00630000000000000000

...
From: Kent M Pitman
Subject: Re: Package: symbol reading woes
Date: 
Message-ID: <uprvv8f54.fsf@nhplace.com>
Rainer Joswig <······@lisp.de> writes:

> In article <··································@4ax.com>,
>  George Neuner <·········@/comcast.net> wrote:
> 
> ...
> 
> > Not sure what you're trying to say here.  Scheme symbols are both data
> > themselves and references to data ... the only real difference between
> > CL and Scheme symbols is that CL symbols can bind multiple thingies
> > simultaneously.
> 
> I'm not sure if that is (still) true. Variables in Scheme are
> not really symbols and symbols are nothing more than
> representation of strings.
> 
> http://www.r6rs.org/final/html/r6rs/r6rs-Z-H-4.html#node_sec_Temp_10
> 
>   A symbol is an object representing a string, the symbol's name.
>   Unlike strings, two symbols whose names are spelled the same way are
>   never distinguishable. Symbols are useful for many applications;
>   for instance, they may be used the way enumerated values are used
>   in other languages.
> 
> No word about references to data.

Moreover, symbols in Scheme don't have a variable namespace associated with
them in the way CL does.  In Scheme, the variable is in the environment,
and named by a symbol.  In CL, the variable IS the name, and cannot have
a different meaning in some other environment.  This is by design, and is
not the same.

Consequently, when George says "the only real difference between CL
and Scheme symbols is that CL symbols can bind multiple thingies
simultaneously", I assume he's saying that Scheme can bind one.  But
that's not the issue here. In CL, the symbol IS bound, not can be
bound.  It is bound twice.  In Scheme, it is not bound at all and
cannot be bound.  (Symbols don't have bindings in Scheme.  They name
bindings.  But only with respect to an environment.)

But also, importantly, the notion of property list is critical here.
Scheme has no property lists.  And a lot of this relates to that. You
can emulate some aspects of property lists with environments, but for
some kinds of purposes, Scheme symbols are not an adequate data
structure to replace some CL uses of symbols because they cannot
simultaneously exist, as data, side-by-side in the same environment,
with two different meanings.

Here's an excerpt of a longer post I started to write and didn't get to
sending that explains the basic issue:

 I don't think that problem can be solved in Scheme using environments
 without using some data structure other than symbols.  I think you're
 thinking that these symbols will stay in separate environments, but
 that isn't necessarily the case in CL.  The symbols SMITH::JOHN and
 JONES::JOHN can be used in the same program, in the same lexical
 environment, without confusion.  And can still be symbols.  The only
 way I know to do the separation in Scheme is to (naively) assume that
 the symbol JOHN can be used in both cases and to make an AGE function
 works on SMITH symbols and another AGE function (based on a hash
 table or some such) that works on JONES symbols ... or else to make a
 single AGE function that takes symbols with extra characters in their
 name; that is, what would in CL be called |SMITH::JOHN| and
 |JONES::JOHN| ... which means you have to anticipate the collision
 even for simple programs, or else rewrite your program when the
 collision occurs.  But if you call it just JOHN, no matter what you
 do, the symbol JOHN is going to be the symbol JOHN, and so it's going
 to have only one possible AGE.  You can use a non-symbol--for
 example, a record structure.  But then you can't do other things that
 CL people prefer to do, not the least of which is rely on the reader
 for interning and convenient display without a lot of custom
 programming.  Also, you can't rely on the felicity that two people
 each realizing that they will need such a non-symbol structure will
 use the same or compatible datatypes in order to implement their
 extension, so it is not a true protocol--it's just a non-general
 hack.

 The value to me is that when I'm in the debugger and I want to name a
 symbol, I can just type it.  From the same context, I can name the
 symbol SMITH::JOHN and JONES::JOHN and I can use them easily in one
 expression.  In Scheme, there is only the symbol JOHN and its
 interpretation is based on some sort of context, usually
 lexical. Using a symbol from another environment is meaningless
 because there is only the one symbol.  This is great from a "data
 hiding" theoretical point of view, and I know some people like it for
 that point of view.  But it's not good for what has sometimes been
 termed "symbolic programming", where you attach data to symbols
 themselves, not to environments, because collisions result.

 The package system in CL was added exactly because on big systems
 like the Lisp Machine and CL, when big programs that used symbolic
 programming were loaded, there was no way to get them to co-exist
 neatly without a massive rewriting effort of a deep nature.  It is
 not a simple syntactic change to make the change from packages to
 modules.  And, more importantly, it's not a change that is willingly
 accepted as a good change by all.

George says "So Scheme symbols generally serve me fine."  But the reason
I'm belaboring the matter is this summed up by this passage, again from
the other longer post I didn't send:

 The statement "I've never observed a need for x" is not a disproof
 that someone lse has observed a need.  By contrast, the statement "I
 have observed a need for x" can only be refuted by "You're an
 idiot/liar/whatever and so I don't believe you."  since it is
 otherwise an "existence proof".

 I am asserting I have a need for what CL does, and that CL does not
 offer what Scheme does.  You're responding by saying that you haven't
 got that need. I don't know what to do with that information if not
 to assume that either you aren't working from the same rules of
 inference that I am, or else that you're regarding me an
 idiot/liar/whatever per my explanation in the previous paragraph.

 My point is to document that this is something people value. And I
 feel like you're responding "Well, I don't see the value, so that is
 proof that there is no value."  I put my remarks on record so that
 you could read them and come to understand the value I and others
 perceive.  Are you trying to read what I have written and come to
 such an understanding? Or are you just trying to suggest that
 whatever I'm saying, it must surely be based on my lack of
 understanding of Scheme's better way.  That may not be what you
 intend, but it keeps being how I'm hearing things.  And its probably
 why my replies seem a bit annoyed.  I feel as if the responses are
 not acknowledging any legitimacy of my position, nor directly addressing
 the concrete examples I've raised to illustrate my point.

My remarks on this subject are the result of about 25+ years worth of
thought subsequent to my first considering this matter in 1982, when
Jonathan Rees and I were first designing T (Yale Scheme).  I was the
one that originated T's "locale" concept, and the design of the T
object system (which I also participated in heavily) was based on a
clear understanding the difference between how Scheme uses symbols and
how Lisp does.  And I've followed this issue over the years since then
in a variety of capacities in a variety of public and private forums.

I am not denying that there is a self-consistent, elegant, and useful
way that Scheme uses symbols. I've gone out of my way to say that all
of these qualities are things many people legitimately find in Scheme.
What I AM saying, though, is that Scheme doesn't have a unique lock on
any of these qualities.  There is self-consistency, elegance, and
usefulness in the way CL does things, too.  And I find it frustrating
when people don't do the legwork to understand that, and when they
speak dismissively about CL as if its features were just the trivial
result of a failure to understand Scheme's better way.

CL bit off a bigger problem than Scheme did with its symbol model.  So
it's easier to point at the rough edges because Scheme didn't even
confront the issues that CL went after.  A major reason (not the only
reason) I don't use Scheme in practice on a day-to-day basis is indeed
that for my own taste, Scheme did not do what I wanted to make things
feel comfortable for me.  Whereas CL did. 

I'd like to have that point at least acknowledged, even if having it
actively appreciated seems out of reach.
From: Waldek Hebisch
Subject: Re: Package: symbol reading woes
Date: 
Message-ID: <fn0sfc$rh6$1@z-news.pwr.wroc.pl>
szergling <···············@gmail.com> wrote:
> On Jan 20, 8:30 pm, Kent M Pitman <······@nhplace.com> wrote:
> > Your problem, it looks to me (I didn't execute your code and don't
> > have whatever Lisp version you're using, I suspect), is caused by
> > wanting to load a package definition in the same expression as you
> > use the symbols you'll get back.  The Lisp package system is not
> > directly intended to do this and you have to do a kind of dance
> > like this to protect the expression long enough to be able to read
> > it and execute the first forms.
> 
> Yes, that is exactly the problem. Don't some Schemes 'solve' this
> 'transparently' with their module systems? I don't know.

You can define and use package in the same expression if you
use read time evaluation like:

'(#.(make-package "FOO-PAK") foo-pak::bar foo-pak::baz)

-- 
                              Waldek Hebisch
·······@math.uni.wroc.pl 
From: Kent M Pitman
Subject: Re: Package: symbol reading woes
Date: 
Message-ID: <usl0rn50f.fsf@nhplace.com>
Waldek Hebisch <·······@math.uni.wroc.pl> writes:

> You can define and use package in the same expression if you
> use read time evaluation like:
> 
> '(#.(make-package "FOO-PAK") foo-pak::bar foo-pak::baz)

But if you do this, you might as well have just defined the package before
you executed the lisp-fork command in the example he provided.  The form you
show above is little different than first doing

 (make-package "FOO-PAK")

and then doing

 '(#.*package* foo-pack::bar foo-pack::baz)

The real question is why all that package loading stuff has to happen 
downstream of the fork at all.

Also, with respect to the issue of pre-loading a package with the relevant
symbols, you can in fact do that, and DEFPACKAGE has no problem upgrading 
the package later when it sees the new definition as long as it's compatible.
HOWEVER, the only hitch is that the loadup code for whatever library is
being loaded _could_ use reflection to ask (find-package "whatever") and to
not do essential setup a second time based on whether the package exists 
or not.  You have to know this detail about the nature of the implementation
of the code module you're loading to know whether it's ok to make a stub
package in advance of the real one.
From: Richard M Kreuter
Subject: Re: Package: symbol reading woes
Date: 
Message-ID: <87hch7njey.fsf@progn.net>
Kent M Pitman <······@nhplace.com> writes:
> Waldek Hebisch <·······@math.uni.wroc.pl> writes:
>
>> You can define and use package in the same expression if you
>> use read time evaluation like:
>> 
>> '(#.(make-package "FOO-PAK") foo-pak::bar foo-pak::baz)
>
> But if you do this, you might as well have just defined the package
> before you executed the lisp-fork command in the example he
> provided.  The form you show above is little different than first
> doing
>
>  (make-package "FOO-PAK")
>
> and then doing
>
>  '(#.*package* foo-pack::bar foo-pack::baz)
>
> The real question is why all that package loading stuff has to
> happen downstream of the fork at all.

Or why it's happening /upstream/ of the fork at all.  

By my way of thinking, the original problem's constraints seem to be
arranged for maximal inconvenience:

1. some program, P, creates a fresh Lisp instance, L, 

2. P supplies L some text, T, that must be readable in the standard
   syntax,

3. T must READ into exactly one S-expression, S,

4. L is to execute S using run-time bindings of symbols, and not EVAL,

5. no other state or access to resources can be assumed between P and
   L,

6. P must be a Lisp program,

7. T must be the printed representation of some S-expression E in P,

8. E must be printed using the standard Lisp printer,

9. E must be the result of P having called READ using a standard
   readtable on some input.

I think that /any/ of these constraints can be relaxed, most of the
inconveniences go away:

1. If P is a Lisp instance and doesn't have to create a fresh Lisp
   instance to do the work in, then the package namespace has to be
   set up only once.

2. If P and L can agree on some protocol other than standard CL
   syntax, then it can be possible to avoid reader errors when the
   package namespace isn't set up properly in L.

3. If T can be the printed representation of more than one
   S-expression, the problems go away.

4. If L can evaluate S using EVAL, then the idea of trying to defer
   bindings goes away.

5. If P can direct L to run code from a file or URL or process
   environment variable or database or directory or something, then
   these problems go away in approximately the same ways as in 3.

6. If P can be an arbitrarily dumb program in any language that just
   sends plain text to L, then the problems of setting up state in P
   before invoking L can go away, or at least be simpler than trying
   to coordinate two Lisp instances' package namespaces.

7. Even if P is a Lisp program, if E were something other than a tree
   of symbols, it would probably be simpler to set up the state in P
   to requisite for constructing E.

8. Even if E is a tree of symbols, if it can be printed with machinery
   other than the standard Lisp printer, then other data (e.g.,
   attributes on the plist) can be used to construct the text for L,
   and so the package namespace in P need not match that in L.

9. Even if P is a Lisp program that sends a printed representation of
   a complicated datum to L, if that datum can be constructed in P by
   means other than READ with a standard readtable, then the problems
   of setting up state in P before invoking L can be simplified.
   
So ISTM that while there's interesting things to discuss about the
package system, the OP's problem is really just the old joke:

Patient: "Doctor, it hurts when I do this."
Doctor: "Don't do that, then."

--
RmK
From: szergling
Subject: Re: Package: symbol reading woes
Date: 
Message-ID: <9a64726b-00f0-423b-bdee-cb6449f3d087@t1g2000pra.googlegroups.com>
One long post to reply and thank everyone:

Pascal:

I think I finally understood the "Ambitious Evaluation" article (third
time lucky). There were lots of other things going on, and I kept
going off on a tangent trying to frame things in terms of
continuations, (read-1-word actually gets restarted from the
beginning, and does not resume from a previously saved continuation).

Rainer seems to come closest to the same itch I'm feeling about the
early binding of symbols. Come to think of it, an autoload facility
(if I understand its function correctly) might be useful...

Kent:
Symbols in Scheme vs CL: I'm clueless. Will need to read up on the
interaction between modules, compilation/loading order, lexical
environments, etc.

Here's a meta-circular pseudo-code for the difference between cl and
Scheme plists (incomplete package handling). Would it be a rough
approximation of what you were saying?

(defclass cl-symbol ()
  ((name    :reader symbol-name)
   (package :reader symbol-package)
   (plist   :accessor symbol-plist)
   (variable-value :accessor symbol-value)
   (function-value :accessor symbol-function)))

(defun cl-get (sym indicator)
  (getf (symbol-plist sym) indicator))

(defparameter *cl-lexical-env* (...))
(defparameter *cl-dynamic-env* (...))

;; Scheme requires separate accounting of symbol properties. Now also
;; need another hash-table for *package*. In general one can add new
;; "external slots" to the scheme-symbol class (similar to *plist*).

(defclass scheme-symbol ()
  (name))

(defparameter *plist* (make-hash-table))

(defun scheme-get (symbol key)
  (getf (gethash symbol *plist1*) key))

(defparameter *scheme-lexical-env* (...))
(defparameter *scheme-dynamic-env* (...))


Waldek Hebisch:
> You can define and use package in the same expression if you
> use read time evaluation like:
>
> '(#.(make-package "FOO-PAK") foo-pak::bar foo-pak::baz)

As Kent Pitman has pointed out, I had to deal with 2 Lisps as well as
the reader here. In this case, read-time evaluation happens on the
host Lisp, not the slave. This should look more like:

"(#.(make-package \"FOO-PAK\") foo-pak::bar foo-pak::baz)", where I'm
sending a string instead of a sexp.

Also from KMP:
> The real question is why all that package loading stuff has to
> happen downstream of the fork at all.

Is there some confusion here about fork? When I called it lisp-fork
(in hindsight a bad name), it's not the unix fork() at all, but more
the English sense of starting another program. It's a new fresh Lisp
(and as noted before, possibly not on the same host).

Richard:
> By my way of thinking, the original problem's constraints seem to be
> arranged for maximal inconvenience:

I was looking for convenience at the REPL. There were alternatives,
but I thought maybe something like late symbol intern (which is a more
general problem) might be out there somewhere. I've seen some asdf
files of open source libs that delay interns using eval, so I see this
as a general issue.

> 1. some program, P, creates a fresh Lisp instance, L,
>
> 2. P supplies L some text, T, that must be readable in the standard
>    syntax,
>
> 3. T must READ into exactly one S-expression, S,
>
> 4. L is to execute S using run-time bindings of symbols, and not EVAL,
>
> 5. no other state or access to resources can be assumed between P and
>    L,
>
> 6. P must be a Lisp program,
>
> 7. T must be the printed representation of some S-expression E in P,
>
> 8. E must be printed using the standard Lisp printer,
>
> 9. E must be the result of P having called READ using a standard
>    readtable on some input.
>

Nice! This list is a very good way to collect my train of thought. My
thoughts on the list items:

1. some program, P, creates a fresh Lisp instance, L,

I don't know if I can do much about this.

3. T must READ into exactly one S-expression, S,

That's probably the one. Kent Pitman suggested reading multiple sexps
from a string & eval, which seems most convenient and general to me.

4. L is to execute S using run-time bindings of symbols, and not EVAL,

No, eval is ok. I tried a few examples (shown in original post).

5. no other state or access to resources can be assumed between P and
   L,

This is possible, but not as convenient as just C-xC-e on 1 sexp.

6. P must be a Lisp program,

Does this need explanation on comp.lang.lisp? Yeah, I was using shell
scripts + "cluster job submission commands" + Matlab, but one Lisp to
do it all (P, L, T, etc) is way better. The impedance mismatch between
languages/environments gets multiplied quite a few times on the system
I'm on, and is not very pleasant.

Incidentally, the quoting & eval-escaping mechanism of the scripting
languages (Bash in this case, for me) seem, to my inexperienced eyes,
way behind Lisp's quasi-quote in many ways (especially when I tried
nesting them). i.e they tend to have ` and , but not ' for fine
grained
control. What do you all think?

2, 8, 9, and to a certain extent, 7, about standard
syntax/read-tables/program representation: I haven't thought of
non-standard ways of doing things. Cannot really understand you, but
it sounds exotic. Things to ponder I guess.

Thanks again everyone.
From: Kent M Pitman
Subject: Re: Package: symbol reading woes
Date: 
Message-ID: <uk5lzda9w.fsf@nhplace.com>
szergling <···············@gmail.com> writes:

> Here's a meta-circular pseudo-code for the difference between cl and
> Scheme plists (incomplete package handling). Would it be a rough
> approximation of what you were saying?

Well, as an approximation? Sure.  You've neglected some details, of
course, so some might question your degree of approximation.  I think
you did capture some of the main elements I was getting at.  The following
remarks will hopefully clarify a few additional things...

> (defclass cl-symbol ()
>   ((name    :reader symbol-name)
>    (package :reader symbol-package)

The name and package here are only half the story--they affect
printing.  Reading is done by reverse structures that you have not
represented.  There is VERY APPROXIMATELY a hash table somewhere from
package names to packages, which are, in turn, and VERY APPROXIMATELY,
each hash tables from symbol names to instances of your CL-SYMBOL
above.

(There's also a side rant about gensyms in Scheme that I could get into 
 here but won't, for lack of time/energy.)

>    (plist   :accessor symbol-plist)
>    (variable-value :accessor symbol-value)
>    (function-value :accessor symbol-function)))

Whether these three items are "in" the symbol or just associated by
some other fast indexing mechanism is left to the implementation.
(This detail is important mainly because some people worry that
symbols might need to be allocated on read-only pages and still be
settable in some circumstances, so it's left to implementations to
work out that detail.  What really matters is that there is only one
possible association for each of these "attributes", not many, which
is implemented in your sketch by having each be part of the symbol
itself.)

And so, yes, the essential point of your sketch is right, which is
that there is such data and it is intrinsically part of the symbol.
That means having access to the symbol is having access to the plist
and to these places without having to scrounge.  By contrast, in the
picture you've sketched of scheme below, having access to the symbol
not only doesn't give you access to the variable and function value,
but there might be more than one such place such that the use of the
determiner "the" is improper; there can only be "a variable value" 
(we'll ignore the "function" value to not bring up the red herring of
Lisp1/Lisp2, which really doesn't play in here at all in other than
an incidental way).

> (defun cl-get (sym indicator)
>   (getf (symbol-plist sym) indicator))

Right.

And there's a minor occasional speed win here of direct access over hash
table access, but I'll skip belaboring that.  It works to CL's general
advantage that the plist is not a hash table indirection.

> (defparameter *cl-lexical-env* (...))
> (defparameter *cl-dynamic-env* (...))

There is no global lexical env at all for CL, though you can add one as
a user program.  Individual implementations are permitted to add them.
See http://www.nhplace.com/kent/CL/Issues/proclaim-lexical.html

There is a global dynamic environment, but it is not necessarily
represented this way.  There's a whole discussion about shallow and
deep binding I think I'll skip here and just make reference to.  If
those words don't mean anything to you--someone else can probably pick
up where I left off.  But given how you've named your variables, I'm
going to say that I think you've got an inconsistency if you think there
is something interesting to the user in *cl-dynamic-env*; what are you
imagining is in there?

> ;; Scheme requires separate accounting of symbol properties. Now also
> ;; need another hash-table for *package*. In general one can add new
> ;; "external slots" to the scheme-symbol class (similar to *plist*).
> 
> (defclass scheme-symbol ()
>   (name))
> 
> (defparameter *plist* (make-hash-table))
> 
> (defun scheme-get (symbol key)
>   (getf (gethash symbol *plist1*) key))

I assume that '1' is a typo.
 
> (defparameter *scheme-lexical-env* (...))
> (defparameter *scheme-dynamic-env* (...))

It's been a while since I looked but I'm not sure Scheme officially
takes a position on a global toplevel environment.  I think most Scheme
implementations offer what I'd call an "initial" one.  The difference
being that you could find yourself in a situation that was not the initial
one and not know how to name the root.

This is why I made the analogy to file systems.  In a file system you
get an initial working directory.  And that's often super-handy for
installing little collections of files that work together.  But
sometimes you are the user, not an installer, and sometimes you're in
your home directory and really want to look at /usr/dict/words or
/etc/hosts or something like that.  And you don't want to do it by 
relative means.  You just want to know its name.  

Scheme does not have that, and it is, by deliberate design, not easy
to add.  If you are cut off from the name of something, you're not
going to get there from here.  Or, put another way, you can say the
name but it won't have its relevance unless you have the associated
environment. In CL, if you can say the name of the symbol, you can
access the symbol's value.  And generally you can, because the package
system tries to make that easy.

As for the scheme dynamic (or sometimes "fluid") environment, I haven't
looked at this recently enough to say.  But the way in which you're
right is that the data is not in the symbol and I'm guessing that's just
a way of visually giving it a place to be, which I guess is sort of ok.

If you don't know about dynamic-wind in Scheme, btw, you might find it
interesting.  I'll spare you a summary of my opinions about it, since
that lately ends in disaster with some friends of mine, and I'm not up
to perturbing them tonight. :)