I think I found The Middle Way
http://paste.lisp.org/display/74832
merge-packages-and-reexport::! makes package dest which exports all
distinct external symbols of source packages. Warns on symbol clashes
unless they're put on dont-warn-on list.
Example:
(merge-packages-and-reexport::! :iterate+cl-utilities (:iterate :cl-
utilites) :dont-warn-on (:collect))
If you want more customized behaviour, it is good idea to make an
intermediate package
with merge then use it in your package and then reexport from
intermidiate package. E.g., if in
(defpackage :p1 (:use :cl) (:export :sym :s1 :nth))
(defpackage :p2 (:use :cl) (:export :sym :s2 :nth))
you want new package with p1:sym interned, do the following:
(merge-packages-and-reexport::! :intermediate (:p1 :p2) :dont-warn-on
(:sym))
; now we get :intermediate package with p1:s1, p2:s2 and cl:nth
exported
(defpackage :p1+p2-with-sym-from-p1
(:use :intermediate :cl)
(:import-from :p1 :sym)
(:export :sym))
(merge-packages-and-reexport::reexport :intermediate :p1+p2-with-sym-
from-p1))
; reexport allows to export all external symbols which are
; found in :p1+p2-with-sym-from-p1 and which are external
in :intermediate
So, we get sql-like namespace merging behavior. It is cool, as it
allows us to split our
functionality in a large number of small packages. We can combinate
them freely
with using just merge-packages-and-reexport::! It is rather safe -
neither of clashing
symbols are exported and warnings are issued on clashes. If we want to
get rid of that
warnings, there is :dont-warn-on clause.
As application scales up and new symbols are exported from packages it
comprises, new
clashes can occur. In this case we get clash warnings again. If some
new symbol clashes
with some old, we need just to qualify old symbol with its package
prefix everywhere in the source
and then we can add its name to :dont-warn-on.
If symbol moves from package to package in course of refactoring, it
is likely that nothing is be done.
Consider we have moved :s2 from :p2 to :p1 and thus changed our
packages in the example above to
(defpackage :p1 (:use :cl) (:export :sym :s2 :s1 :nth))
(defpackage :p2 (:use :cl) (:export :sym :nth))
Then intermediate package won't change at all and it gives us a
freedom to re-organize packages in our
application. This is very important when application scales.
If we want to use full power of lisp defpackage form, we can do so
with use-package and merge-packages-and-reexport::reexport as example
states.
So, this solution is likely to fit to the following requirements when
compared to ordinary defpackaging practice:
- simplicity. SQL namespace merging policy is very simple and I follow
it
- better support for modularity. Split your app into multiple modules
freely just as SQL database is split to hundreds of tables
- better support for scalability. You can use any package in your
package and you don't need to
solve symbol clashes. If symbols clash, they're not imported so
qualify them always. If new clashes occur, compiler would inform you
and you won't err.
- lightweight implementation. no changes to lisp reader or standard
lisp functions/macros/special forms
- compatibility. You can still use full defpackage power if you follow
some style.
Is this solution ideal? No. You can still write :sym in :intermediate
package and get junky symbol. You would think
that it is from p1 or from p2, and it will be wrong. You can do the
same in any common lisp package though.
But common lisp does not allow to do any protection here without using
some at implementation-specific tweaks. For some implementations, it
seems to be impossible at all without modifying a reader. So, Middle
Way approach here is to do what is relatively easy to do.
I didn't practiced this solution yet, but I hope it is ok. Maybe there
are some pitfalls I didn't notice.
Comments/Critics are welcome.
What about TFEB conduit packages? http://www.tfeb.org/lisp/hax.html
Cheers
--
Marco
On Feb 4, 12:58 pm, budden <···········@mail.ru> wrote:
> I think I found The Middle Way
>
> http://paste.lisp.org/display/74832
>
> merge-packages-and-reexport::! makes package dest which exports all
> distinct external symbols of source packages. Warns on symbol clashes
> unless they're put on dont-warn-on list.
> Example:
>
> (merge-packages-and-reexport::! :iterate+cl-utilities (:iterate :cl-
> utilites) :dont-warn-on (:collect))
>
> If you want more customized behaviour, it is good idea to make an
> intermediate package
> with merge then use it in your package and then reexport from
> intermidiate package. E.g., if in
>
> (defpackage :p1 (:use :cl) (:export :sym :s1 :nth))
> (defpackage :p2 (:use :cl) (:export :sym :s2 :nth))
>
> you want new package with p1:sym interned, do the following:
>
> (merge-packages-and-reexport::! :intermediate (:p1 :p2) :dont-warn-on
> (:sym))
> ; now we get :intermediate package with p1:s1, p2:s2 and cl:nth
> exported
>
> (defpackage :p1+p2-with-sym-from-p1
> (:use :intermediate :cl)
> (:import-from :p1 :sym)
> (:export :sym))
>
> (merge-packages-and-reexport::reexport :intermediate :p1+p2-with-sym-
> from-p1))
> ; reexport allows to export all external symbols which are
> ; found in :p1+p2-with-sym-from-p1 and which are external
> in :intermediate
>
> So, we get sql-like namespace merging behavior. It is cool, as it
> allows us to split our
> functionality in a large number of small packages. We can combinate
> them freely
> with using just merge-packages-and-reexport::! It is rather safe -
> neither of clashing
> symbols are exported and warnings are issued on clashes. If we want to
> get rid of that
> warnings, there is :dont-warn-on clause.
>
> As application scales up and new symbols are exported from packages it
> comprises, new
> clashes can occur. In this case we get clash warnings again. If some
> new symbol clashes
> with some old, we need just to qualify old symbol with its package
> prefix everywhere in the source
> and then we can add its name to :dont-warn-on.
>
> If symbol moves from package to package in course of refactoring, it
> is likely that nothing is be done.
> Consider we have moved :s2 from :p2 to :p1 and thus changed our
> packages in the example above to
>
> (defpackage :p1 (:use :cl) (:export :sym :s2 :s1 :nth))
> (defpackage :p2 (:use :cl) (:export :sym :nth))
>
> Then intermediate package won't change at all and it gives us a
> freedom to re-organize packages in our
> application. This is very important when application scales.
>
> If we want to use full power of lisp defpackage form, we can do so
> with use-package and merge-packages-and-reexport::reexport as example
> states.
>
> So, this solution is likely to fit to the following requirements when
> compared to ordinary defpackaging practice:
> - simplicity. SQL namespace merging policy is very simple and I follow
> it
> - better support for modularity. Split your app into multiple modules
> freely just as SQL database is split to hundreds of tables
> - better support for scalability. You can use any package in your
> package and you don't need to
> solve symbol clashes. If symbols clash, they're not imported so
> qualify them always. If new clashes occur, compiler would inform you
> and you won't err.
> - lightweight implementation. no changes to lisp reader or standard
> lisp functions/macros/special forms
> - compatibility. You can still use full defpackage power if you follow
> some style.
>
> Is this solution ideal? No. You can still write :sym in :intermediate
> package and get junky symbol. You would think
> that it is from p1 or from p2, and it will be wrong. You can do the
> same in any common lisp package though.
>
> But common lisp does not allow to do any protection here without using
> some at implementation-specific tweaks. For some implementations, it
> seems to be impossible at all without modifying a reader. So, Middle
> Way approach here is to do what is relatively easy to do.
>
> I didn't practiced this solution yet, but I hope it is ok. Maybe there
> are some pitfalls I didn't notice.
>
> Comments/Critics are welcome.
I've tried to use hierarchical packages and conduit packages by Tim
before.
First of all, I've glanced (again) through Conduts definition and I
still don't see conduits offer just the semantics I intended to
implement.
My intention was to introduce some clean, simple and reasonable
_policy_ for automated resolving of clashes. Idea is borrowed from
SQL.
Contrary to that, Conduits is a layer around defpackage form which
does not introduce any policy, but only makes easier to do standard CL
practices
by adding some shorthands for common cases where manual package
management is painful.
I hope that the policy I introduce would be good enough to allow one
not to use manual package management at all. That is,
I hope that no explicit calls to import/export/shadowing-import would
be needed most of the time. SQL lacks those primitives, but it
is rather convinient to use. My macros are not as powerful as package
system - I do not redefine primitives. So if you export new symbol
you need to redefine all dependent packages you defined with my
macros. But in general, if you have projects and build system, you'll
do that anyway,
even if you use more dynamic package definition facilities.
Second, redefining any standard from creates a potential problem with
libraries and in a collective work. You should convince everyone to
put your favorite lisp extensions to their initialization files. Bugs/
limitations/incompatibilities/non-intuitive behaviour in extensions (I
do not
say concretely about TFEB ones now) can cause trouble.
Third, minimalism is one thing I've learned here at c.l.l. It is
rather easy in lisp to redefine everything, but
good style is to make only a minimal changes. It is better to do a
minimal injection of a vacccine instead of redefining
basic constructs.
Nice thing in Tim Bradshaws' code is per-package aliases (sql features
per statement aliases which are handy), but they depend on
non-portable hierarchical packages and they would interfere with
developers IDE some of which are closed-source. We have 'symbol-
macrolet,
after all.
On Feb 4, 2:09 pm, budden <···········@mail.ru> wrote:
> My intention was to introduce some clean, simple and reasonable
> _policy_ for automated resolving of clashes.
My stuff had no intention of doing that (I think that doing that is a
really bad idea actually), so it's not surprising you didn;t find it.
> SQL lacks those primitives, but it
> is rather convinient to use.
Some might differ about that.
>
> Second, redefining any standard from creates a potential problem with
> libraries and in a collective work.
conduits doesn't do that. It creates *another* macro called
DEFPACKAGE, which lives in a different package to the standard macro.
Code that uses conduits can coexist with code that does not.
>
> Nice thing in Tim Bradshaws' code is per-package aliases (sql features
> per statement aliases which are handy), but they depend on
> non-portable hierarchical packages and they would interfere with
> developers IDE some of which are closed-source. We have 'symbol-
> macrolet,
> after all.
Hierarchical packages, on the other hand, *does* modify bits of CL.
Symbol macros are not at all the same thing, I am not sure why you
think they are.
Hallo, Tim!
> It creates *another* macro called
> DEFPACKAGE, which lives in a different package to the standard macro.
Ok, maybe I'm wrong. As I remember, conduits are dynamic - they
propagate changes through package dependency graph. If some package
was defined with standard defpackage form and then I define a conduit
in terms of this package, would it notice changes to package on which
it depends? I guess not. If I am right, the best way to go is to use
conduits everywhere.
> My stuff had no intention of doing that (I think that doing that is a
really bad idea actually), so it's not surprising you didn;t find it.
Thanks for the confirmation. What about is this a good idea or not, I
have had some dispute here about it before and no-one was convinced
finally. So let me avoid this discussion :)
> Symbol macros are not at all the same thing, I am not sure why you
think they are.
I do not think they are :) Initial problem is long package names.
There is a limited amount of short nicknames available, so we want to
find some other way to abbreviate. Symbol-macrolet is another way to
abbreviate. We might even define a set of lexically bound
abbreviations with a macro.
BTW, if you're interested in a way to make them portable, I was
working in this direction. I found no simpler solution then redefining
entire reader and delegating reading process to this reader with heavy
patching of readtable. It looks like this can be done in a portable
way. I used Pascal Bourguignon's GNU reader, but it is GNU and it has
bugs. I also adapted some parts of SBCL reader for that, but it has
problems with dotted pairs. CCL reader might be better, but I have no
more resources for that. And it looks like one needs to modify printer
and SLIME for this solution to be finished. I didn't even investigated
if it can be done and how. Initial state of what I've done is here,
and it gives only some hints what to do. I do not publish redirecting
to Pascal's reader as I'm afraid of GNU.
http://paste.lisp.org/display/70334
On Feb 4, 8:36 pm, budden <···········@mail.ru> wrote:
>
> Ok, maybe I'm wrong. As I remember, conduits are dynamic - they
> propagate changes through package dependency graph. If some package
> was defined with standard defpackage form and then I define a conduit
> in terms of this package, would it notice changes to package on which
> it depends? I guess not. If I am right, the best way to go is to use
> conduits everywhere.
No, they're not dynamic. Once you've defined one its just a package.
I think (but I forget) that redefining one will pick up changes
(actually, it really should!). But all the whole thing was was a way
of making various operations on packages easier - really, what I
wanted to be able to do was to say "given all these libraries with
their own packages, make a single package which exports *all* the
external symbols of the libraries, specifying how to deal with any
clashes".
> I do not think they are :) Initial problem is long package names.
> There is a limited amount of short nicknames available, so we want to
> find some other way to abbreviate. Symbol-macrolet is another way to
> abbreviate. We might even define a set of lexically bound
> abbreviations with a macro.
Oh, yes, I see that.
Hi!
> No, they're not dynamic. Once you've defined one its just a package.
I took a new look at conduits source and I see it defines 'export this
way:
(defun export (symbol/s &optional (package *package*))
(prog1
(cl:export symbol/s package)
(recompute-conduits-for package)))
For me, this does mean they're trying to be dynamic and emulate cl
package system to some extent.
> But all the whole thing was was a way
> of making various operations on packages easier - really, what I
> wanted to be able to do was to say "given all these libraries with
> their own packages, make a single package which exports *all* the
> external symbols of the libraries, specifying how to deal with any
> clashes".
It is very likely that first of all you would reexport every non-
clashing symbols of all your libraries.
Then you'd want to know which symbols clash. Third, you'd like to
manage clashes manually in advance instead
of breaking to debugger and to save your work as a package definition.
This is exactly what you get with my macro.
1. It reexports all non-clashing symbols from "merged" package.
2. It prints all clashes in one warning which you can disable per
symbol. You read it and make a decision.
3. Then you define new package in terms of merged package, importing
desired of clashing symbols and exporting them.
4. You can also use reexport to export all symbols you have imported
from some package without enumerating them.
But I don't see how I'll do that with conduits. Package itself can be
defined with :EXTENDS/EXCLUDING, but first of all you need to find all
the clashes. Maybe you don't want to resolve conflicts with my default
policy, but, at least, my macro does this for you in a convinient way.
So you can merge multiple packages and not being afraid of dozens of
clashes.
I've found one more useful fix to package system.
Some implementations allow to modify find-symbol/intern behaviour.
I didn't still test it thoroughly, but it looks like it is possible
in
lispworks. Also it looks like it is possible in ccl (no source
patching is
needed). Sbcl still fails to work after function modification, clisp
seems
to ignore redefinitions completely.
Problems often arise from trash symbols.
E.g. you type in a REPL:
(in-package :p1)
's1
(in-package :p2)
's1 ; intended to mean p1:s1
If you forgot to import p1::s1 to p2, you
will get trash symbol p2::s1. If this happen
in a context of asdf system being refactored, it
is rather hard to diagnose where this
trash symbol comes from and this is an obstacle
for refactoring.
Suggested solution is to have a list of symbol-names to
care of and make lisp warn you when you intern the symbol to
a wrong package (or to any package).
I have some solution for that which does work in a lispworks and
which helped me already to refactor my code, but this solution is
not mature yet so I'd like to hear opinions about the idea. Maybe
some suggestions/extensions, comments about portability, etc.
Hi!
Now it looks that I succeed in modifying reader in a way which
allows to intersect package system access from reader portably. So,
there is only one step to having portable and flexible package locks
and hierarchical package.
I have made some testing under lispworks, sbcl and allegro. It is
early
to say all is ok, but at least I have loaded alexandria, cffi and
clsql
with a success.
I think it would also be interesting to create custom symbol
parsers.
E.g. to expand $a to (dollar a) and a.b.c to (dots a b c) under some
circumstances. This is not an ideal thing, as it might cause problems
with
some IDEs and make printed representation to look very different from
input.
Last month I have done some work to prepare my code for publication,
but
it is not finished yet.
Hi list!
Unfortunately, portable reader is not ready for publication still.
But design goes on too.
What do you think about some extensions to token parser function. I
can suggest some extensions
which would be possible to do with our reader.
1. Date reader. One could give a sence to tokens like 2009.04.02 to
return a date.
Any ideas about date format? I think the simplest one is yyyy.mm.dd.
It's advantage
is that it is sortable as string. Is it reasonable to have a date
format controlled
by a variable to allow yy.mm.dd or dd.mm.yy etc? I incline to disallow
it.
2. How to add a time reader? It could be, say, [yyyy.mm.dd:]hh:mm[:ss
[:msms]][+nn] where nn
is a timezone spec and brackets delimit an optional parts. Ideas/
suggestions are welcome.
3. Macro-characters scale poorly. I think I could make
my-package::<anything> to act as a "readmacro"
(let ((*package* "MY-PACKAGE")
(*readtable* (gethash
"MY-PACKAGE"
*package-to-readtable*
*readtable*)))
(read))
This seems to be compatible to standard and allows one to associate a
readtable with some/any
package. Alternative is to change reader to dispatch on a package name
too. It is harder to implement
though as it requires to redefine some part or an implementation.
Of course, it would be better to allow for one colon instead of two,
but I see some
semantical problem here... It is not compatible to CLHS in case of
just a symbol.
Any ideas how to resolve this?
Also, any pointers to prior art are welcome.
budden <···········@mail.ru> writes:
> 1. Date reader. One could give a sence to tokens like 2009.04.02 to
> return a date.
> Any ideas about date format? I think the simplest one is yyyy.mm.dd.
> It's advantage
> is that it is sortable as string. Is it reasonable to have a date
> format controlled
> by a variable to allow yy.mm.dd or dd.mm.yy etc? I incline to disallow
> it.
> 2. How to add a time reader? It could be, say, [yyyy.mm.dd:]hh:mm[:ss
> [:msms]][+nn] where nn
> is a timezone spec and brackets delimit an optional parts. Ideas/
> suggestions are welcome.
Take a look at the date and time parser that's included with CMUCL. It
is the function PARSE-TIME and can be found in the code/parse-time.lisp
file in the CMUCL sources. See http://tinyurl.com/c8zwvs
--
Thomas A. Russ, USC/Information Sciences Institute
On Apr 2, 10:34 pm, ····@sevak.isi.edu (Thomas A. Russ) wrote:
> Take a look at the date and time parser that's included with CMUCL. It
> is the function PARSE-TIME and can be found in the code/parse-time.lisp
> file in the CMUCL sources. Seehttp://tinyurl.com/c8zwvs
I'm pretty sure that things like parsing dates and times are *not* the
sort of thing you want to add to the general CL reader.
Hi Tim!
> I'm pretty sure that things like parsing
dates and times are *not* the sort of thing
you want to add to the general CL reader.
I see no reason for not doing that. Tokens
like 2009.04.04:23:59:59 are senseless in CL,
why don't give them a sense? Of course, parsing
any hairy date representation like Monday,14,205 BC
is out of the scope of general reader, but having
a simple default seem to be reasonable. I'm short
in a resources now, but I really need to input and
output dates to/from SQL for my current work.
Frankly, I see no difference between integer
and a date. Both are primitive and practically
important data types and I see no reason why
there is a syntax for integers and there is no
syntax for date/time in a high-level language.
Meanwhile, I've finally came to conclusion that
it is impossible for me to understand True Lispers
and vice versa. I even have admitted that and this
does not hurt me anymore. So let me avoid further
debate :)
> My answer to that is that, if there are clashes,
> I always want to end up in the debugger (or somewhere
> equivalent), because it always indicates a problem
> with the code which I will need to think about.
In SQL, you have error only when you trying to use
clashing symbols, but not at the time when the conflict
becomes possible. When you join two tables with the same
fields, you may have no clash report. E.g.
create table foo (id int, name char(1));
create table bar (id int, short_name char(5));
select name from foo,bar;
In a last statement, between "select" and "from"
we have a temporarily merged namespace with all
fields from foo and bar.
You can refer to 'name' as it does not clash.
But you can't refer to id w/o qualifier:
select id from foo,bar /* error */
You can do
select foo.id from foo.bar /* no error */
This behavior is practical, I'm using it for
14 years in SQL and I see no contradiction with any
aspect of CL standard which would prevent using it
in CL.
The only thing we need is to allow multiple arguments
for in-package form:
(in-package :a-package :cl :other-library-package)
When reader meets unqualified symbol name, it looks for
it in every package we are "in". If there is an internal
symbol with that name in :a-package, or external symbol
with that name in &rest, and there is only one symbol
of that kind, it is used.
If there is no such symbol, it is interned to :a-package
Otherwise (more than one), error occurs.
So, for all clashing symbols, error would occur at the
time symbol name is about to be interned, not at the
time when in-package form is evaluated.
Advantages:
- if you don't use symbol a-package::foo clashing
with cl:foo, you need not care if it clashes.
Just use non-clashing a-package::bar and cl:quux
- if you use foo, you get to debugger as you wish
- a-package need not import symbols from cl-user.
So, a-package can be kept very small.
For a practical use, IDEs need to be changed a bit
too. Otherwise symbol completion won't work. So, while
I am able to redefine in-package form as I need, I
won't do that as I use some closed-source
IDE I'm unable to modify.
merge-packages-and-reexport suffice, it
requires no change to CL and it doesn't
harm completion.
Pascal uses similar clash resolution policy, but
clashing symbols do not report a error. First of
clashing symbols is used instead. This policy is
sometimes reasonable too. In fact, I'm using it
most frequently in merge-packages-and-reexport but
it is even harder to admit for True Lispers.
Also I've developed a tool which intercepts
attempts to intern one of listed symbol names to
a wrong package (other than package listed for that
symbol name). This tool is implementation-depended,
but it helped me a lot to refactor my code and
reorganize my packages layout.
I have already written about all that before and
noone understood me. I have no resources for proving
that this is more convinient than CL standard package
behaviour. I just use my merge-packages-and-reexport
in my code rather massively and I found that this is
really convinient and useful. Much more convinient
than other package handling techniques I've tried
before or that I read about. I'm not going to appeal
to anyone to follow me, but I will use this approach
until I stop using CL at all.
Oops, 2009.04.01 is not senseless, it is a symbol name.
Worse is that it looks a bit similar to ip address.
I'm starting to hesitate if I really should use it...
There is also 2009:04:01 which is not a symbol name
as it contains two package designators. But then
we can hardly use it for a time...
Hmmmmmm. Tim, why you didn't point to that? I had
to guess :)
budden <···········@mail.ru> writes:
> Hi Tim!
>
> > I'm pretty sure that things like parsing
> dates and times are *not* the sort of thing
> you want to add to the general CL reader.
>
> I see no reason for not doing that. Tokens
> like 2009.04.04:23:59:59 are senseless in CL,
> why don't give them a sense? Of course, parsing
> any hairy date representation like Monday,14,205 BC
> is out of the scope of general reader, but having
> a simple default seem to be reasonable.
Well, that is the real crux of the problem, now isn't it.
> I'm short
> in a resources now, but I really need to input and
> output dates to/from SQL for my current work.
Well, there are a number of ways to get around that which don't require
fundamentally changing the built-in reader. In fact, that is exactly
what all of the reader-macros, etc. are for. To allow you to easily
provide user extensions to the standard reader for a particular
application.
So you can just create a syntax, perhaps like #d"2009-04-04T23:59:59"
that allows you to read in dates in a restricted syntax. BTW, if you
were to go this route, I would recommend using ISO8601 or a subset of it
rather than creating your own idiosyncratic date format.
There's nothing wrong with that, but it isn't something that you would
want to include in the built-in reader, especially if it isn't complete.
> Frankly, I see no difference between integer
> and a date. Both are primitive and practically
> important data types and I see no reason why
> there is a syntax for integers and there is no
> syntax for date/time in a high-level language.
Perhaps because there is small set of established integer
representations that is generally agreed upon? There is nothing at all
like that for dates. Even things like ISO8601 are not that universally
adopted. And now you've opened the floodgate for all sorts of other
things.
For starters, you now have introduced a new interpretation for the ":"
package separator character, because you want to use it for time
expressions. And then there's the whole issue of time zones. etc.
And if we can read dates, why not times, durations, etc.
I would think that introducing dimensioned numbers would be much higher
on the list of primitive data types to support in a programming
language. But in fact, there is a very package written by Roman Cunis
that does this. And it does it without needing to re-write the
fundamental reader.
> Meanwhile, I've finally came to conclusion that
> it is impossible for me to understand True Lispers
> and vice versa. I even have admitted that and this
> does not hurt me anymore. So let me avoid further
> debate :)
Undoubtedly.
In this particular case, the difference lies in True Lispers wanting (as
usual) to Do the Right Thing, instead of cobbling together a partial
solution that handles only part of the problem. The latter seems to
belong more to the New Jersey School of Programming.
Many things were left out of the standard (or deferred from the early
standard work) because it was felt that there was not enough
understanding of the features to feel certain about including them in
the standard. So, there is a real desire in the community to try to
avoid inserting things into the language that are a bit half-baked, or
at least not very widely used or tested.
--
Thomas A. Russ, USC/Information Sciences Institute
Ok, I have to agree that yyyy.mm.dd is not a very good
idea as it clashes with many other possible uses of
nn.nn.nn form. Either wrong are yy:mm:dd and yy/mm/dd.
But dispatch macro characters scale poorly
and they can't print readably in the terse form.
Not having a default date format is another problem
but it doesn't seem to be so critical. Afterall, having
some default is better than nothing.
Maybe it would be better to use package-dependent token
parsers so that dates:|yyyy.mm.dd hh:mm:ss| would read
as date and date would print in this form.
Thins ip-addrs:127.0.0.1 would read as ip address.
On Fri, 03 Apr 2009 14:14:11 -0700, budden wrote:
> to/from SQL for my current work. Frankly, I see no difference between
> integer and a date. Both are primitive and practically important data
> types and I see no reason why there is a syntax for integers and there
> is no syntax for date/time in a high-level language.
Moreover, CL also lacks read syntax for knitting patterns, which also
map to integers --- I find it hard to contain my outrage at this
oversight.
All modern language should have knitting patterns --- it's not like
languages are extensible so you can add stuff later on. No Sir, if it
is not in the standard, the language should be redesigned on a
fundamental level. I shall shortly submit a CDR that specifies syntax
for weft and warp knitting, including knit, purl and various plaited
stitches.
Tamas
Tamas K Papp <······@gmail.com> writes:
> All modern language should have knitting patterns --- it's not like
> languages are extensible so you can add stuff later on. No Sir, if it
> is not in the standard, the language should be redesigned on a
> fundamental level. I shall shortly submit a CDR that specifies syntax
> for weft and warp knitting, including knit, purl and various plaited
> stitches.
>
> Tamas
Don't forget chess and music notation!
Nicolas
Nicolas Neuss wrote:
> Tamas K Papp <······@gmail.com> writes:
>
>> All modern language should have knitting patterns --- it's not like
>> languages are extensible so you can add stuff later on. No Sir, if it
>> is not in the standard, the language should be redesigned on a
>> fundamental level. I shall shortly submit a CDR that specifies syntax
>> for weft and warp knitting, including knit, purl and various plaited
>> stitches.
>>
>> Tamas
>
> Don't forget chess and music notation!
And juggling patterns. Then you can do number theory in realtime.
Paul Wallich wrote:
> Nicolas Neuss wrote:
>> Tamas K Papp <······@gmail.com> writes:
>>
>>> All modern language should have knitting patterns --- it's not like
>>> languages are extensible so you can add stuff later on. No Sir, if it
>>> is not in the standard, the language should be redesigned on a
>>> fundamental level. I shall shortly submit a CDR that specifies syntax
>>> for weft and warp knitting, including knit, purl and various plaited
>>> stitches.
>>>
>>> Tamas
>>
>> Don't forget chess and music notation!
>
> And juggling patterns. Then you can do number theory in realtime.
http://video.google.com/videoplay?docid=-8860158196198824415
--
ELS'09: http://www.european-lisp-symposium.org/
My website: http://p-cos.net
Common Lisp Document Repository: http://cdr.eurolisp.org
Closer to MOP & ContextL: http://common-lisp.net/project/closer/
Ok, now I'm evil enough and I know for sure: if I someday finish my
work on reader,
I add date format to it. Also I would hard-code as many readmacros as
possible to
make it very specific and hard to extend :)))
Also *read-eval-stream* would be added to my reader.
And one more thing that suggests itself:
read-eval binding for symbols.
This would allow do things like
sql:[ select * from foo ]
if sql:|[| has readtime-function
binding, without using macro characters.
Or even shorter:
> select * from foo
where id=4 ; optional comment
There are two ways to implement it.
1. Add a function readtime-function which assignes
reader-time function of two arguments (symbol and a stream).
to a symbol.
2. (more general) allow for a custom token parser.
Custom token parser is associtated with a regexp and
does an evaluation based on its registers.
Thus, instead of assigning read-eval binding per
symbol, one could also assign it to a "class" of symbols.
Say, token from "note" package would
read as a note where optional part after
- is a duration designator.
note:C4.32
note:C4.5/16
note:[ would read a chord, with optional
note:play* is inherited from "music" package
and plays its arguments with optional
numeric duration
> in-package :note
> play* [C4 E4 G4 1/16] [G3 H3 D4 1/16]
Note that C4, E4, etc symbol names do not denote
real symbols. They are processed by custom
token parser to produce a note object and never
get interned as symbols.
So, it is suggested that custom token parser should
be installed with
set-custom-token-parser (package regexp fn &key priority)
Where fn have a signature like this.
custom-token-parser-fn (package stream &rest registers)
Custom token parser would likely check if symbol is
already interned. This allows note package to
contain symbols with special meanings which
match the regexp. So, note parser could look like this
(defun note-parser (package stream register &rest more-registers)
(let1 sym (find-symbol register package)
(if sym sym
(parse-note-name-and-duration register))))
(set-custom-note-parser :note "(.*)" 'note-parser :priority 2)
(set-custom-note-parser :note "\[" 'read-chord :priority 1)
But there is still a choice. Token parsers are
more flexible, but they're bit harder to use.
Priorities would likely to become messy at some point.
On the other hand, making both token parsers and read-eval
bindings seems to be superflous.
Flame is not welcome. It would be useful to
know about possible pitfalls and prior art.
Also, until I forgot. It would be nice for to have
readtable-case package dependent, but I'm unsure if I
can implement it with current design of my reader.
Hi!
It is wonderful! It looks like it works. Sorry, code
is not ready to be published yet as it is not yet tested
under all implementations I have. Also I need to remove
some of Russian docstrings. Something is to be
done to handle extended character sets, and to make the
whole thing convinient to use.
Question 1 is: I need to set macro characters to any
character which can occur at the beginning of symbol.
Everything is clear with first 256 characters. But how
do I enumerate other characters? E.g. I want to use Russian
characters in identifiers, so I have to set macro character
to all of them. Ok, I can enumerate them. But is there some
more general (and portable) way to enumerate alphanumeric
Unicode characters of a given unicode set?
I'm new to Unicode :)
Also there is some design question. It looks like I have
enabled per-package aliases to work portably. For now, it
works like this:
(in-package :p)
(setf (gethash (find-package :p) *per-package-alias-table*)
`(("$" . "BUDDEN") ("RE" . "CL-PPCRE")))
Then I can write
(re:scan $::my-regex "a string")
instead of
(cl-ppcre:scan budden::my-regex "a string")
I also extended reader to recognize package
qualifier not only for symbols, but for
any expressions too. So, I can now
write
re::(a b c)
and get
'(cl-ppcre::a cl-ppcre::b cl-ppcre::c)
Also I use swank:*readtable-alist* to
associate readtable to a package. When I
use in-package, I still need to switch
readtable manually (once a file, I do it
with the help of named-readtables), but in
the scope of qualified expression readtables
are switched automatically.
So, if there are [] macro-characters in the
readtable of package :sql,
I can write:
(in-package :p)
sql::[select * from customer]
The question is about interaction of
two these features. It is likely I would
want to use symbols from my p package in
a DSL, when I call dsl from p package.
E.g. I want some of variables to be
passed to dsl as parameters.
I could just bind *package* to
(find-package :sql) and *readtable* to its
readtable from *readtable-alist*. But I would
likely need to write
sql:[select id, name from budden::tbl-name where id = p::*id*]
which is not very convinient (per package aliases
are assigned to :p package, not to :sql package. And it
is not good to assign them to a "library" sql package.
My suggestion is to switch package and readtable
in the scope of qualified expression, but to keep
per package aliases from old package. Also suggestion
is to assign temporary |.| alias for a *package* as
it was at the time of reading package
qualifier.
Then I could write:
(in-package :p)
(setf (gethash (find-package :p) *per-package-alias-table*)
`(("$" . "BUDDEN") ("RE" . "CL-PPCRE")))
(defparameter *id* 453)
(defparamter $::tbl-name "customer")
sql:[select id, name from $::tbl-name where id = .:*id*]
You can note that we changed package to :sql temporarily,
but aliases are still taken from package p.
This looks fine, but in fact this behaviour is rather
irregular and might be confusing. I'm uncertain if it
is ok.
Alternative is simply to bind old package
to something like *outer-package* and DSL implementor
could make use of it. This design seems simpler, but
more work is required to make behavior convinient. I
ask for an advice how to organize this better.
* budden
Wrote on Tue, 28 Apr 2009 02:54:24 -0700 (PDT):
| It is wonderful! It looks like it works. Sorry, code is not ready to
| be published yet as it is not yet tested under all implementations I
| have. Also I need to remove some of Russian docstrings. Something is
| to be done to handle extended character sets, and to make the whole
| thing convinient to use. Question 1 is: I need to set macro
| characters to any character which can occur at the beginning of
| symbol. Everything is clear with first 256 characters. But how do I
| enumerate other characters? E.g. I want to use Russian characters in
| identifiers, so I have to set macro character to all of them. Ok, I
| can enumerate them. But is there some more general (and portable) way
| to enumerate alphanumeric Unicode characters of a given unicode set?
I think smells of bad design if you have to define macro characters for
every character in the readtable.
The set of code points will keep growing as long as the Unicode
Consortium is in business. Your reader will have to be updated with
each new UCD update which they push to your lisp implementation.
| I also extended reader to recognize package qualifier not only for
| symbols, but for any expressions too. So, I can now write
|
| re::(a b c)
| and get
| '(cl-ppcre::a cl-ppcre::b cl-ppcre::c)
|
This is one syntax I miss in common lisp especially at the REPL. I'm
always jealous when seeing old snippets which use this. Allegro's
reader implements this.
| Also I use swank:*readtable-alist* to associate readtable to a
| package. When I use in-package, I still need to switch readtable
| manually (once a file, I do it with the help of named-readtables), but
| in the scope of qualified expression readtables are switched
| automatically.
There is no principle which says a package should be tied to a
readtable. You may perceive it as a convenient notion, and I agree, but
it is also unnecessarily restrictive. I think the package <-> readtable
association should happen at an application level, if the application so
desires. See how XMLisp handles this if you are interested. Having the
IDE force this upon me or forcing me to make a decision is not very
pleasant.
| So, if there are [] macro-characters in the readtable of package :sql,
| I can write:
[snip]
| The question is about interaction of two these features. It is likely
| I would want to use symbols from my p package in a DSL, when I call
| dsl from p package. E.g. I want some of variables to be passed to dsl
| as parameters.
[snip]
| Alternative is simply to bind old package to something like
| *outer-package* and DSL implementor could make use of it. This design
| seems simpler, but more work is required to make behavior
| convinient. I ask for an advice how to organize this better.
The question as you realize is of tradeoffs. You are missing the
possibility that the designers of common lisp already made pretty good
decisions.
I find it hard to relate to any of the "improvements" you post,
superficial as they may be, i.e. related to syntax, making things more
convenient, etc.
Now most of these seem to be motivated by clear-sky thinking for a few
minutes and a rapid implementation after which it is published.
Contrast this with what went into the making of CL: (generally speaking)
features that competed to get included were features that people had
been using for years, where the features proved themselves to be useful
after solid experience.
The package/reader system is one area which I think cannot be simplified
except at great cost to the language as a whole, and is, as it stands
good enough to support huge projects and complex software.
I've often thought about the things you mention, I even have several
pieces of experimental code that works at accessing symbols from other
packages, to circumvent the apparently unecessary rigidities of the
package system. etc. Then I realise surely others have thought of this
before, and that makes me pause a bit. In the end I always decided they
are not a good idea, and would definitely not be a good idea for others
to use if I were to work with their code. In the end I don't have any
general idea that proved its value in being useful to me, such that I
could think it would be useful to others in the long run, and so worth
publishing. I just think such convenience features would be more useful
should they come in the context of specific applications.
I'm not trying to disparage or discourage or stop you from wasting your
time on these superficial issues, insert smiley, (The issues do not get
in your way of writing complex or useful systems. But working
excessively on them might), and I *am* interested in the outcome. But I
would also exclude myself from your potential user group the minute you
tie any of these "innovations" to an ASDF based stack :)
--
Madhu
>I think smells of bad design if you have to define
>macro characters for every character in the readtable.
>The set of code points will keep growing as long as the
>Unicode Consortium is in business.
Maybe. But I see no alternatives for doing this portably
and with reasonably small effort. Also you might want to
inspect Allegro's default readtable.
> There is no principle which says a package should be
> tied to a readtable.
It is not imposed. Just leave association empty and you're free.
In fact, I don't know how does SLIME uses this binding. I only know
readtable is rebound to (copy-readtable nil) around loading of
every file by CL, so it seems to be reasonable to have a hook to
change this binding. I _think_ SLIME does this. For now, I only
use a variable from swank to avoid creating new variable with
a similar sense. This might prove to be wrong.
> You are missing the possibility that the designers of
> common lisp already made pretty good decisions.
If so, how do I scale readmacros? There is just too little
characters available. I had no way to use two
sences of brackets in the same program conviniently.
With my reader, it is trivial:
sql:[select 1 from dual]
sh:[pwd]
...
It is a link of readtable to package which enables this. I see
no other fair design. Also I see no solution w/o changing a reader.
Maybe something like
#?(with-sql-syntax) [select 1 from dual]
and/or
(shell #~pwd~)
both are not elegant and not scalable. If you think about using CL as
a
kind of integrated shell and a center of your working environment
(which is very natural), every extra character in frequently repeated
commands is annoying.
>But I would also exclude myself from your potential user group the
>minute you tie any of these "innovations" to an ASDF based stack :)
I dislike ASDF, but see no alternatives to it. Currently there are
no such bindings.
On Apr 28, 7:32 am, budden <···········@mail.ru> wrote:
> >I think smells of bad design if you have to define
> >macro characters for every character in the readtable.
> >The set of code points will keep growing as long as the
> >Unicode Consortium is in business.
> Maybe. But I see no alternatives for doing this portably
> and with reasonably small effort.
How about doing it non-portably with reasonably small effort? Unless
you yourself need to use many implementations on a regular basis, it
might be best to leave the porting to others.
Solving a problem non-portably is always better than not solving a
problem portably, IMO.
[...]
> With my reader, it is trivial:
> sql:[select 1 from dual]
> sh:[pwd]
> ...
> It is a link of readtable to package which enables this.
Isn't it just linking a readtable to a *name* which enables this? You
can still pun on package names if you like, of course.
> Maybe something like
> #?(with-sql-syntax) [select 1 from dual]
> and/or
> (shell #~pwd~)
> both are not elegant and not scalable.
Here's a sketch of an idea. Use a macro character to start things off,
read a string up to the first ":", and then read from an opening
delimiter to a closing delimiter. Something that looks a bit like,
@sh:[cd .. && rm *]
@sql:{select 1 from dual}
You can associate the strings "sh" and "sql" with functions that do
the required reading, including setting up the relevant readtable if
you want a whole new one. They'd get the stream as an argument, as
well as the opening delimiter. You'd leave more work to the
implementors of these functions then you might otherwise, but you'd
also probably give them a lot more freedom.
I dunno. Maybe it's a crap idea, or maybe there's a portability
obstacle I'm not seeing, but I think it might be worth a try.
Cheers,
Pillsy
> How about doing it non-portably with reasonably small effort? Unless
> you yourself need to use many implementations on a regular basis, it
> might be best to leave the porting to others.
I meant only enumerating unicode characters. Of course,
I'm going to do this non-portably. Some sequence or generator
enumerating all interesting characters would be required at
bootstrapping. Messing with different character encodings is
the thing I especially dislike. For my own use, there are
standard consituent characters and 66 cyrillic characters. Maybe
greek ones sometimes. No reason to worry.
> @sh:[cd .. && rm *]
> @sql:{select 1 from dual}
@ is a constituent character, so this design is not portable.
> You can associate the strings "sh" and "sql" with functions that do
> the required reading, including setting up the relevant readtable if
> you want a whole new one. They'd get the stream as an argument, as
> well as the opening delimiter. You'd leave more work to the
> implementors of these functions then you might otherwise, but you'd
> also probably give them a lot more freedom.
There are also custom token readers which in my project.
Each package have a list of functions associated to it. They are
called from left to right on a potential symbol name until one of
them reports "success" and returns some object which is
interpreted as a result of reading. So, my reader allows to
assign arbitrary code which evaluates at readtime. E.g.
I can cause budden::sql to read sql statement up to some closing
delimiter, say:
(in-package :budden)
sql
select id from customer
end sql
(in-package :cl-user)
budden:sql
delete from customer where id=80
end sql
and I can, say, close all "unsafe" symbols
and all introspection to try to produce a
restricted CL.
On Apr 28, 12:55 pm, budden <···········@mail.ru> wrote:
[...]
> > @sh:[cd .. && rm *]
> > @sql:{select 1 from dual}
> @ is a constituent character, so this design is not portable.
Huh. I didn't even realize that making a constituent character into a
macro character wasn't portable; I didn't see anything along those
lines in the CLHS, and CLTL2 mentions that {}[]! (constituent
characters all) are all specifically reserved for the user.
Cheers,
Pillsy
This piece of mud can be found here:
http://depositfiles.com/files/gxq189sxd
I'm really sorry for the current state of the sources.
But it looks like they work. It is just a
proof-of-concept. Start studying from tests.
Hope you would not have problems with Russian
encodings.
Iterate-keywords can be downloaded here:
http://sourceforge.net/projects/iteratekeywords/
What's in a box?
- portable hierarchical packages
- portable per-package aliases. Tim Bradshaw seem to have
some declarative form for that, but I didn't enabled this
yet
- expressions qualified with packages, e.g. foo::(a b c).
Currently they just change package and readtable
in its scope, but I think I'll change this to keep local
package aliases from surrounding package.
- package-readtable association from SWANK is used
in qualified expressions. So, you can qualify
macro-characters with package qualifiers.
E.g.
sql::[sql command]
sh::[shell command]
Also, with readtable-case, you can make some separate
packages case-sensitive, that is:
C::nameOfCFunction will read with case preserved if
readtable of C package is case-sensitive.
- custom symbol name parsers. One can set up a parser(s)
for a symbol names matching some criteria. This parser
would be called instead of just interning symbol, and
read would return whatever the parser returns. E.g.
you can set up date::2009.04.29 to read as date object,
not as a symbol
- intern-time checks. There is (should be) a variable
which would cause any reference to
qualified-package::non-existing-symbol
from the reader to cause a continuable error.
Also there is some code not related to a reader:
- iterate-keywords - a fix of design flaw in iterate.
Write (iter (:for ....)) instead of (iter (for ....))
this would save you from importing iter:for, which
is often painful. Download separatedly.
- look up of asdf packages with gnu readline on windows.
Make your-system.asd.lnk file and put all lnk files
to one dir, as you do in Unix.
- undefsystem. When asdf gets mad, make it forget
about your system. This helps sometimes
- package-file component for asdf
- dependable file. You can depend on it, but it is
not built by any process.
- proga. Removes some unnecessary nesting and parens.
- let1 - simplifies let with one argument
- merge-packages-and-reexport - resolve symbol
clashes in defpackage automagically. Clashes
are reported so you know about them, but you do
not fall to debugger per clash. Some relief from
package hell. May become depreceated someday.
- trivial-deftest. Yet another test facility.
Place your tests just in your source, not
in a separate file and get autodocumented
source this way. Tests=examples. You
can disable testing if you change one variable,
so they won't slowdown loading of your
stable code.
Just to whip up interest. This library contains:
- port of Allegros/Tim Bradshaw's hierarchical
packages to (at least) CCL, CLISP and SBCL.
Code should be completely portable to any
standard-compliant CL.
- ability to set up package nicknames per *package*,
ported to implemetations listed above
- package qualifier can be applied to any expression,
including readmacros, not to symbols only. In the
scope of that expression, readtable can be
switched to some other readtable associated with
the package qualified.
E.g.:
'keyword::(a b c)
would read as (:a :b :c)
sql:[select 1 from dual]
would invoke #\[ readmacro from readtable
bound to sql package
sh:[ls]
would invoke #\[ readmacro from readtable
bound to sh package
- custom token parsers. Normally, reader
attempts to intern any symbol it reads
to some package. In this library, you
can intercept this call and run arbitrary
code which is associated to a *package*
or a package qualifier of the symbol.
And some utilities:
- asdf::undefsystem. Sometimes asdf gets
mad, and undefsystem make it forget
about your system. This is sometimes
an alternative to reloading everything.
- asdf looking for system.asd.lnk shortcut
under win32 which can be then parsed
with GNU readlink to produce real
system.asd location. Works under
allegro and lispworks.
- asdf can dump md5 sums of all file
components, useful for comparing
systems on different computers
without being able to mix up
directories.
- asdf::look-tabooed-systems
On SBCL, oos ... :force t sometimes
tries to rebuild part of SBCL
and then SBCL gets destroyed.
I suggest a way to prevent some
systems from being rebuild.
- proga = get rid of some unnecessary
parens.
- merge-packages-and-reexport
resolve clashes on :use-package
automatically with some policy.
This way you can be able to
manage large number of smaller
packages.
- iterate-keywords (download it
separatedly from
http://sourceforge.net/projects/iteratekeywords/
simplify use of iterate with
any other package. Just
use keywords as clause heads
and get free of import headache.
- trivial-deftest. Yes, there are many
deftest macros already. Mine is, first
of all, simple. Second, it allows to
run tests at load time. So, no separate
test suite is required. Tests can be
placed in the main source file next
to function definitions and serve
as an examples. This is handy.
If you don't want to run them at every
load, just set/bind *run-tests* to nil.
State is alpha, and code really
needs cleanup, de-localisation and more
documentation.
Madhu <·······@meer.net> writes:
> * budden
> | I also extended reader to recognize package qualifier not only for
> | symbols, but for any expressions too. So, I can now write
> |
> | re::(a b c)
> | and get
> | '(cl-ppcre::a cl-ppcre::b cl-ppcre::c)
> |
>
> This is one syntax I miss in common lisp especially at the REPL. I'm
> always jealous when seeing old snippets which use this. Allegro's
> reader implements this.
Historic note. I believe that this was syntax pioneered by the Lisp
Machines. I'm not sure why this didn't get added to Common Lisp.
Maybe because at the time, only Zetalisp supported it?
--
Thomas A. Russ, USC/Information Sciences Institute
On Apr 3, 2:07 pm, Tim Bradshaw <··········@tfeb.org> wrote:
> On Apr 2, 10:34 pm, ····@sevak.isi.edu (Thomas A. Russ) wrote:
>
> > Take a look at the date and time parser that's included with CMUCL. It
> > is the function PARSE-TIME and can be found in the code/parse-time.lisp
> > file in the CMUCL sources. Seehttp://tinyurl.com/c8zwvs
>
> I'm pretty sure that things like parsing dates and times are *not* the
> sort of thing you want to add to the general CL reader.
I second that...
Marco
On Feb 5, 11:09 pm, budden <···········@mail.ru> wrote:
> But I don't see how I'll do that with conduits. Package itself can be
> defined with :EXTENDS/EXCLUDING, but first of all you need to find all
> the clashes. Maybe you don't want to resolve conflicts with my default
> policy, but, at least, my macro does this for you in a convinient way.
> So you can merge multiple packages and not being afraid of dozens of
> clashes.
[Sorry, I did not notice this, and this thread is now old]
My answer to that is that, if there are clashes, I always want to end
up in the debugger (or somewhere equivalent), because it always
indicates a problem with the code which I will need to think about.
> Another effort in that direction is James Anderson's modPackage:
> <URL:http://home.arcor.de/james_anderson/setf/library/de/setf/utility/pack...>
> (see the :export-through option, though I prefer slightly different
> behaviour)
Again, I see no indication of altering clash resolution policy. Am I
wrong?