From: Peter Seibel
Subject: Features that can only be provided by the implementation?
Date:
Message-ID: <m2odyubatm.fsf@gigamonkeys.com>
So folks have been discussing the desirability of having a way to
change "Common Lisp" a lot lately. And other folks point out that
many, or even most, changes that folks want could be built *in* Common
Lisp. But not all. So my question is, what features that you think
might usefully be available to you in a Common Lisp environment have
to be provided by the CL implementation itself. With the help of a few
folks on #lisp I've got this list so far:
- multithreading
- unicode
- foreign function interface
- extensible streams (e.g. Gray or Simple)
- specific behaviors of standard CL functions allow but not mandated
by the standard, e.g. a particular semantics for the various
pathname-related functions.
Any others? (I'm not counting things that could be provided using an
FFI, such as an interface to sockets.)
-Peter
--
Peter Seibel * ·····@gigamonkeys.com
Gigamonkeys Consulting * http://www.gigamonkeys.com/
Practical Common Lisp * http://www.gigamonkeys.com/book/
:test for `make-hash-table' should accept a function? (or there should
be a "standard" way to make this happen)
--
mvh,
Lars Rune Nøstdal
http://lars.nostdal.org/
Oh, and (setf (readtable-case *readtable*) :modern) .. or mabe
:non-insane, or :preserve-but-lowercase-the-stuff-in-cl :)
Threading also means mutux- and condition-stuff?
--
mvh,
Lars Rune Nøstdal
http://lars.nostdal.org/
Peter Seibel <·····@gigamonkeys.com> writes:
> Any others? (I'm not counting things that could be provided using an
> FFI, such as an interface to sockets.)
Real user-space sockets would also requires extensible streams.
I'd add:
- an event-handling/poll/etc facility that allows me to write code
to, eg, block waiting for events from Ltk and a socket, in the
same thread of control.
This could be something like SERVE-EVENT or something like the QUEUEs
in Allegro's MP package. Depending on the other facilities, it may be
possible to build such a thing on what's already there, but certainly
isn't in CL as specified.
--
/|_ .-----------------------.
,' .\ / | Free Mumia Abu-Jamal! |
,--' _,' | Abolish the racist |
/ / | death penalty! |
( -. | `-----------------------'
| ) |
(`-. '--.)
`. )----'
On Sat, 22 Apr 2006 02:31:17 GMT, Peter Seibel <·····@gigamonkeys.com> wrote:
>
> So folks have been discussing the desirability of having a way to
> change "Common Lisp" a lot lately. And other folks point out that
> many, or even most, changes that folks want could be built *in* Common
> Lisp. But not all. So my question is, what features that you think
> might usefully be available to you in a Common Lisp environment have
> to be provided by the CL implementation itself. With the help of a few
> folks on #lisp I've got this list so far:
>
> - multithreading
>
> - unicode
>
> - foreign function interface
>
> - extensible streams (e.g. Gray or Simple)
>
> - specific behaviors of standard CL functions allow but not mandated
> by the standard, e.g. a particular semantics for the various
> pathname-related functions.
>
> Any others? (I'm not counting things that could be provided using an
> FFI, such as an interface to sockets.)
- weak hash tables
--
Jack Unrue
From: Wade Humeniuk
Subject: Re: Features that can only be provided by the implementation?
Date:
Message-ID: <nbh2g.3703$2c3.2517@edtnps89>
Peter Seibel wrote:
> So folks have been discussing the desirability of having a way to
> change "Common Lisp" a lot lately. And other folks point out that
> many, or even most, changes that folks want could be built *in* Common
> Lisp. But not all. So my question is, what features that you think
> might usefully be available to you in a Common Lisp environment have
> to be provided by the CL implementation itself. With the help of a few
> folks on #lisp I've got this list so far:
>
> - multithreading
>
> - unicode
>
> - foreign function interface
>
> - extensible streams (e.g. Gray or Simple)
>
> - specific behaviors of standard CL functions allow but not mandated
> by the standard, e.g. a particular semantics for the various
> pathname-related functions.
>
> Any others? (I'm not counting things that could be provided using an
> FFI, such as an interface to sockets.)
- delivery/tree shaker
- embeddable Assembler support
- packaging as a shared library
- multiuser/shared memory security
- modular/swappable GC
- class browser
- package browser
- profiler
- debugger
Wade
On 2006-04-22, Wade Humeniuk <··················@telus.net> wrote:
> Peter Seibel wrote:
>> So folks have been discussing the desirability of having a way to
>> change "Common Lisp" a lot lately. And other folks point out that
>> many, or even most, changes that folks want could be built *in*
>> Common Lisp. But not all. So my question is, what features that you
>> think might usefully be available to you in a Common Lisp
>> environment have to be provided by the CL implementation itself.
>> With the help of a few folks on #lisp I've got this list so far:
>>
<snip>
> - class browser
> - package browser
I assume when you say this, you mean "functions to support class /
package browsers". The actual *browser* part could be in "user land",
yes?
Or am I totally hosed? Never having used either a class or package
browser, perhaps I'm incorrectly generalizing from "web browser".
-- L
From: Wade Humeniuk
Subject: Re: Features that can only be provided by the implementation?
Date:
Message-ID: <K%q2g.4239$2c3.1387@edtnps89>
Larry Clapp wrote:
> On 2006-04-22, Wade Humeniuk <··················@telus.net> wrote:
>> Peter Seibel wrote:
>>> So folks have been discussing the desirability of having a way to
>>> change "Common Lisp" a lot lately. And other folks point out that
>>> many, or even most, changes that folks want could be built *in*
>>> Common Lisp. But not all. So my question is, what features that you
>>> think might usefully be available to you in a Common Lisp
>>> environment have to be provided by the CL implementation itself.
>>> With the help of a few folks on #lisp I've got this list so far:
>>>
> <snip>
>> - class browser
>> - package browser
>
> I assume when you say this, you mean "functions to support class /
> package browsers". The actual *browser* part could be in "user land",
> yes?
>
> Or am I totally hosed? Never having used either a class or package
> browser, perhaps I'm incorrectly generalizing from "web browser".
>
Not just the functions. I mean is like the browsers you get with LW.
Class Browser - Graphical tool to traverse the CLOS/CL types/classes
Inspector - Graphical/Listener tool to inspect instances and other
data structures
Process Browser - Graphical/Listener tool to see what processes/threads
are running within the image and their states
Profiler - Both graphical and command line
Window Browser - Inspect/Analyze any GUI window currently instantiated within the
image.
Package Browser - Tool to see all the symbols within all the packages
within the image (written by me)
Not only are these things useful for development but something like the
Debugger can be used at runtime to automate error logging and reporting.
Wrapping stuff up with handler-bind, catching the problem, dumping the
stack with all its bindings and writing it to a file (for instance).
Wade
From: Ari Johnson
Subject: Re: Features that can only be provided by the implementation?
Date:
Message-ID: <m27j5hwsed.fsf@hermes.theari.com>
Wade Humeniuk <··················@telus.net> writes:
> Not just the functions. I mean is like the browsers you get with LW.
>
> Class Browser - Graphical tool to traverse the CLOS/CL types/classes
> Inspector - Graphical/Listener tool to inspect instances and other
> data structures
> Process Browser - Graphical/Listener tool to see what processes/threads
> are running within the image and their states
> Profiler - Both graphical and command line
> Window Browser - Inspect/Analyze any GUI window currently instantiated within the
> image.
> Package Browser - Tool to see all the symbols within all the packages
> within the image (written by me)
>
> Not only are these things useful for development but something like the
> Debugger can be used at runtime to automate error logging and reporting.
> Wrapping stuff up with handler-bind, catching the problem, dumping the
> stack with all its bindings and writing it to a file (for instance).
My interpretation of the original question was:
What features can only be provided by a Lisp implementation and
cannot be written in Lisp on top of those features?
Basically, this is a question of the minimum reasonable subset of
features that must be present in order to implement everything else.
Given the functions to access all the things you want to browse and
inspect along with a GUI facility, the browsers and inspectors
themselves can be written using those functions and the GUI facility.
I see a lot of this on this thread - answering the wrong question. A
few people are posting feature requests or standard-extension
requests. The question at hand doesn't appear to relate to either of
those.
On 2006-04-22, Wade Humeniuk <··················@telus.net> wrote:
> Larry Clapp wrote:
>> On 2006-04-22, Wade Humeniuk <··················@telus.net> wrote:
>>> Peter Seibel wrote:
>>>> So folks have been discussing the desirability of having a way to
>>>> change "Common Lisp" a lot lately. And other folks point out that
>>>> many, or even most, changes that folks want could be built *in*
>>>> Common Lisp. But not all. So my question is, what features that
>>>> you think might usefully be available to you in a Common Lisp
>>>> environment have to be provided by the CL implementation itself.
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>>>> With the help of a few folks on #lisp I've got this list so far:
>>>>
>> <snip>
>>> - class browser
>>> - package browser
>>
>> I assume when you say this, you mean "functions to support class /
>> package browsers". The actual *browser* part could be in "user
>> land", yes?
>>
>> Or am I totally hosed? Never having used either a class or package
>> browser, perhaps I'm incorrectly generalizing from "web browser".
>
> Not just the functions. I mean is like the browsers you get with LW.
>
> Class Browser - Graphical tool to traverse the CLOS/CL types/classes
> Inspector - Graphical/Listener tool to inspect instances and other
> data structures
> Process Browser - Graphical/Listener tool to see what
> processes/threads are running within the image and
> their states
> Profiler - Both graphical and command line
> Window Browser - Inspect/Analyze any GUI window currently
> instantiated within the image.
> Package Browser - Tool to see all the symbols within all the
> packages within the image (written by me)
>
> Not only are these things useful for development but something like
> the Debugger can be used at runtime to automate error logging and
> reporting. Wrapping stuff up with handler-bind, catching the
> problem, dumping the stack with all its bindings and writing it to a
> file (for instance).
But, you don't need to go into the guts of an implementation to
provide these things. You need only provide the foundation to build
these things upon. The CL implementation need not provide all the GUI
bells and whistles, but to build a window browser, for example, it
*must* provide a function that returns a list/tree of all open
windows.
For example, say you could wave a magic wand and make all
implementations provide an identical interface to GUI primitives, and
that said interface includes introspection to whatever required level
to allow the construction of a Window Browser. You could then write a
completely portable Window Browser on top of those primitives and run
it on all implementations. So the implementations have provided the
foundation, but you can write the browser itself outside of the
implementation. That's all I'm saying.
-- Larry
From: Luís Oliveira
Subject: Re: Features that can only be provided by the implementation?
Date:
Message-ID: <m2u08m2ud0.fsf@deadspam.com>
Peter Seibel <·····@gigamonkeys.com> writes:
> Any others? (I'm not counting things that could be provided using an
> FFI, such as an interface to sockets.)
Weak datastructures.
--
Luís Oliveira
luismbo (@) gmail (.) com
Equipa Portuguesa do Translation Project
http://www.iro.umontreal.ca/translation/registry.cgi?team=pt
From: Bill Atkins
Subject: Re: Features that can only be provided by the implementation?
Date:
Message-ID: <87ejzquxi4.fsf@rpi.edu>
·············@deadspam.com (Luís Oliveira) writes:
> Peter Seibel <·····@gigamonkeys.com> writes:
>> Any others? (I'm not counting things that could be provided using an
>> FFI, such as an interface to sockets.)
>
> Weak datastructures.
>
Weak puns. (self-referential humor?!)
From: Luís Oliveira
Subject: Re: Features that can only be provided by the implementation?
Date:
Message-ID: <m2psja2tx1.fsf@deadspam.com>
·············@deadspam.com (Luís Oliveira) writes:
> Weak datastructures.
Also, finalizers and the MOP.
--
Luís Oliveira
luismbo (@) gmail (.) com
Equipa Portuguesa do Translation Project
http://www.iro.umontreal.ca/translation/registry.cgi?team=pt
From: Pascal Costanza
Subject: Re: Features that can only be provided by the implementation?
Date:
Message-ID: <4au52aFukfgdU2@individual.net>
Luís Oliveira wrote:
> ·············@deadspam.com (Luís Oliveira) writes:
>> Weak datastructures.
>
> Also, finalizers and the MOP.
For the MOP, only funcallable instances need to be provided by the
underlying implementation. Afterwards, CLOS and the MOP can in principle
be implemented as a mere library. (I am not saying this is easy. ;)
Pascal
--
3rd European Lisp Workshop
July 3-4 - Nantes, France - co-located with ECOOP 2006
http://lisp-ecoop06.bknr.net/
···@zedat.fu-berlin.de (Stefan Ram) writes:
> Peter Seibel <·····@gigamonkeys.com> writes:
>>But not all. So my question is, what features that you think
>>might usefully be available to you in a Common Lisp environment
>>have to be provided by the CL implementation itself.
> [...]
>>folks on #lisp I've got this list so far:
> [...]
>>Any others? (I'm not counting things that could be provided using an
>>FFI, such as an interface to sockets.)
>
> A default GUI and regexp-operations similar to Perl's.
> (forgive my ignorance if those should already be part
> of Common Lisp).
Those MUST already be part of Common Lisp _implementations_ at least.
The need to standardize comes from existing divergent implementations.
It's because all this already exists, but with arbitrary differences
between the various implementations that the situation is painful.
--
__Pascal Bourguignon__ http://www.informatimago.com/
ATTENTION: Despite any other listing of product contents found
herein, the consumer is advised that, in actuality, this product
consists of 99.9999999999% empty space.
--- Better (portable) reflection outside CLOS/MOP, for example:
- reader (f.e, I want to get/map all entries from readtable)
- printer (f.e, I want to get/map all entries from pretty print
dispatch table)
- functions (access to lamda list)
- types (f.e, I want to check if particular symbol denotes a type (and
that type is not a class))
--- almost everything about enviroments (CLHS 3.1.1) is implementation
dependent,
I think this chapter should contain much more stuff... first class
enviroments is
nice idea imo.
--- series integrated with rest of Lisp (and with direct compiler
support).
Things above are not very important to me, btw. But it would be nice to
have them.
.
Peter Seibel schrieb:
> So folks have been discussing the desirability of having a way to
> change "Common Lisp" a lot lately. And other folks point out that
> many, or even most, changes that folks want could be built *in* Common
> Lisp. But not all. So my question is, what features that you think
> might usefully be available to you in a Common Lisp environment have
> to be provided by the CL implementation itself. With the help of a few
> folks on #lisp I've got this list so far:
notion of time:
eg:
(progn (send "OPEN VALVE" :RS232)
(wait 0.01) ;;wait 10ms
(send "START PUMP" :RS232)
(wait 2.25)
(send "STOP PUMP" :RS232)
(send "CLOSE VALVE" :RS232))
Norman Werner
Ok bad example ... let me clarify:
In assembly (at least for simple hardware) one can determine pretty
exactly what the hardware does in every clock cycle.
Most higher level languanges lost this capability. By looking at the
source
code one cannot eg. conclude which time is needed for a statement.
There is no
guarantee and little hope that this time is constant.
One cannot specify that two actions (eg. using I/O) are to be executed
exactly in parallel (given suitable hardware).
AFAIK even the order of evaluation of functional argumentes may vary .
This certainly is irrelevant for web- or database programming. But it
certainly is relevant for realtime computing and embedded systems.
There are billions of microcontrollers already
shipped worldwide. They are already powerfull enough to run scheme or
special versions
of linux (eg. http://www.uclinux.org/).
There are conferences, books, research departments and real money.
As to (get-internal-real-time):
on my system INTERNAL-TIME-UNITS-PER-SECOND is 1000 which amounts to an
maximal accuracy of 1ms this is pretty lousy for a machine with
maximal clock-frequency > 1GHz.
Norman Werner
> AFAIK even the order of evaluation of functional argumentes may vary .
This is wrong. I looked it up. From the hyperspec:
"The subforms in the cdr of the original form are evaluated in
left-to-right order in the current lexical and dynamic environments.
The primary value of each such evaluation becomes an argument to the
named function; any additional values returned by the subforms are
discarded.
The functional value of the operator is retrieved from the lexical
environment, and that function is invoked with the indicated
arguments."
It is 'only' unspecified
(http://www.lisp.org/HyperSpec/Issues/iss171-writeup.html)
whether the functional value of the operator is retrieved befor or
after evaluation of the arguments.
Norman Werner
From: Bill Atkins
Subject: Re: Features that can only be provided by the implementation?
Date:
Message-ID: <873bg5jndr.fsf@rpi.edu>
"Norman Werner" <······@web.de> writes:
> Ok bad example ... let me clarify:
>
> In assembly (at least for simple hardware) one can determine pretty
> exactly what the hardware does in every clock cycle.
>
> Most higher level languanges lost this capability. By looking at the
> source
> code one cannot eg. conclude which time is needed for a statement.
> There is no
> guarantee and little hope that this time is constant.
>
> One cannot specify that two actions (eg. using I/O) are to be executed
> exactly in parallel (given suitable hardware).
>
> AFAIK even the order of evaluation of functional argumentes may vary .
>
> This certainly is irrelevant for web- or database programming. But it
> certainly is relevant for realtime computing and embedded systems.
> There are billions of microcontrollers already
> shipped worldwide. They are already powerfull enough to run scheme or
> special versions
> of linux (eg. http://www.uclinux.org/).
> There are conferences, books, research departments and real money.
>
>
> As to (get-internal-real-time):
> on my system INTERNAL-TIME-UNITS-PER-SECOND is 1000 which amounts to an
> maximal accuracy of 1ms this is pretty lousy for a machine with
> maximal clock-frequency > 1GHz.
>
>
> Norman Werner
>
Are you asking for a real-time Lisp?
The problem of not being able to estimate how long a piece of code
will run has more to do with the possibility of the operating system
doing context switches in different situations than with the
high-levelness of a language. Maybe I misunderstand you.
--
"...and when, another time, I discovered that he considered not
unworthy of reflection in one of those mirrors of absolute truth which
were his writings a remark similar to one which I had had occasion to
make about our friend M. Legrandin, ...then it was suddenly revealed
to me that my own humble existence and the realms of the true were
less widely separated than I had supposed, that at certain points they
actually collided, and in my newfound confidence and joy, I had wept
upon his printed page as in the arms of a long-lost father."
Of course realtime capabilities are dependent on hardware and OS.
I do not say this is an easy task. But it is certainly a worthwile
feature.
And it would be usefull.
Is it possible? - look at this "real-time" Java with dedicated
hardware:
http://ipr.ira.uka.de/komodo/komodoEng.html .
Norman Werner
From: Marcin 'Qrczak' Kowalczyk
Subject: Re: Features that can only be provided by the implementation?
Date:
Message-ID: <87vet1l1vx.fsf@qrnik.zagroda>
"Norman Werner" <······@web.de> writes:
> As to (get-internal-real-time):
> on my system INTERNAL-TIME-UNITS-PER-SECOND is 1000 which amounts to an
> maximal accuracy of 1ms this is pretty lousy for a machine with
> maximal clock-frequency > 1GHz.
On Linux/x86 gettimeofday() and clock_gettime(CLOCK_REALTIME, _)
have the accuracy of microseconds (MHz), even though the interface of
clock_gettime uses nanoseconds. On my machine the gettimeofday() call
itself takes 2�s, so finer accuracy would be pointless.
--
__("< Marcin Kowalczyk
\__/ ······@knm.org.pl
^^ http://qrnik.knm.org.pl/~qrczak/
The system has more parts than only the CL-implementation.
CPU, operating system, memory, caches, busses , ... Certainly the
CL-implementation is not the first to blame.
But as I said - Lisp has even no notion of time. you tell Lisp what to
do and what to evaluate. You don't tell anything (besides sleep) about
time. Why not write something like:
"Compute f(x) - make sure this takes 20ms every invocation"
It would be up to the compiler to insert wait-cycles, disbale gc
temporarily,
or to trade speed for memory.
Norman Werner
I don't see why the programmer can't do this in a bog standard lisp by
defining new DEFUN like macros:
(DEFFUN-LESS-THAN 0.02 somefunc (args))
(DEFFUN-TIME-EXACT 0.02 somefunc (args))
These macros can inject code that asserts the function ran in < 20ms,
or pad out the funtion to exactly 20ms in the second case. If the code
in the function inherantly takes longer than 20ms, there really isn't
much the compiler can do about it, once you miss the time slot once
your hard RT system is broken. Optimising the function to take less
than a given time is the job of the programmer.
Cheers
Brad
From: Pascal Costanza
Subject: Re: Features that can only be provided by the implementation?
Date:
Message-ID: <4au4t3FukfgdU1@individual.net>
Peter Seibel wrote:
> So folks have been discussing the desirability of having a way to
> change "Common Lisp" a lot lately. And other folks point out that
> many, or even most, changes that folks want could be built *in* Common
> Lisp. But not all. So my question is, what features that you think
> might usefully be available to you in a Common Lisp environment have
> to be provided by the CL implementation itself. With the help of a few
> folks on #lisp I've got this list so far:
>
> - multithreading
>
> - unicode
>
> - foreign function interface
>
> - extensible streams (e.g. Gray or Simple)
>
> - specific behaviors of standard CL functions allow but not mandated
> by the standard, e.g. a particular semantics for the various
> pathname-related functions.
Two questions:
- The character type in Common Lisp is extensible. Is this not
sufficient to provide Unicode support?
- It should be possible to provide extensible streams and (some)
specific behaviors by shadowing the respective symbols from the
common-lisp packages and providing new implementations, right? (I do
this in Closer to MOP for some CLOS-related functionality, and this
seems to work fine.)
Pascal
--
3rd European Lisp Workshop
July 3-4 - Nantes, France - co-located with ECOOP 2006
http://lisp-ecoop06.bknr.net/
From: Peter Seibel
Subject: Re: Features that can only be provided by the implementation?
Date:
Message-ID: <m2ejzpbu3f.fsf@gigamonkeys.com>
Pascal Costanza <··@p-cos.net> writes:
> Peter Seibel wrote:
>> So folks have been discussing the desirability of having a way to
>> change "Common Lisp" a lot lately. And other folks point out that
>> many, or even most, changes that folks want could be built *in* Common
>> Lisp. But not all. So my question is, what features that you think
>> might usefully be available to you in a Common Lisp environment have
>> to be provided by the CL implementation itself. With the help of a few
>> folks on #lisp I've got this list so far:
>> - multithreading
>> - unicode
>> - foreign function interface
>> - extensible streams (e.g. Gray or Simple)
>> - specific behaviors of standard CL functions allow but not
>> mandated
>> by the standard, e.g. a particular semantics for the various
>> pathname-related functions.
>
> Two questions:
>
> - The character type in Common Lisp is extensible. Is this not
> sufficient to provide Unicode support?
But it's not extensible in user land--how do I make CL:CHAR-CODE grok
my new bigger characters? I gues I didn't say it in my original post
but I think I want to rule out "solutions" that require shadowing CL
symbols and providing a new implementation. Because what you're doing
at that point is essentially implementing your own CL (or other Lisp
dialect) as a very thin layer on top of an existing CL--i.e. you
become the implementation. (That you can do this relatively easily in
Common Lisp is incredibly cool, but not what I was thinking
of--otherwise I could claim to be able to add features to C by writing
a C compiler in C.)
> - It should be possible to provide extensible streams and (some)
> specific behaviors by shadowing the respective symbols from the
> common-lisp packages and providing new implementations, right? (I do
> this in Closer to MOP for some CLOS-related functionality, and this
> seems to work fine.)
Well, you have the advantage there that the MOP symbols aren't
standardized so your MOP is as good as anyone elses's MOP, including
the one that comes with the implementation. Though I wonder, suppose I
use Lisp X + Closer and try to interoperate with some library that
uses Lisp X's native MOP implementatation--will that always work? For
instance, if you implement any of your own metaobjects, what happens
when I get some metaobjects from Closer and other meta objects from
the native MOP?
To take a simpler example, suppose I wanted to write a library to
provide a consistent pathnames interface across implementations.
According to your strategy I could define my own PATHNAME class and
then shadow all the pathname related functions *and* CL:OPEN and
CL:WITH-OPEN-FILE, and any other function that accepts or returns a
pathname object. (CL:LOAD, CL:COMPILE-FILE, is this ever going to
stop?) But supposing that doesn't actually require me to shadow every
symbol in COMMON-LISP, I'm still screwed if I try to run my own code,
that uses my version of the pathnames API, with 3rd party libraries
that call the original CL functions because suddenly I'm may be given
a CL:PATHNAME object that isn't the same class as my own PATHNAME
replacement. I suppose it's possible that I could write my own
pathname related functions so they can seemlessly deal with either my
own pathname objects *or* CL:PATHNAMEs and maybe make it all work. But
I'm not convinced that there's any in-principle reason to expect that
I won't end up having to implement my whole own CL on top of the
native CL to get everything wrapped up properly.
In other words, the claim, that "You can always change any aspect of
Common Lisp by shadowing symbols from CL and using your own
NEW-IMPROVED-CL package", is technically true but in practice it may
require shadowing *every* symbol in CL. (I'm not saying you're making
that claim; just that what you said reminded me of it.)
-Peter
--
Peter Seibel * ·····@gigamonkeys.com
Gigamonkeys Consulting * http://www.gigamonkeys.com/
Practical Common Lisp * http://www.gigamonkeys.com/book/
From: Duane Rettig
Subject: Re: Features that can only be provided by the implementation?
Date:
Message-ID: <o0irp1jzoi.fsf@franz.com>
Peter Seibel <·····@gigamonkeys.com> writes:
> Pascal Costanza <··@p-cos.net> writes:
>
>> - The character type in Common Lisp is extensible. Is this not
>> sufficient to provide Unicode support?
>
> But it's not extensible in user land--how do I make CL:CHAR-CODE grok
> my new bigger characters?
Bigger than what? Actually, I don't see any evidence in the hyperspec
that the character type is extensible, but that doesn't preclude using
unicode Unicode subsets as the implementations' character set(s). There
are certain ordering requirements in CL, but as far as I know, these are
simple enough to include Unicode, at least ucs1 and ucs2, as well as
other character sets as ebcdic, besides ascii/latin1 (from which I presume
you are making your comparison).
Not to say that CL doesn't have some issues wrt character sets. Most
fundamentally, CL was developed (note I did not say standardized) at
a time before internationalization was popular, when "characters" were the
fundamental building block, and bytes were still potentially variable.
The description of cl:listen implies this. The time line is of course
slushy, and there are those that will argue (incorrectly) that the byte
has always been 8 bits, or that characters have always been un-tied to
the octet (8-bit-byte). But though this certainly has been true in
some pockets of locality, it has only been recent (perhaps the last 10
to 20 years) that the tide has turned toward the proliferation of this
reversal, where the octet (8-bit-byte) has become the fundamental data
unit of transfer, and the character has achieved some variability.
Also, there are locale issues that I suppose could be considered to be
"extensibility" issues. For example, there are colation standards
within unicode that provide for specific ways to order character sets,
and these do not fit in with the standard ordering rules of CL. But
then again, I don't know of any _language_ that supports unicode colation
directly; there are instead usually library interfaces that provide for
this, even in C, etc.
One of the things one must always consider in a character-set
implementation _by_ the _language_ is what tradeoffs one is willing to
consider. Something as simple as a variable-width character set could
wreak havoc with any programs which assume same-size characters and
which thus used random-access (e.g. aref/schar) on its strings (and
a lot of the language implementation itself will be taking these
advantages). This issue does not single out Common Lisp; all
langauges have to make these decisions.
All in all, I'd say that character sets are not a very large issue
in CL.
--
Duane Rettig ·····@franz.com Franz Inc. http://www.franz.com/
555 12th St., Suite 1450 http://www.555citycenter.com/
Oakland, Ca. 94607 Phone: (510) 452-2000; Fax: (510) 452-0182
From: Peter Seibel
Subject: Re: Features that can only be provided by the implementation?
Date:
Message-ID: <m2acadbgdf.fsf@gigamonkeys.com>
Duane Rettig <·····@franz.com> writes:
> Peter Seibel <·····@gigamonkeys.com> writes:
>
>> Pascal Costanza <··@p-cos.net> writes:
>>
>
>>> - The character type in Common Lisp is extensible. Is this not
>>> sufficient to provide Unicode support?
>>
>> But it's not extensible in user land--how do I make CL:CHAR-CODE grok
>> my new bigger characters?
>
> Bigger than what?
Bigger than what's already provided by an implementation. Obviously if
the implementation already supports Unicode then there's nothing to do
to add support. But Pascal's claim seemed to be that characters are
extensible in some way that would allow user code to add Unicode
support to a Lisp whose native character type was, say, ASCII.
-Peter
--
Peter Seibel * ·····@gigamonkeys.com
Gigamonkeys Consulting * http://www.gigamonkeys.com/
Practical Common Lisp * http://www.gigamonkeys.com/book/
From: Duane Rettig
Subject: Re: Features that can only be provided by the implementation?
Date:
Message-ID: <o0d5f94als.fsf@franz.com>
Peter Seibel <·····@gigamonkeys.com> writes:
> Duane Rettig <·····@franz.com> writes:
>
>> Peter Seibel <·····@gigamonkeys.com> writes:
>>
>>> Pascal Costanza <··@p-cos.net> writes:
>>>
>>
>>>> - The character type in Common Lisp is extensible. Is this not
>>>> sufficient to provide Unicode support?
>>>
>>> But it's not extensible in user land--how do I make CL:CHAR-CODE grok
>>> my new bigger characters?
>>
>> Bigger than what?
>
> Bigger than what's already provided by an implementation. Obviously if
> the implementation already supports Unicode then there's nothing to do
> to add support. But Pascal's claim seemed to be that characters are
> extensible in some way that would allow user code to add Unicode
> support to a Lisp whose native character type was, say, ASCII.
Right. The point is that "extensible" is the wrong word; more accurate
is "implementation dependent" which has a whole different set of
implications.
--
Duane Rettig ·····@franz.com Franz Inc. http://www.franz.com/
555 12th St., Suite 1450 http://www.555citycenter.com/
Oakland, Ca. 94607 Phone: (510) 452-2000; Fax: (510) 452-0182
From: Pascal Costanza
Subject: Re: Features that can only be provided by the implementation?
Date:
Message-ID: <4avja7Fv916gU1@individual.net>
Duane Rettig wrote:
> Peter Seibel <·····@gigamonkeys.com> writes:
>
>> Duane Rettig <·····@franz.com> writes:
>>
>>> Peter Seibel <·····@gigamonkeys.com> writes:
>>>
>>>> Pascal Costanza <··@p-cos.net> writes:
>>>>
>>>>> - The character type in Common Lisp is extensible. Is this not
>>>>> sufficient to provide Unicode support?
>>>> But it's not extensible in user land--how do I make CL:CHAR-CODE grok
>>>> my new bigger characters?
>>> Bigger than what?
>> Bigger than what's already provided by an implementation. Obviously if
>> the implementation already supports Unicode then there's nothing to do
>> to add support. But Pascal's claim seemed to be that characters are
>> extensible in some way that would allow user code to add Unicode
>> support to a Lisp whose native character type was, say, ASCII.
>
> Right. The point is that "extensible" is the wrong word; more accurate
> is "implementation dependent" which has a whole different set of
> implications.
Ah, I have drawn the wrong conclusions from the name of the type
'extended-char. ;)
Sorry for the fuzz.
Pascal
--
3rd European Lisp Workshop
July 3-4 - Nantes, France - co-located with ECOOP 2006
http://lisp-ecoop06.bknr.net/
From: Pascal Costanza
Subject: Re: Features that can only be provided by the implementation?
Date:
Message-ID: <4ausisFu78usU1@individual.net>
Peter Seibel wrote:
> Pascal Costanza <··@p-cos.net> writes:
>
>> Peter Seibel wrote:
>>> So folks have been discussing the desirability of having a way to
>>> change "Common Lisp" a lot lately. And other folks point out that
>>> many, or even most, changes that folks want could be built *in* Common
>>> Lisp. But not all. So my question is, what features that you think
>>> might usefully be available to you in a Common Lisp environment have
>>> to be provided by the CL implementation itself. With the help of a few
>>> folks on #lisp I've got this list so far:
>>> - multithreading
>>> - unicode
>>> - foreign function interface
>>> - extensible streams (e.g. Gray or Simple)
>>> - specific behaviors of standard CL functions allow but not
>>> mandated
>>> by the standard, e.g. a particular semantics for the various
>>> pathname-related functions.
>> Two questions:
>>
>> - The character type in Common Lisp is extensible. Is this not
>> sufficient to provide Unicode support?
>
> But it's not extensible in user land--how do I make CL:CHAR-CODE grok
> my new bigger characters? I gues I didn't say it in my original post
> but I think I want to rule out "solutions" that require shadowing CL
> symbols and providing a new implementation. Because what you're doing
> at that point is essentially implementing your own CL (or other Lisp
> dialect) as a very thin layer on top of an existing CL--i.e. you
> become the implementation. (That you can do this relatively easily in
> Common Lisp is incredibly cool, but not what I was thinking
> of--otherwise I could claim to be able to add features to C by writing
> a C compiler in C.)
>
>> - It should be possible to provide extensible streams and (some)
>> specific behaviors by shadowing the respective symbols from the
>> common-lisp packages and providing new implementations, right? (I do
>> this in Closer to MOP for some CLOS-related functionality, and this
>> seems to work fine.)
>
> Well, you have the advantage there that the MOP symbols aren't
> standardized so your MOP is as good as anyone elses's MOP, including
> the one that comes with the implementation. Though I wonder, suppose I
> use Lisp X + Closer and try to interoperate with some library that
> uses Lisp X's native MOP implementatation--will that always work? For
> instance, if you implement any of your own metaobjects, what happens
> when I get some metaobjects from Closer and other meta objects from
> the native MOP?
Sorry, misunderstanding. Of course in the general case, in order to have
a working MOP, the different libraries must use the same protocols. In
other words, they have to call and override the same generic functions
provided by the native CLOS implementation. Closer to MOP doesn't
violate that requirement (I hope).
What I meant is that I shadow, and provide new definitions, for
user-level macros, like defclass and defgeneric, to fix a few
superficial things. It seems to me that this works pretty well.
So the important question here is: Does an extension rely on everyone
else using the same extension, or is it sufficient to provide it for
local use?
> In other words, the claim, that "You can always change any aspect of
> Common Lisp by shadowing symbols from CL and using your own
> NEW-IMPROVED-CL package", is technically true but in practice it may
> require shadowing *every* symbol in CL. (I'm not saying you're making
> that claim; just that what you said reminded me of it.)
Of course, if an extension requires shadowing of (almost) all symbols,
then this is pretty useless. I think there are cases where shadowing
(relatively) few symbols does the job, but I am again not sure whether
the cases you mention fall under this category.
Pascal
--
3rd European Lisp Workshop
July 3-4 - Nantes, France - co-located with ECOOP 2006
http://lisp-ecoop06.bknr.net/
Peter Seibel <·····@gigamonkeys.com> writes:
> But supposing that doesn't actually require me to shadow every
> symbol in COMMON-LISP, I'm still screwed if I try to run my own code,
> that uses my version of the pathnames API, with 3rd party libraries
> that call the original CL functions
Indeed that's a problem for construction of large applications where
Scheme IMHO is on a better track. From what I understand, Scheme
libraries can shadow the old definition and ensure that all
subsequently loaded ilbraries can only see the new functionality. This
is a consequence on referential transparency for macros and relying on
bindings instead of packages.
In contrast, CL libraries that provide xyz:LET, qux:DEFUN etc. macros
expand into CL:LET and CL:DEFUN. But they should expand into
<previously-loaded-shadowing-package>:LET/DEFUN to at least have a
hope for interoperability of these two packages.
As a result, libraries with same overrides won't stack.
Manual intervention is needed: change the macro expansion in the
package's source code.
Of course, Scheme's appraoch does not guarantee that there's no
problem. Loading the stack of shadowing libraries in some "right"
(least problematic) order will certainly be an issue. This is shared
with CL.
Even in Scheme, extensions can cause trouble. E.g. BRL's SQL accessing
macros don't grok Kawa's extended LET syntax. Cause is a limited code
walker in BRL which does not know about Kawa's extended syntax. It
just recognizes plain R5RS syntax.
I'm curious what Scheme research in syntactic extensions will come up
with in the future.
Stackable macro systems is certainly good to have.
Regards,
Jorg Hohle
Telekom/T-Systems Technology Center
Peter Seibel wrote:
> So folks have been discussing the desirability of having a way to
> change "Common Lisp" a lot lately. And other folks point out that
> many, or even most, changes that folks want could be built *in* Common
> Lisp. But not all. So my question is, what features that you think
> might usefully be available to you in a Common Lisp environment have
> to be provided by the CL implementation itself. With the help of a few
> folks on #lisp I've got this list so far:
>
> - multithreading
I wouldn't necessarily care to have complete multithreading support.
The important thing would be for the Lisp implementation to be
thread-safe and thread-aware where it needs to be. The actual support
for creating and synchronizing threads could be left to the foreign
function interface. It's too platform-specific, and doesn't belong in
the language. You want to avoid the Java disease.
For instance, on Win32, I'd use critical sections and events, which are
different from POSIX mutexes and condition variables. There are lots of
other differences. Structured exception handling versus POSIX signals
and cleanup handlers, thread cancelation, thread-specific storage
cleanup with destructors, etc.
If you go too far in wrapping the platform, you basically create a new
platform which has a different set of inefficiencies and pathologies on
every underlying platform that it's ported to. You either get a crappy
subset of all of their capabilities that you have to program against,
or else you get some deluxe interfaces with all the features, half of
which are emulated on each platform (and it's a different half on each
platform).
Here is an example of such a thing: Windows sockets and select(). The
fd_set on Win32 is implemented as an unordered array of handles rather
than a bitmask. This has a number of implications:
- the first argument to select is unnecessary (the number of
descriptors to poll) since the fd_set knows how many descriptors are in
its vector. Some Win32 code leaves that argument it as zero and breaks
when ported to "real" sockets.
- The FD_ISSET macro actually performs a linear search through the
array instead of directly accessing the bit.
- the "struct timeval" timeout isn't passed into the kernel directly,
but has to be recalculated into some other kind of time, like probably
the number of milliseconds to wait. In UNIX-like systems, it would go
right down into the system call: a struct timeval is known between
kernel and userland.
So basically, the intended design of select() is badly emulated for the
sake of keeping the interface compatible, while coupling it to a
different underlying kernel interface on which polling is done using
arrays of handles. This could be identified as a case of abstraction
inversion, because the abstracting is done at an inappropriately low
level.
So what I do on Windows sockets is to do something completely different
anyway. In a highly efficient server, I'd use IO completion ports. In
client code, I'd just crack open the fd_set and iterate it myself,
inverting the logic: rather than testing using FD_SET, just directly
process the descriptors placed in the array, violating the
encapsulation within an #ifdef WIN32 block of code. Or just use the
WSA*() functions.
Often it's better to just rewrite a high level chunk of application
code rather than to keep it the same and coerce the the dumber, lower
pieces into behaving the same way.
> Any others? (I'm not counting things that could be provided using an
> FFI, such as an interface to sockets.)
And, more importantly, to /platform-specific/ sockets (etc.) that let
you program properly for that platform.
From: Peter Seibel
Subject: Re: Features that can only be provided by the implementation?
Date:
Message-ID: <m264l1bfvv.fsf@gigamonkeys.com>
"Kaz Kylheku" <········@gmail.com> writes:
> Peter Seibel wrote:
>> So folks have been discussing the desirability of having a way to
>> change "Common Lisp" a lot lately. And other folks point out that
>> many, or even most, changes that folks want could be built *in* Common
>> Lisp. But not all. So my question is, what features that you think
>> might usefully be available to you in a Common Lisp environment have
>> to be provided by the CL implementation itself. With the help of a few
>> folks on #lisp I've got this list so far:
>>
>> - multithreading
>
> I wouldn't necessarily care to have complete multithreading support.
> The important thing would be for the Lisp implementation to be
> thread-safe and thread-aware where it needs to be. The actual
> support for creating and synchronizing threads could be left to the
> foreign function interface.
That's in interesting point. So then the question that leaps to mind
is what set of characteristics and/or features would a language have
to provide to allow the rest to be built in userland using regular
FFI? Off the top of my head it seems, you'd need:
- A way to call back into Lisp from threads (so you can use an FFI
call to spin up a thread and then have that thread turn around and
call back into Lisp.)
- The basic Lisp runtime needs to be thread safe (and, presumably,
thread hot) so multiple threads calling back into it don't stomp
on each other.
- The semantics of dynamic variables in the face of threads need to
be specified.
Anything else. I'm assuming that one can use FFI to get access to
functions for locking, protecting critical sections, etc. Or does any
of that need to be more deeply intertwingled into the implementation?
I wonder in particular whether you'll be able to write a thread hot
runtime if the GC can't count on any cooperation from the compiler. I
don't know that you can't nor, even, do I have any particular idea in
mind about why it would be a problem. Just wondering.
-Peter
--
Peter Seibel * ·····@gigamonkeys.com
Gigamonkeys Consulting * http://www.gigamonkeys.com/
Practical Common Lisp * http://www.gigamonkeys.com/book/
From: Marcin 'Qrczak' Kowalczyk
Subject: Re: Features that can only be provided by the implementation?
Date:
Message-ID: <87k69hbdha.fsf@qrnik.zagroda>
"Kaz Kylheku" <········@gmail.com> writes:
> I wouldn't necessarily care to have complete multithreading support.
> The important thing would be for the Lisp implementation to be
> thread-safe and thread-aware where it needs to be.
The work needed to be thread-safe is almost all that is needed to
support threads, so it would be strange for an implementation to stop
at this point.
It's not clear which user-level operations should be thread-safe.
Memory allocation definitely should (which is the major reason why
many implementations won't be automatically thread-safe, unless they
use a third-party GC which is thread-aware by itself, like Boehm GC).
Interning symbols probably should, as well as DEFMETHOD. Accessing the
same stream probably need not. What about LOAD? What about DEFPACKAGE?
Special variables must have distinct bindings in different threads.
This certainly won't work unless the implementation has supported that
explicitly.
> The actual support for creating and synchronizing threads could be
> left to the foreign function interface. It's too platform-specific,
> and doesn't belong in the language.
I disagree. Threads can't be added as a library because of too subtle
interactions with other language features.
Don't be misled by the pthreads library: it would be impossible to
implement if the core library wasn't thread-aware. E.g. errno is
thread-local, Linux libc uses internal locks in malloc, in the dynamic
linker, in functions using the locale data, the timezone data, the
charset recoding framework, the resolver, the nsswitch mechanism etc.
I.e. in those areas which appear as thread-local but are implemented
using process-wide resources.
Threads are needed for good finalizers.
> You want to avoid the Java disease.
I disagree with various decisions that Java has made (implicitly
associating every object with a mutex, making it recursive, supporting
only a single implicit condition variable per that mutex) but I agree
with the idea that threads belong to the core language.
> For instance, on Win32, I'd use critical sections and events, which
> are different from POSIX mutexes and condition variables.
I've heard about problems with implementing condition variables in
Win32; I don't know the details. But if they are unimplementable, then
Win32 threading must be braindead. Mutexes with condition variables
are fundamental. Anyway, Cygwin appears to provide them.
> There are lots of other differences. Structured exception handling
> versus POSIX signals and cleanup handlers, thread cancelation,
> thread-specific storage cleanup with destructors, etc.
Lisp already provides its mechanism of conditions, there should be no
need to expose exceptions differently.
There is a need for asynchronous conditions ("conditions" in the Lisp
sense). I believe they require language support for temporary blocking,
which should be implicit in certain language features, and thus this
is an aspect which can't be added as a library. It makes the problem
of implementing threads harder, but it doesn't mean that it should not
be done - on the contrary, it means that it must be done in the core,
not added afterwards.
Thread cancellation is an instance of asynchronous conditions.
POSIX requires a separate mechanism for that because C doesn't support
exceptions.
Thread-specific storage should be exposed in the already existing
interface of special variables.
There is no need to expose cleanup of thread-local storage.
GC, finalizers and UNWIND-PROTECT should take care of that.
> You either get a crappy subset of all of their capabilities that you
> have to program against, or else you get some deluxe interfaces with
> all the features, half of which are emulated on each platform (and
> it's a different half on each platform).
The subset doesn't have to be crappy if the language is sufficiently
high-level. C needs ugly features like pthread_cleanup_push because
it doesn't have GC, exceptions nor UNWIND-PROTECT.
--
__("< Marcin Kowalczyk
\__/ ······@knm.org.pl
^^ http://qrnik.knm.org.pl/~qrczak/
Peter Seibel wrote:
> So folks have been discussing the desirability of having a way to
> change "Common Lisp" a lot lately. And other folks point out that
> many, or even most, changes that folks want could be built *in* Common
> Lisp. But not all. So my question is, what features that you think
> might usefully be available to you in a Common Lisp environment have
> to be provided by the CL implementation itself. With the help of a few
> folks on #lisp I've got this list so far:
>
> - multithreading
>
> - unicode
>
> - foreign function interface
>
> - extensible streams (e.g. Gray or Simple)
>
> - specific behaviors of standard CL functions allow but not mandated
> by the standard, e.g. a particular semantics for the various
> pathname-related functions.
>
> Any others? (I'm not counting things that could be provided using an
> FFI, such as an interface to sockets.)
>
> -Peter
>
Python Library:
* Front Matter
* Contents
* 1. Introduction
* 2. Built-In Objects
o 2.1 Built-in Functions
o 2.2 Non-essential Built-in Functions
o 2.3 Built-in Types
+ 2.3.1 Truth Value Testing
+ 2.3.2 Boolean Operations -- and, or, not
+ 2.3.3 Comparisons
+ 2.3.4 Numeric Types -- int, float, long, complex
+ 2.3.5 Iterator Types
+ 2.3.6 Sequence Types -- str, unicode, list, tuple,
buffer, xrange
+ 2.3.7 Set Types -- set, frozenset
+ 2.3.8 Mapping Types -- classdict
+ 2.3.9 File Objects
+ 2.3.10 Other Built-in Types
+ 2.3.11 Special Attributes
o 2.4 Built-in Exceptions
o 2.5 Built-in Constants
* 3. Python Runtime Services
o 3.1 sys -- System-specific parameters and functions
o 3.2 gc -- Garbage Collector interface
o 3.3 weakref -- Weak references
+ 3.3.1 Weak Reference Objects
+ 3.3.2 Example
+ 3.3.3 Weak References in Extension Types
o 3.4 fpectl -- Floating point exception control
+ 3.4.1 Example
+ 3.4.2 Limitations and other considerations
o 3.5 atexit -- Exit handlers
+ 3.5.1 atexit Example
o 3.6 types -- Names for built-in types
o 3.7 UserDict -- Class wrapper for dictionary objects
o 3.8 UserList -- Class wrapper for list objects
o 3.9 UserString -- Class wrapper for string objects
o 3.10 operator -- Standard operators as functions.
+ 3.10.1 Mapping Operators to Functions
o 3.11 inspect -- Inspect live objects
+ 3.11.1 Types and members
+ 3.11.2 Retrieving source code
+ 3.11.3 Classes and functions
+ 3.11.4 The interpreter stack
o 3.12 traceback -- Print or retrieve a stack traceback
+ 3.12.1 Traceback Example
o 3.13 linecache -- Random access to text lines
o 3.14 pickle -- Python object serialization
+ 3.14.1 Relationship to other Python modules
+ 3.14.2 Data stream format
+ 3.14.3 Usage
+ 3.14.4 What can be pickled and unpickled?
+ 3.14.5 The pickle protocol
+ 3.14.6 Subclassing Unpicklers
+ 3.14.7 Example
o 3.15 cPickle -- A faster pickle
o 3.16 copy_reg -- Register pickle support functions
o 3.17 shelve -- Python object persistence
+ 3.17.1 Restrictions
+ 3.17.2 Example
o 3.18 copy -- Shallow and deep copy operations
o 3.19 marshal -- Internal Python object serialization
o 3.20 warnings -- Warning control
+ 3.20.1 Warning Categories
+ 3.20.2 The Warnings Filter
+ 3.20.3 Available Functions
o 3.21 imp -- Access the import internals
+ 3.21.1 Examples
o 3.22 zipimport -- Import modules from Zip archives
+ 3.22.1 zipimporter Objects
+ 3.22.2 Examples
o 3.23 pkgutil -- Package extension utility
o 3.24 modulefinder -- Find modules used by a script
o 3.25 code -- Interpreter base classes
+ 3.25.1 Interactive Interpreter Objects
+ 3.25.2 Interactive Console Objects
o 3.26 codeop -- Compile Python code
o 3.27 pprint -- Data pretty printer
+ 3.27.1 PrettyPrinter Objects
o 3.28 repr -- Alternate repr() implementation
+ 3.28.1 Repr Objects
+ 3.28.2 Subclassing Repr Objects
o 3.29 new -- Creation of runtime internal objects
o 3.30 site -- Site-specific configuration hook
o 3.31 user -- User-specific configuration hook
o 3.32 __builtin__ -- Built-in objects
o 3.33 __main__ -- Top-level script environment
o 3.34 __future__ -- Future statement definitions
* 4. String Services
o 4.1 string -- Common string operations
+ 4.1.1 String constants
+ 4.1.2 Template strings
+ 4.1.3 String functions
+ 4.1.4 Deprecated string functions
o 4.2 re -- Regular expression operations
+ 4.2.1 Regular Expression Syntax
+ 4.2.2 Matching vs Searching
+ 4.2.3 Module Contents
+ 4.2.4 Regular Expression Objects
+ 4.2.5 Match Objects
+ 4.2.6 Examples
o 4.3 struct -- Interpret strings as packed binary data
o 4.4 difflib -- Helpers for computing deltas
+ 4.4.1 SequenceMatcher Objects
+ 4.4.2 SequenceMatcher Examples
+ 4.4.3 Differ Objects
+ 4.4.4 Differ Example
o 4.5 fpformat -- Floating point conversions
o 4.6 StringIO -- Read and write strings as files
o 4.7 cStringIO -- Faster version of StringIO
o 4.8 textwrap -- Text wrapping and filling
o 4.9 codecs -- Codec registry and base classes
+ 4.9.1 Codec Base Classes
+ 4.9.2 Standard Encodings
+ 4.9.3 encodings.idna -- Internationalized Domain
Names in Applications
o 4.10 unicodedata -- Unicode Database
o 4.11 stringprep -- Internet String Preparation
* 5. Miscellaneous Services
o 5.1 pydoc -- Documentation generator and online help system
o 5.2 doctest -- Test interactive Python examples
+ 5.2.1 Simple Usage: Checking Examples in Docstrings
+ 5.2.2 Simple Usage: Checking Examples in a Text File
+ 5.2.3 How It Works
+ 5.2.4 Basic API
+ 5.2.5 Unittest API
+ 5.2.6 Advanced API
+ 5.2.7 Debugging
+ 5.2.8 Soapbox
o 5.3 unittest -- Unit testing framework
+ 5.3.1 Basic example
+ 5.3.2 Organizing test code
+ 5.3.3 Re-using old test code
+ 5.3.4 Classes and functions
+ 5.3.5 TestCase Objects
+ 5.3.6 TestSuite Objects
+ 5.3.7 TestResult Objects
+ 5.3.8 TestLoader Objects
o 5.4 test -- Regression tests package for Python
+ 5.4.1 Writing Unit Tests for the test package
+ 5.4.2 Running tests using test.regrtest
o 5.5 test.test_support -- Utility functions for tests
o 5.6 decimal -- Decimal floating point arithmetic
+ 5.6.1 Quick-start Tutorial
+ 5.6.2 Decimal objects
+ 5.6.3 Context objects
+ 5.6.4 Signals
+ 5.6.5 Floating Point Notes
+ 5.6.6 Working with threads
+ 5.6.7 Recipes
+ 5.6.8 Decimal FAQ
o 5.7 math -- Mathematical functions
o 5.8 cmath -- Mathematical functions for complex numbers
o 5.9 random -- Generate pseudo-random numbers
o 5.10 whrandom -- Pseudo-random number generator
o 5.11 bisect -- Array bisection algorithm
+ 5.11.1 Examples
o 5.12 collections -- High-performance container datatypes
+ 5.12.1 Recipes
o 5.13 heapq -- Heap queue algorithm
+ 5.13.1 Theory
o 5.14 array -- Efficient arrays of numeric values
o 5.15 sets -- Unordered collections of unique elements
+ 5.15.1 Set Objects
+ 5.15.2 Example
+ 5.15.3 Protocol for automatic conversion to immutable
+ 5.15.4 Comparison to the built-in set types
o 5.16 itertools -- Functions creating iterators for
efficient looping
+ 5.16.1 Itertool functions
+ 5.16.2 Examples
+ 5.16.3 Recipes
o 5.17 ConfigParser -- Configuration file parser
+ 5.17.1 RawConfigParser Objects
+ 5.17.2 ConfigParser Objects
+ 5.17.3 SafeConfigParser Objects
o 5.18 fileinput -- Iterate over lines from multiple input
streams
o 5.19 calendar -- General calendar-related functions
o 5.20 cmd -- Support for line-oriented command interpreters
+ 5.20.1 Cmd Objects
o 5.21 shlex -- Simple lexical analysis
+ 5.21.1 shlex Objects
+ 5.21.2 Parsing Rules
* 6. Generic Operating System Services
o 6.1 os -- Miscellaneous operating system interfaces
+ 6.1.1 Process Parameters
+ 6.1.2 File Object Creation
+ 6.1.3 File Descriptor Operations
+ 6.1.4 Files and Directories
+ 6.1.5 Process Management
+ 6.1.6 Miscellaneous System Information
+ 6.1.7 Miscellaneous Functions
o 6.2 os.path -- Common pathname manipulations
o 6.3 dircache -- Cached directory listings
o 6.4 stat -- Interpreting stat() results
o 6.5 statcache -- An optimization of os.stat()
o 6.6 statvfs -- Constants used with os.statvfs()
o 6.7 filecmp -- File and Directory Comparisons
+ 6.7.1 The dircmp class
o 6.8 subprocess -- Subprocess management
+ 6.8.1 Using the subprocess Module
+ 6.8.2 Popen Objects
+ 6.8.3 Replacing Older Functions with the subprocess
Module
o 6.9 popen2 -- Subprocesses with accessible I/O streams
+ 6.9.1 Popen3 and Popen4 Objects
+ 6.9.2 Flow Control Issues
o 6.10 datetime -- Basic date and time types
+ 6.10.1 Available Types
+ 6.10.2 timedelta Objects
+ 6.10.3 date Objects
+ 6.10.4 datetime Objects
+ 6.10.5 time Objects
+ 6.10.6 tzinfo Objects
+ 6.10.7 strftime() Behavior
o 6.11 time -- Time access and conversions
o 6.12 sched -- Event scheduler
+ 6.12.1 Scheduler Objects
o 6.13 mutex -- Mutual exclusion support
+ 6.13.1 Mutex Objects
o 6.14 getpass -- Portable password input
o 6.15 curses -- Terminal handling for character-cell displays
+ 6.15.1 Functions
+ 6.15.2 Window Objects
+ 6.15.3 Constants
o 6.16 curses.textpad -- Text input widget for curses programs
+ 6.16.1 Textbox objects
o 6.17 curses.wrapper -- Terminal handler for curses programs
o 6.18 curses.ascii -- Utilities for ASCII characters
o 6.19 curses.panel -- A panel stack extension for curses.
+ 6.19.1 Functions
+ 6.19.2 Panel Objects
o 6.20 getopt -- Parser for command line options
o 6.21 optparse -- More powerful command line option parser
+ 6.21.1 Background
+ 6.21.2 Tutorial
+ 6.21.3 Reference Guide
+ 6.21.4 Option Callbacks
+ 6.21.5 Extending optparse
o 6.22 tempfile -- Generate temporary files and directories
o 6.23 errno -- Standard errno system symbols
o 6.24 glob -- Unix style pathname pattern expansion
o 6.25 fnmatch -- Unix filename pattern matching
o 6.26 shutil -- High-level file operations
+ 6.26.1 Example
o 6.27 locale -- Internationalization services
+ 6.27.1 Background, details, hints, tips and caveats
+ 6.27.2 For extension writers and programs that embed
Python
+ 6.27.3 Access to message catalogs
o 6.28 gettext -- Multilingual internationalization services
+ 6.28.1 GNU gettext API
+ 6.28.2 Class-based API
+ 6.28.3 Internationalizing your programs and modules
+ 6.28.4 Acknowledgements
o 6.29 logging -- Logging facility for Python
+ 6.29.1 Logger Objects
+ 6.29.2 Basic example
+ 6.29.3 Logging to multiple destinations
+ 6.29.4 Sending and receiving logging events across a
network
+ 6.29.5 Handler Objects
+ 6.29.6 Formatter Objects
+ 6.29.7 Filter Objects
+ 6.29.8 LogRecord Objects
+ 6.29.9 Thread Safety
+ 6.29.10 Configuration
o 6.30 platform -- Access to underlying platform's
identifying data.
+ 6.30.1 Cross Platform
+ 6.30.2 Java Platform
+ 6.30.3 Windows Platform
+ 6.30.4 Mac OS Platform
+ 6.30.5 Unix Platforms
* 7. Optional Operating System Services
o 7.1 signal -- Set handlers for asynchronous events
+ 7.1.1 Example
o 7.2 socket -- Low-level networking interface
+ 7.2.1 Socket Objects
+ 7.2.2 SSL Objects
+ 7.2.3 Example
o 7.3 select -- Waiting for I/O completion
+ 7.3.1 Polling Objects
o 7.4 thread -- Multiple threads of control
o 7.5 threading -- Higher-level threading interface
+ 7.5.1 Lock Objects
+ 7.5.2 RLock Objects
+ 7.5.3 Condition Objects
+ 7.5.4 Semaphore Objects
+ 7.5.5 Event Objects
+ 7.5.6 Thread Objects
+ 7.5.7 Timer Objects
o 7.6 dummy_thread -- Drop-in replacement for the thread module
o 7.7 dummy_threading -- Drop-in replacement for the
threading module
o 7.8 Queue -- A synchronized queue class
+ 7.8.1 Queue Objects
o 7.9 mmap -- Memory-mapped file support
o 7.10 anydbm -- Generic access to DBM-style databases
o 7.11 dbhash -- DBM-style interface to the BSD database library
+ 7.11.1 Database Objects
o 7.12 whichdb -- Guess which DBM module created a database
o 7.13 bsddb -- Interface to Berkeley DB library
+ 7.13.1 Hash, BTree and Record Objects
o 7.14 dumbdbm -- Portable DBM implementation
+ 7.14.1 Dumbdbm Objects
o 7.15 zlib -- Compression compatible with gzip
o 7.16 gzip -- Support for gzip files
o 7.17 bz2 -- Compression compatible with bzip2
+ 7.17.1 (De)compression of files
+ 7.17.2 Sequential (de)compression
+ 7.17.3 One-shot (de)compression
o 7.18 zipfile -- Work with ZIP archives
+ 7.18.1 ZipFile Objects
+ 7.18.2 PyZipFile Objects
+ 7.18.3 ZipInfo Objects
o 7.19 tarfile -- Read and write tar archive files
+ 7.19.1 TarFile Objects
+ 7.19.2 TarInfo Objects
+ 7.19.3 Examples
o 7.20 readline -- GNU readline interface
+ 7.20.1 Example
o 7.21 rlcompleter -- Completion function for GNU readline
+ 7.21.1 Completer Objects
* 8. Unix Specific Services
o 8.1 posix -- The most common POSIX system calls
+ 8.1.1 Large File Support
+ 8.1.2 Module Contents
o 8.2 pwd -- The password database
o 8.3 grp -- The group database
o 8.4 crypt -- Function to check Unix passwords
o 8.5 dl -- Call C functions in shared objects
+ 8.5.1 Dl Objects
o 8.6 dbm -- Simple ``database'' interface
o 8.7 gdbm -- GNU's reinterpretation of dbm
o 8.8 termios -- POSIX style tty control
+ 8.8.1 Example
o 8.9 tty -- Terminal control functions
o 8.10 pty -- Pseudo-terminal utilities
o 8.11 fcntl -- The fcntl() and ioctl() system calls
o 8.12 pipes -- Interface to shell pipelines
+ 8.12.1 Template Objects
o 8.13 posixfile -- File-like objects with locking support
o 8.14 resource -- Resource usage information
+ 8.14.1 Resource Limits
+ 8.14.2 Resource Usage
o 8.15 nis -- Interface to Sun's NIS (Yellow Pages)
o 8.16 syslog -- Unix syslog library routines
o 8.17 commands -- Utilities for running commands
* 9. The Python Debugger
o 9.1 Debugger Commands
o 9.2 How It Works
* 10. The Python Profiler
o 10.1 Introduction to the profiler
o 10.2 How Is This Profiler Different From The Old Profiler?
o 10.3 Instant Users Manual
o 10.4 What Is Deterministic Profiling?
o 10.5 Reference Manual
+ 10.5.1 The Stats Class
o 10.6 Limitations
o 10.7 Calibration
o 10.8 Extensions -- Deriving Better Profilers
o 10.9 hotshot -- High performance logging profiler
+ 10.9.1 Profile Objects
+ 10.9.2 Using hotshot data
+ 10.9.3 Example Usage
o 10.10 timeit -- Measure execution time of small code snippets
+ 10.10.1 Command Line Interface
+ 10.10.2 Examples
* 11. Internet Protocols and Support
o 11.1 webbrowser -- Convenient Web-browser controller
+ 11.1.1 Browser Controller Objects
o 11.2 cgi -- Common Gateway Interface support.
+ 11.2.1 Introduction
+ 11.2.2 Using the cgi module
+ 11.2.3 Higher Level Interface
+ 11.2.4 Old classes
+ 11.2.5 Functions
+ 11.2.6 Caring about security
+ 11.2.7 Installing your CGI script on a Unix system
+ 11.2.8 Testing your CGI script
+ 11.2.9 Debugging CGI scripts
+ 11.2.10 Common problems and solutions
o 11.3 cgitb -- Traceback manager for CGI scripts
o 11.4 urllib -- Open arbitrary resources by URL
+ 11.4.1 URLopener Objects
+ 11.4.2 Examples
o 11.5 urllib2 -- extensible library for opening URLs
+ 11.5.1 Request Objects
+ 11.5.2 OpenerDirector Objects
+ 11.5.3 BaseHandler Objects
+ 11.5.4 HTTPRedirectHandler Objects
+ 11.5.5 HTTPCookieProcessor Objects
+ 11.5.6 ProxyHandler Objects
+ 11.5.7 HTTPPasswordMgr Objects
+ 11.5.8 AbstractBasicAuthHandler Objects
+ 11.5.9 HTTPBasicAuthHandler Objects
+ 11.5.10 ProxyBasicAuthHandler Objects
+ 11.5.11 AbstractDigestAuthHandler Objects
+ 11.5.12 HTTPDigestAuthHandler Objects
+ 11.5.13 ProxyDigestAuthHandler Objects
+ 11.5.14 HTTPHandler Objects
+ 11.5.15 HTTPSHandler Objects
+ 11.5.16 FileHandler Objects
+ 11.5.17 FTPHandler Objects
+ 11.5.18 CacheFTPHandler Objects
+ 11.5.19 GopherHandler Objects
+ 11.5.20 UnknownHandler Objects
+ 11.5.21 HTTPErrorProcessor Objects
+ 11.5.22 Examples
o 11.6 httplib -- HTTP protocol client
+ 11.6.1 HTTPConnection Objects
+ 11.6.2 HTTPResponse Objects
+ 11.6.3 Examples
o 11.7 ftplib -- FTP protocol client
+ 11.7.1 FTP Objects
o 11.8 gopherlib -- Gopher protocol client
o 11.9 poplib -- POP3 protocol client
+ 11.9.1 POP3 Objects
+ 11.9.2 POP3 Example
o 11.10 imaplib -- IMAP4 protocol client
+ 11.10.1 IMAP4 Objects
+ 11.10.2 IMAP4 Example
o 11.11 nntplib -- NNTP protocol client
+ 11.11.1 NNTP Objects
o 11.12 smtplib -- SMTP protocol client
+ 11.12.1 SMTP Objects
+ 11.12.2 SMTP Example
o 11.13 smtpd -- SMTP Server
+ 11.13.1 SMTPServer Objects
+ 11.13.2 DebuggingServer Objects
+ 11.13.3 PureProxy Objects
+ 11.13.4 MailmanProxy Objects
o 11.14 telnetlib -- Telnet client
+ 11.14.1 Telnet Objects
+ 11.14.2 Telnet Example
o 11.15 urlparse -- Parse URLs into components
o 11.16 SocketServer -- A framework for network servers
+ 11.16.1 Server Creation Notes
+ 11.16.2 Server Objects
+ 11.16.3 RequestHandler Objects
o 11.17 BaseHTTPServer -- Basic HTTP server
o 11.18 SimpleHTTPServer -- Simple HTTP request handler
o 11.19 CGIHTTPServer -- CGI-capable HTTP request handler
o 11.20 cookielib -- Cookie handling for HTTP clients
+ 11.20.1 CookieJar and FileCookieJar Objects
+ 11.20.2 FileCookieJar subclasses and co-operation
with web browsers
+ 11.20.3 CookiePolicy Objects
+ 11.20.4 DefaultCookiePolicy Objects
+ 11.20.5 Cookie Objects
+ 11.20.6 Examples
o 11.21 Cookie -- HTTP state management
+ 11.21.1 Cookie Objects
+ 11.21.2 Morsel Objects
+ 11.21.3 Example
o 11.22 xmlrpclib -- XML-RPC client access
+ 11.22.1 ServerProxy Objects
+ 11.22.2 Boolean Objects
+ 11.22.3 DateTime Objects
+ 11.22.4 Binary Objects
+ 11.22.5 Fault Objects
+ 11.22.6 ProtocolError Objects
+ 11.22.7 MultiCall Objects
+ 11.22.8 Convenience Functions
+ 11.22.9 Example of Client Usage
o 11.23 SimpleXMLRPCServer -- Basic XML-RPC server
+ 11.23.1 SimpleXMLRPCServer Objects
+ 11.23.2 CGIXMLRPCRequestHandler
o 11.24 DocXMLRPCServer -- Self-documenting XML-RPC server
+ 11.24.1 DocXMLRPCServer Objects
+ 11.24.2 DocCGIXMLRPCRequestHandler
o 11.25 asyncore -- Asynchronous socket handler
+ 11.25.1 asyncore Example basic HTTP client
o 11.26 asynchat -- Asynchronous socket command/response handler
+ 11.26.1 asynchat - Auxiliary Classes and Functions
+ 11.26.2 asynchat Example
* 12. Internet Data Handling
o 12.1 formatter -- Generic output formatting
+ 12.1.1 The Formatter Interface
+ 12.1.2 Formatter Implementations
+ 12.1.3 The Writer Interface
+ 12.1.4 Writer Implementations
o 12.2 email -- An email and MIME handling package
+ 12.2.1 Representing an email message
+ 12.2.2 Parsing email messages
+ 12.2.3 Generating MIME documents
+ 12.2.4 Creating email and MIME objects from scratch
+ 12.2.5 Internationalized headers
+ 12.2.6 Representing character sets
+ 12.2.7 Encoders
+ 12.2.8 Exception and Defect classes
+ 12.2.9 Miscellaneous utilities
+ 12.2.10 Iterators
+ 12.2.11 Package History
+ 12.2.12 Differences from mimelib
+ 12.2.13 Examples
o 12.3 mailcap -- Mailcap file handling.
o 12.4 mailbox -- Read various mailbox formats
+ 12.4.1 Mailbox Objects
o 12.5 mhlib -- Access to MH mailboxes
+ 12.5.1 MH Objects
+ 12.5.2 Folder Objects
+ 12.5.3 Message Objects
o 12.6 mimetools -- Tools for parsing MIME messages
+ 12.6.1 Additional Methods of Message Objects
o 12.7 mimetypes -- Map filenames to MIME types
+ 12.7.1 MimeTypes Objects
o 12.8 MimeWriter -- Generic MIME file writer
+ 12.8.1 MimeWriter Objects
o 12.9 mimify -- MIME processing of mail messages
o 12.10 multifile -- Support for files containing distinct parts
+ 12.10.1 MultiFile Objects
+ 12.10.2 MultiFile Example
o 12.11 rfc822 -- Parse RFC 2822 mail headers
+ 12.11.1 Message Objects
+ 12.11.2 AddressList Objects
o 12.12 base64 -- RFC 3548: Base16, Base32, Base64 Data Encodings
o 12.13 binascii -- Convert between binary and ASCII
o 12.14 binhex -- Encode and decode binhex4 files
+ 12.14.1 Notes
o 12.15 quopri -- Encode and decode MIME quoted-printable data
o 12.16 uu -- Encode and decode uuencode files
o 12.17 xdrlib -- Encode and decode XDR data
+ 12.17.1 Packer Objects
+ 12.17.2 Unpacker Objects
+ 12.17.3 Exceptions
o 12.18 netrc -- netrc file processing
+ 12.18.1 netrc Objects
o 12.19 robotparser -- Parser for robots.txt
o 12.20 csv -- CSV File Reading and Writing
+ 12.20.1 Module Contents
+ 12.20.2 Dialects and Formatting Parameters
+ 12.20.3 Reader Objects
+ 12.20.4 Writer Objects
+ 12.20.5 Examples
* 13. Structured Markup Processing Tools
o 13.1 HTMLParser -- Simple HTML and XHTML parser
+ 13.1.1 Example HTML Parser Application
o 13.2 sgmllib -- Simple SGML parser
o 13.3 htmllib -- A parser for HTML documents
+ 13.3.1 HTMLParser Objects
o 13.4 htmlentitydefs -- Definitions of HTML general entities
o 13.5 xml.parsers.expat -- Fast XML parsing using Expat
+ 13.5.1 XMLParser Objects
+ 13.5.2 ExpatError Exceptions
+ 13.5.3 Example
+ 13.5.4 Content Model Descriptions
+ 13.5.5 Expat error constants
o 13.6 xml.dom -- The Document Object Model API
+ 13.6.1 Module Contents
+ 13.6.2 Objects in the DOM
+ 13.6.3 Conformance
o 13.7 xml.dom.minidom -- Lightweight DOM implementation
+ 13.7.1 DOM Objects
+ 13.7.2 DOM Example
+ 13.7.3 minidom and the DOM standard
o 13.8 xml.dom.pulldom -- Support for building partial DOM trees
+ 13.8.1 DOMEventStream Objects
o 13.9 xml.sax -- Support for SAX2 parsers
+ 13.9.1 SAXException Objects
o 13.10 xml.sax.handler -- Base classes for SAX handlers
+ 13.10.1 ContentHandler Objects
+ 13.10.2 DTDHandler Objects
+ 13.10.3 EntityResolver Objects
+ 13.10.4 ErrorHandler Objects
o 13.11 xml.sax.saxutils -- SAX Utilities
o 13.12 xml.sax.xmlreader -- Interface for XML parsers
+ 13.12.1 XMLReader Objects
+ 13.12.2 IncrementalParser Objects
+ 13.12.3 Locator Objects
+ 13.12.4 InputSource Objects
+ 13.12.5 The Attributes Interface
+ 13.12.6 The AttributesNS Interface
o 13.13 xmllib -- A parser for XML documents
+ 13.13.1 XML Namespaces
* 14. Multimedia Services
o 14.1 audioop -- Manipulate raw audio data
o 14.2 imageop -- Manipulate raw image data
o 14.3 aifc -- Read and write AIFF and AIFC files
o 14.4 sunau -- Read and write Sun AU files
+ 14.4.1 AU_read Objects
+ 14.4.2 AU_write Objects
o 14.5 wave -- Read and write WAV files
+ 14.5.1 Wave_read Objects
+ 14.5.2 Wave_write Objects
o 14.6 chunk -- Read IFF chunked data
o 14.7 colorsys -- Conversions between color systems
o 14.8 rgbimg -- Read and write ``SGI RGB'' files
o 14.9 imghdr -- Determine the type of an image
o 14.10 sndhdr -- Determine type of sound file
o 14.11 ossaudiodev -- Access to OSS-compatible audio devices
+ 14.11.1 Audio Device Objects
+ 14.11.2 Mixer Device Objects
* 15. Cryptographic Services
o 15.1 hmac -- Keyed-Hashing for Message Authentication
o 15.2 md5 -- MD5 message digest algorithm
o 15.3 sha -- SHA-1 message digest algorithm
* 16. Graphical User Interfaces with Tk
o 16.1 Tkinter -- Python interface to Tcl/Tk
+ 16.1.1 Tkinter Modules
+ 16.1.2 Tkinter Life Preserver
+ 16.1.3 A (Very) Quick Look at Tcl/Tk
+ 16.1.4 Mapping Basic Tk into Tkinter
+ 16.1.5 How Tk and Tkinter are Related
+ 16.1.6 Handy Reference
o 16.2 Tix -- Extension widgets for Tk
+ 16.2.1 Using Tix
+ 16.2.2 Tix Widgets
+ 16.2.3 Tix Commands
o 16.3 ScrolledText -- Scrolled Text Widget
o 16.4 turtle -- Turtle graphics for Tk
+ 16.4.1 Pen and RawPen Objects
o 16.5 Idle
+ 16.5.1 Menus
+ 16.5.2 Basic editing and navigation
+ 16.5.3 Syntax colors
o 16.6 Other Graphical User Interface Packages
* 17. Restricted Execution
o 17.1 rexec -- Restricted execution framework
+ 17.1.1 RExec Objects
+ 17.1.2 Defining restricted environments
+ 17.1.3 An example
o 17.2 Bastion -- Restricting access to objects
* 18. Python Language Services
o 18.1 parser -- Access Python parse trees
+ 18.1.1 Creating AST Objects
+ 18.1.2 Converting AST Objects
+ 18.1.3 Queries on AST Objects
+ 18.1.4 Exceptions and Error Handling
+ 18.1.5 AST Objects
+ 18.1.6 Examples
o 18.2 symbol -- Constants used with Python parse trees
o 18.3 token -- Constants used with Python parse trees
o 18.4 keyword -- Testing for Python keywords
o 18.5 tokenize -- Tokenizer for Python source
o 18.6 tabnanny -- Detection of ambiguous indentation
o 18.7 pyclbr -- Python class browser support
+ 18.7.1 Class Descriptor Objects
+ 18.7.2 Function Descriptor Objects
o 18.8 py_compile -- Compile Python source files
o 18.9 compileall -- Byte-compile Python libraries
o 18.10 dis -- Disassembler for Python byte code
+ 18.10.1 Python Byte Code Instructions
o 18.11 pickletools -- Tools for pickle developers.
o 18.12 distutils -- Building and installing Python modules
* 19. Python compiler package
o 19.1 The basic interface
o 19.2 Limitations
o 19.3 Python Abstract Syntax
+ 19.3.1 AST Nodes
+ 19.3.2 Assignment nodes
+ 19.3.3 Examples
o 19.4 Using Visitors to Walk ASTs
o 19.5 Bytecode Generation
* 20. SGI IRIX Specific Services
o 20.1 al -- Audio functions on the SGI
+ 20.1.1 Configuration Objects
+ 20.1.2 Port Objects
o 20.2 AL -- Constants used with the al module
o 20.3 cd -- CD-ROM access on SGI systems
+ 20.3.1 Player Objects
+ 20.3.2 Parser Objects
o 20.4 fl -- FORMS library for graphical user interfaces
+ 20.4.1 Functions Defined in Module fl
+ 20.4.2 Form Objects
+ 20.4.3 FORMS Objects
o 20.5 FL -- Constants used with the fl module
o 20.6 flp -- Functions for loading stored FORMS designs
o 20.7 fm -- Font Manager interface
o 20.8 gl -- Graphics Library interface
o 20.9 DEVICE -- Constants used with the gl module
o 20.10 GL -- Constants used with the gl module
o 20.11 imgfile -- Support for SGI imglib files
o 20.12 jpeg -- Read and write JPEG files
* 21. SunOS Specific Services
o 21.1 sunaudiodev -- Access to Sun audio hardware
+ 21.1.1 Audio Device Objects
o 21.2 SUNAUDIODEV -- Constants used with sunaudiodev
* 22. MS Windows Specific Services
o 22.1 msvcrt - Useful routines from the MS VC++ runtime
+ 22.1.1 File Operations
+ 22.1.2 Console I/O
+ 22.1.3 Other Functions
o 22.2 _winreg - Windows registry access
+ 22.2.1 Registry Handle Objects
o 22.3 winsound -- Sound-playing interface for Windows
* A. Undocumented Modules
o A.1 Frameworks
o A.2 Miscellaneous useful utilities
o A.3 Platform specific modules
o A.4 Multimedia
o A.5 Obsolete
o A.6 SGI-specific Extension modules
[...]
Scheme Library (slib):
* Textual Conversion Packages
o Precedence Parsing
+ Precedence Parsing Overview
+ Rule Types
+ Ruleset Definition and Use
+ Token definition
+ Nud and Led Definition
+ Grammar Rule Definition
o Format (version 3.1)
+ Format Interface
+ Format Specification (Format version 3.1)
# Implemented CL Format Control Directives
# Not Implemented CL Format Control Directives
# Extended, Replaced and Additional Control
Directives
# Configuration Variables
# Compatibility With Other Format Implementations
o Standard Formatted I/O
+ stdio
+ Standard Formatted Output
# Exact Conversions
# Inexact Conversions
# Other Conversions
+ Standard Formatted Input
o Program and Arguments
+ Getopt
+ Getopt---
+ Command Line
+ Parameter lists
+ Getopt Parameter lists
+ Filenames
+ Batch
o HTML
o HTML Forms
o HTML Tables
+ HTML editing tables
+ HTML databases
o HTTP and CGI
o Parsing HTML
o URI
o Printing Scheme
+ Generic-Write
+ Object-To-String
+ Pretty-Print
o Time and Date
+ Time Zone
+ Posix Time
+ Common-Lisp Time
+ Time Infrastructure
o NCBI-DNA
o Schmooz
* Mathematical Packages
o Bit-Twiddling
+ Bitwise Operations
+ Integer Properties
+ Bit Within Word
+ Field of Bits
+ Bits as Booleans
o Modular Arithmetic
o Prime Numbers
o Random Numbers
+ Exact Random Numbers
+ Inexact Random Numbers
o Fast Fourier Transform
o Cyclic Checksum
o Graphing
+ Character Plotting
+ PostScript Graphing
# Column Ranges
# Drawing the Graph
# Graphics Context
# Rectangles
# Legending
# Legacy Plotting
# Example Graph
o Solid Modeling
o Color
+ Color Data-Type
# External Representation
# White
+ Color Spaces
+ Spectra
+ Color Difference Metrics
+ Color Conversions
+ Color Names
+ Daylight
o Root Finding
o Minimizing
o The Limit
o Commutative Rings
o Rules and Rulesets
o How to Create a Commutative Ring
o Matrix Algebra
* Database Packages
o Relational Database
+ Using Databases
+ Table Operations
# Single Row Operations
# Match-Keys
# Multi-Row Operations
# Indexed Sequential Access Methods
# Sequential Index Operations
# Table Administration
+ Database Interpolation
+ Embedded Commands
# Database Extension
# Command Intrinsics
# Define-tables Example
# The *commands* Table
# Command Service
# Command Example
+ Database Macros
# Within-database
# Within-database Example
+ Database Browser
o Relational Infrastructure
+ Base Table
# The Base
# Base Tables
# Base Field Types
# Composite Keys
# Base Record Operations
# Match Keys
# Aggregate Base Operations
# Base ISAM Operations
+ Catalog Representation
+ Relational Database Objects
+ Database Operations
o Weight-Balanced Trees
+ Construction of Weight-Balanced Trees
+ Basic Operations on Weight-Balanced Trees
+ Advanced Operations on Weight-Balanced Trees
+ Indexing Operations on Weight-Balanced Trees
* Other Packages
o Data Structures
+ Arrays
+ Subarrays
+ Array Mapping
+ Array Interpolation
+ Association Lists
+ Byte
+ Byte/Number Conversions
+ MAT-File Format
+ Portable Image Files
+ Collections
+ Dynamic Data Type
+ Hash Tables
+ Macroless Object System
+ Concepts
+ Procedures
+ Examples
# Inverter Documentation
# Number Documention
# Inverter code
+ Priority Queues
+ Queues
+ Records
o Sorting and Searching
+ Common List Functions
# List construction
# Lists as sets
# Lists as sequences
# Destructive list operations
# Non-List functions
+ Tree operations
+ Chapter Ordering
+ Sorting
+ Topological Sort
+ Hashing
+ Space-Filling Curves
# Hilbert Space-Filling Curve
# Gray code
# Bitwise Lamination
# Peano Space-Filling Curve
# Sierpinski Curve
+ Soundex
+ String Search
+ Sequence Comparison
o Procedures
+ Type Coercion
+ String-Case
+ String Ports
+ Line I/O
+ Multi-Processing
+ Metric Units
# SI Prefixes
# Binary Prefixes
# Unit Symbols
o Standards Support
+ RnRS
+ With-File
+ Transcripts
+ Rev2 Procedures
+ Rev4 Optional Procedures
+ Multi-argument / and -
+ Multi-argument Apply
+ Rationalize
+ Promises
+ Dynamic-Wind
+ Eval
+ Values
+ SRFI
# SRFI-1
o Session Support
+ Repl
+ Quick Print
+ Debug
+ Breakpoints
+ Tracing
o System Interface
+ Directories
+ Transactions
+ CVS
o Extra-SLIB Packages
[...]
Peter Seibel <·····@gigamonkeys.com> writes:
> So my question is, what features that you think
> might usefully be available to you in a Common Lisp environment have
> to be provided by the CL implementation itself.
Environment access for code-walking.
I recently tried working out how do code walking with just
ANSI CL. I thought I could do something clever with the
macroexpand hook, but looking at 8.5 Environments in CLtL2 I
noticed something very odd, you could augment an environment
with a function name. Why bother?
Well think about
(defmacro f (...)...)
(my-walking-macro (...)
(flet ((f (...)...))
(his-walking-macro (...)
(do this)
(setf (f that) something))))
So I need to use macroexpand, passing his-walking-macro an
environment argument describing the lexical environment in
which it is expanded. In particular his-walking-macro is
supposed to pass a lexical environment in which f is
function to get-setf-expansion, which will recognise that f
is a function and not attempt to expand it at all. So
my-walking-macro needs to augment the environment that it
lives in by shadowing the outer macro.
I think that I lose totally unless the implementation
provides a way of augmenting the environment.
The copy of the hyper spec that I've downloaded contains the
reasoning behind
Issue SYNTACTIC-ENVIRONMENT-ACCESS Writeup
Status: Proposal SMALL passed June 89
but simply records
SYNTACTIC-ENVIRONMENT-ACCESS:RETRACTED-MAR91
unexplained.
Being able to write code walkers is one of CL's selling
points. Why was syntactic-environment-access retracted?
One for the faq perhaps?
Alan Crowe
Edinburgh
Scotland
From: Duane Rettig
Subject: Re: Features that can only be provided by the implementation?
Date:
Message-ID: <o0u08j4j2e.fsf@franz.com>
Alan Crowe <····@cawtech.freeserve.co.uk> writes:
> Peter Seibel <·····@gigamonkeys.com> writes:
>> So my question is, what features that you think
>> might usefully be available to you in a Common Lisp environment have
>> to be provided by the CL implementation itself.
>
> Environment access for code-walking.
This is true. However, much of the work has been done, and is
available in an open-source module:
http://www.lispwire.com/entry-proganal-envaccess-des
It worked on many of the current lisps at the time of the
ILC2005 presentation at what I call "level 1" support (i.e.
the module works, but without integration into either the interpreter
or the compiler of the lisp implementation). It also works on
Allegro CL 8.0 at "level 4" support (completely integrated, and our
older proprietary environment implementation is deprecated and/or
non-existent).
If you want something to do here, pick a free lisp and integrate
Environments Access into it. Or I suppose you could just use
Allegro CL, but that's not the topic of this thread :-)
> I recently tried working out how do code walking with just
> ANSI CL. I thought I could do something clever with the
> macroexpand hook, but looking at 8.5 Environments in CLtL2 I
> noticed something very odd, you could augment an environment
> with a function name.
Don't make the mistake of thinking of CLtL2 as being Ansi - in some
areas (especially in this one) it is not close at all.
Why bother?
>
> Well think about
>
> (defmacro f (...)...)
>
> (my-walking-macro (...)
> (flet ((f (...)...))
> (his-walking-macro (...)
> (do this)
> (setf (f that) something))))
>
> So I need to use macroexpand, passing his-walking-macro an
> environment argument describing the lexical environment in
> which it is expanded. In particular his-walking-macro is
> supposed to pass a lexical environment in which f is
> function to get-setf-expansion, which will recognise that f
> is a function and not attempt to expand it at all. So
> my-walking-macro needs to augment the environment that it
> lives in by shadowing the outer macro.
>
> I think that I lose totally unless the implementation
> provides a way of augmenting the environment.
Agreed. That is why the Environments section was included in the
X3J13 committee in the first place; the need was apparent.
> The copy of the hyper spec that I've downloaded contains the
> reasoning behind
>
> Issue SYNTACTIC-ENVIRONMENT-ACCESS Writeup
> Status: Proposal SMALL passed June 89
>
> but simply records
>
> SYNTACTIC-ENVIRONMENT-ACCESS:RETRACTED-MAR91
>
> unexplained.
>
> Being able to write code walkers is one of CL's selling
> points. Why was syntactic-environment-access retracted?
There were many semantic difficulties involved. If you download the
powerpoint presentation from the above-mentioned url, and look at it
using the Notes Page view (I just tried it out on OpenOffice and it
seems to work fine there), you can see some of the issues we had to
deal with in coming up with a workable module. There weren't too many
earth-shattring problems, but there were enough problems with the initial
CLtL2 definition that would have precluded a useful module if that
interface had been followed completely. I think the committee recognized
this, and decided to wait for some implemetational experience. Hopefully,
if this module, which has been proven on Allegro CL, also is ported to
other lisps, then that experience will be substantiated for whatever
standardization is necessary for widespread adoption (including nothing
more than de-facto standardization coupled with the open-source nature
of the module).
> One for the faq perhaps?
One for some implementation efforts; then a FAQ entry would not be
necessary.
--
Duane Rettig ·····@franz.com Franz Inc. http://www.franz.com/
555 12th St., Suite 1450 http://www.555citycenter.com/
Oakland, Ca. 94607 Phone: (510) 452-2000; Fax: (510) 452-0182
Duane Rettig <·····@franz.com> writes:
> Alan Crowe <····@cawtech.freeserve.co.uk> writes:
>
> > Peter Seibel <·····@gigamonkeys.com> writes:
> >> So my question is, what features that you think
> >> might usefully be available to you in a Common Lisp environment have
> >> to be provided by the CL implementation itself.
> >
> > Environment access for code-walking.
>
> This is true. However, much of the work has been done, and is
> available in an open-source module:
>
> http://www.lispwire.com/entry-proganal-envaccess-des
>
> It worked on many of the current lisps at the time of the
> ILC2005 presentation at what I call "level 1" support (i.e.
> the module works, but without integration into either the interpreter
> or the compiler of the lisp implementation). It also works on
> Allegro CL 8.0 at "level 4" support (completely integrated, and our
> older proprietary environment implementation is deprecated and/or
> non-existent).
>
> If you want something to do here, pick a free lisp and integrate
> Environments Access into it. Or I suppose you could just use
> Allegro CL, but that's not the topic of this thread :-)
Strangely enough I've just built SBCL-0.9.11 from source,
hosting the build on SBCL-0.8.17 which is the version in the
ports tree for FreeBSD 5.4. The extended test suite finished
while I was reading c.l.l.
ok
//apparent success (reached end of run-tests.sh normally)
Mon Apr 24 10:23:54 BST 2006
However I don't think my lisp-fu is strong enough yet to try
integrating environment access. I had a different, much
simpler, project in mind. I want to subclass xlib:window and have
(make-instance 'xlib:window ...) work. That too is a topic
for a different thread.
> > I think that I lose totally unless the implementation
> > provides a way of augmenting the environment.
>
> Agreed. That is why the Environments section was included in the
> X3J13 committee in the first place; the need was apparent.
>
> There were many semantic difficulties involved. If you download the
> powerpoint presentation from the above-mentioned url, and look at it
> using the Notes Page view (I just tried it out on OpenOffice and it
> seems to work fine there), you can see some of the issues we had to
> deal with in coming up with a workable module. There weren't too many
> earth-shattring problems, but there were enough problems with the initial
> CLtL2 definition that would have precluded a useful module if that
> interface had been followed completely. I think the committee recognized
> this, and decided to wait for some implemetational experience. Hopefully,
> if this module, which has been proven on Allegro CL, also is ported to
> other lisps, then that experience will be substantiated for whatever
> standardization is necessary for widespread adoption (including nothing
> more than de-facto standardization coupled with the open-source nature
> of the module).
My intuition here is that a code walker does just what one
does as a programmer when when one thinks about ones code in
order to debug it. So these semantic difficulties are worth
understanding simply in order to write code manually. A new task:
"free up 4GB of disk space and install OpenOffice" just got
added to my todo list :-)
Alan Crowe
Edinburgh
Scotland
Some of these suggestions are particular to writing fast numerical
code. I don't necessarily consider this a goal of CL, but these are
absolutely litmus tests for whether or not I can use a CL in my work:
arrays of unboxed basic elements (e.g. double floats)
a compiler that respects declarations and generates code for basic
mathematical operations that is roughly equivlaent to C (Note: I'm not
saying "must generate native code" --- java these days can do numerics
fast enough. CLISP cannot. (I'm talking about floating point math,
CLISP of course has excellent bignum math)).
finalizers
for the FFI, the ability to "pin" items so they're not moved during
garbage collection
Cheers,
rif
rif <···@mit.edu> writes:
> Some of these suggestions are particular to writing fast numerical
> code. I don't necessarily consider this a goal of CL, but these are
> absolutely litmus tests for whether or not I can use a CL in my work:
>
> arrays of unboxed basic elements (e.g. double floats)
>
> a compiler that respects declarations and generates code for basic
> mathematical operations that is roughly equivlaent to C (Note: I'm not
> saying "must generate native code" --- java these days can do numerics
> fast enough. CLISP cannot. (I'm talking about floating point math,
> CLISP of course has excellent bignum math)).
>
One phrase that I see doing the rounds on the internet is
"the right tool for the job". Using the right tool for the
job strikes me as the luxury of the production engineer who
has landed himself a prestigious role on a very high volume
production line. The rest of us must comprise, using dual
purpose tools and general purpose tools, carefully managing
a limited budget for purchasing tools, for initial training
on how to use them, and for training and practice to stay
current with them.
One of the attractions to me of CL is that it has a wide
range of applicability. One might hope to standardise on it
as ones sole general purpose language and save the costs of
a heterogenous computer language environment. Arrays of
unboxed basic elements, such as double floats, are
potentially a big deal. I think they are a key piece of the
performance picture. They open up the possibility of
leaving C out of the mix, even if ones application does some
signal processing.
Code I'm working on directly synthesises music, sample by
sample. It runs adequately fast on CMUCL. I hope to write
some def-instrument macros. If I had to write the
instruments in C to get adequate performance that would kill
off this fun project.
I hope to write some causal inference code. This seems
likely to involve some fairly heavy number crunching for the
Bayesian inference side of things, but closely integrated
with searching acyclic directed graphs. My goal would be to
make the issues of causal inference suficiently concrete to
me that I can understand the intellectual debate about the
extent to which it is truly possible. The additional
complication of splitting such code across two languages
runs directly contrary to my efforts to find an easier way
into a difficult subject.
Fast numerical code is obviously important for number
crunching, but it is also important for language
consolidation. CL implementations that are fast enough to
let C be dropped from a C+Perl project can compete much more
strongly with Perl, because they are offering the
possibility of using one language instead of two.
Alan Crowe
Edinburgh
Scotland
From: Thomas A. Russ
Subject: Re: Features that can only be provided by the implementation?
Date:
Message-ID: <ymiwtddty21.fsf@sevak.isi.edu>
Peter Seibel <·····@gigamonkeys.com> writes:
> So my question is, what features that you think
> might usefully be available to you in a Common Lisp environment have
> to be provided by the CL implementation itself.
Well, this is perhaps a minor issue which would be a nice language-wide
option, but I have often wished to be able to query the current compiler
optimization settings. There have been some times when I would have
liked that information to be available during macro-expansion to allow
for alternate expansions based on, for example, safety level.
I know this was actually discussed during the CL standardization
process, but I don't recall what the argument against making this
information available was.
--
Thomas A. Russ, USC/Information Sciences Institute
From: Duane Rettig
Subject: Re: Features that can only be provided by the implementation?
Date:
Message-ID: <o03bg1mtk2.fsf@franz.com>
···@sevak.isi.edu (Thomas A. Russ) writes:
> Peter Seibel <·····@gigamonkeys.com> writes:
>
>> So my question is, what features that you think
>> might usefully be available to you in a Common Lisp environment have
>> to be provided by the CL implementation itself.
>
> Well, this is perhaps a minor issue which would be a nice language-wide
> option, but I have often wished to be able to query the current compiler
> optimization settings. There have been some times when I would have
> liked that information to be available during macro-expansion to allow
> for alternate expansions based on, for example, safety level.
With Environments Access, you can:
CL-USER(1): (sys:declaration-information 'optimize)
((SAFETY 1) (SPACE 1) (SPEED 1) (COMPILATION-SPEED 1) (DEBUG 2))
CL-USER(2):
Note also that the above example is at top-level, but if you have a
handle on the current environment, you can also do queries, especially
at the macro level, as you have asked for it:
(defmacro foo (... &environment env)
...
(let* ((decl (and env (sys:declaration-information 'optimize env)))
(safety (or (and decl (cadr (assoc 'safety decl))) 1))
(speed (or (and decl (cadr (assoc 'speed decl))) 1)))
(if (and (eq speed 3) (eq safety 0))
`(fast-and-loose ...)
`(slow-and-steady ...))))
This is functionality that requires at least Level 2 support for the
Environments Access module (see the slide 23 of the tutorial:
http://www.lispwire.com/entry-proganal-envaccess-tut) - at Level 1,
the declaration-information function will show a list of optimization
qualities, and augment-environment can change and/or push new quality
values that will shadow the deeper ones, but unless the qualities are
actually reflecting the qualities _as_ the lisp sees it, they would be
wrong. When I was working on the integration of the Environments Access
module into Allegro CL (as part of the module's development) I found the
optimize declaration to be one of the easiest to integrate all the way
to Level 4 (prior to 6.1 we kept these qualities in globals with names
like excl::.speed., etc, but it was trivial, once the mechanism was working,
to move them into the Environments Access module and to remove the
variables).
> I know this was actually discussed during the CL standardization
> process, but I don't recall what the argument against making this
> information available was.
Since it was part of the Environments section that was rejected,
the reasoning is subsumed by the reasoning for the module; I don't think
anyone would have argued against the general environments interface but
for the declaration-information function with the 'optimize argument ...
--
Duane Rettig ·····@franz.com Franz Inc. http://www.franz.com/
555 12th St., Suite 1450 http://www.555citycenter.com/
Oakland, Ca. 94607 Phone: (510) 452-2000; Fax: (510) 452-0182
From: Thomas A. Russ
Subject: Re: Features that can only be provided by the implementation?
Date:
Message-ID: <ymiodyltuf1.fsf@sevak.isi.edu>
Great, thanks Duane.
--
Thomas A. Russ, USC/Information Sciences Institute
From: Pascal Costanza
Subject: Re: Features that can only be provided by the implementation?
Date:
Message-ID: <4b78gsFvvh0eU1@individual.net>
Thomas A. Russ wrote:
> Peter Seibel <·····@gigamonkeys.com> writes:
>
>> So my question is, what features that you think
>> might usefully be available to you in a Common Lisp environment have
>> to be provided by the CL implementation itself.
>
> Well, this is perhaps a minor issue which would be a nice language-wide
> option, but I have often wished to be able to query the current compiler
> optimization settings. There have been some times when I would have
> liked that information to be available during macro-expansion to allow
> for alternate expansions based on, for example, safety level.
Seconded. I would have preferred to have access to the (possibly local)
optimization settings as well in ContextL, but I am now using a
workaround using a symbol macro.
Pascal
--
3rd European Lisp Workshop
July 3-4 - Nantes, France - co-located with ECOOP 2006
http://lisp-ecoop06.bknr.net/
"Peter Seibel" <·····@gigamonkeys.com> wrote in message
···················@gigamonkeys.com...
> So folks have been discussing the desirability of having a way to
> change "Common Lisp" a lot lately. And other folks point out that
> many, or even most, changes that folks want could be built *in* Common
> Lisp. But not all. So my question is, what features that you think
> might usefully be available to you in a Common Lisp environment have
> to be provided by the CL implementation itself. With the help of a few
> folks on #lisp I've got this list so far:
>
> - multithreading
>
> - unicode
>
> - foreign function interface
>
> - extensible streams (e.g. Gray or Simple)
>
> - specific behaviors of standard CL functions allow but not mandated
> by the standard, e.g. a particular semantics for the various
> pathname-related functions.
>
> Any others? (I'm not counting things that could be provided using an
> FFI, such as an interface to sockets.)
>
> -Peter
Continuations maybe ?
Sacha
From: Thomas Schilling
Subject: Re: Features that can only be provided by the implementation?
Date:
Message-ID: <4b7hllF102kjdU1@news.dfncis.de>
Sacha wrote:
> Continuations maybe ?
That does interfere with pretty much other stuff, so you have to be
careful. To name one: conditions (can be captured using continuations
and be thrown multiple times). Also I'm not sure whether there are some
problems with dynamic extend variables. It would be nice, but you might
have to change (not extend) the standard.
-ts
Peter Seibel <·····@gigamonkeys.com> writes:
> So my question is, what features that you think
> might usefully be available to you in a Common Lisp environment have
> to be provided by the CL implementation itself.
o Larger ARRAY-DIMENSION-LIMIT
Some people even complain that 4294967296 (= full 32bit) that certain
configurations of CLISP provide is not enough for 64bit machines (!)
Other limits like 16777215 are already annoyingly low.
o User-defined SEQUENCE extensions, which work with POSITION, FILL and
all the other sequence functions.
CLISP has that. It should document it!
Similar to wanting extensible stream types
o User-defined ARRAY types??
o larger CALL-ARGUMENTS-LIMIT
Why do we have to avoid &REST (arbitrary input) for portable code?
Since the low values that ANSI-CL requires lead some familiars of
comp.lang.lisp to avoid using certain programming styles or
constructs, upping these limits would do good.
Regards,
Jorg Hohle
Telekom/T-Systems Technology Center