Hi,
a question for the gurus (I hope).
How can I modify the stream printing facilities.
Is there any possibility to do this portable (and maybe elegant)
in Common Lisp? Would I need an implementation of "Gray Streams"?
For example I would like to output character data to a stream
using the usual printing facilities (format, princ, write-char,
whatever). The characters then should be mime encoded.
In a OO-based stream implementation I would for example
create a new subclass of an appropriate stream class (how
do I know this class?) and write an :around method
for some basic io-function (say, STREAM-TYO).
Or I might want to write something like:
(defun print-mail-to-stream (stream)
(with-mime-encoding (stream)
(princ "Hi there!" stream)))
What would WITH-MIME-ENCODING look like? Using something
like WITH-OUTPUT-TO-STRING would be unfair.
Is the modification of print routines an unusual feature?
Any ideas?
Greetings,
Rainer Joswig
--
http://www.lavielle.com/~joswig/
From: David Gadbois
Subject: Re: Modifying stream printing ? Possible ? Portable ?
Date:
Message-ID: <6a0eos$l06$1@news3.texas.net>
Rainer Joswig <······@lavielle.com> wrote:
>How can I modify the stream printing facilities. Is there any
>possibility to do this portable (and maybe elegant) in Common Lisp?
>Would I need an implementation of "Gray Streams"?
There is no way to do it in vanilla ANS CL, but Gray's proposal
provide for this capability. There is a copy at
ftp://parcftp.xerox.com/pub/cl/cleanup/mail/stream-definition-by-user.mail.
A number of the commercial implementations provide support for it.
>For example I would like to output character data to a stream using
>the usual printing facilities (format, princ, write-char,
>whatever). The characters then should be mime encoded.
The proposal defines an customizable internal protocol that the usual
printing functions call.
>In a OO-based stream implementation I would for example create a new
>subclass of an appropriate stream class (how do I know this class?)
The proposal defines a bunch of stream classes. Here, you would use
FUNDAMENTAL-CHARACTER-OUTPUT-STREAM.
>and write an :around method for some basic io-function (say,
>STREAM-TYO).
>
>Or I might want to write something like:
>
>(defun print-mail-to-stream (stream)
> (with-mime-encoding (stream)
> (princ "Hi there!" stream)))
>
>What would WITH-MIME-ENCODING look like? Using something
>like WITH-OUTPUT-TO-STRING would be unfair.
Presumably, for a filter type application like this one, you'd do
something like:
(defmacro with-mime-encoding ((stream) &body body)
`(with-open-stream (,stream
(make-instance 'mime-encoding-stream
:original-stream ,stream))
,@body))
where MIME-ENCODING-STREAM is an encapsulating stream class. A
STREAM-WRITE-CHAR method would check output characters to see if they
need to be munged and pass the results on to the encapsulated stream.
Other methods, like STREAM-FINISH-OUTPUT, would just trampoline to the
encapsulated stream's method. An INITIALIZE-INSTANCE could check to
see if the stream is input or output and read or write MIME headers.
--David Gadbois
In article <············@news3.texas.net>, ·······@cyc.com (David Gadbois) wrote:
Thanks to Kent Pitman and David Gadbois for their answers.
> Rainer Joswig <······@lavielle.com> wrote:
> >How can I modify the stream printing facilities. Is there any
> >possibility to do this portable (and maybe elegant) in Common Lisp?
> >Would I need an implementation of "Gray Streams"?
>
> There is no way to do it in vanilla ANS CL, but Gray's proposal
> provide for this capability. There is a copy at
> ftp://parcftp.xerox.com/pub/cl/cleanup/mail/stream-definition-by-user.mail.
> A number of the commercial implementations provide support for it.
I'll look into the proposal.
My LWW 4.0.1 does support it. ACL? I guess MCL does not (but MCL has
mechanisms to extend streams). I'll have to check that.
Is there a portable implementation? The wish to
extend the existing streams seems very natural for me - all
kinds of application would benefit from a portable implementation.
A chapter on stream extensions might be good for an advanced CL book
(MIME encoding, (de)compression, format conversion, base64,
encryption, ...). (Hint, hint.)
Kent, do you think this is something for your "next round of standardization"?
Then ask implementors to support it?
Greetings,
Rainer Joswig
--
http://www.lavielle.com/~joswig/
From: Erik Naggum
Subject: Re: Modifying stream printing ? Possible ? Portable ?
Date:
Message-ID: <3094282731377906@naggum.no>
* Rainer Joswig
| My LWW 4.0.1 does support [the Gray Streams proposal]. ACL?
not having looked at the Gray proposal, I cannot say for sure, yet, but
Allegro 4.3.x for Unix and NT does have a STREAM package and allows
extensions to their streams via generic functions in that package. a
chapter in the User Guide explains how. I'll have to compare the
proposal and that chapter to determine if they are the same or even
sufficiently close.
| Is there a portable implementation? The wish to extend the existing
| streams seems very natural for me - all kinds of application would
| benefit from a portable implementation.
I don't think there can be a portable implementation. the current
streams design does not allow sufficient introspection to be able to do
the kinds of things that would be needed, but I fully agree on the
desirability of such portable extensions.
| A chapter on stream extensions might be good for an advanced CL book
| (MIME encoding, (de)compression, format conversion, base64, encryption,
| ...). (Hint, hint.)
I think it would be useful to see if some of this could be handled by the
EXTERNAL-FORMAT argument to existing streams functions. unfortunately,
only the framework for this functionality is specified by the standard:
all values are implementation-defined. a means of registering various
external formats would be more useful to me than subclassing on streams
because entirely new ways of opening files, etc, would be needed with
those subclasses. use of EXTERNAL-FORMAT could, however, mean that
mixins were added to the stream class at runtime. then we could also
have a list of external formats, to be applied in order.
while we're on the topic of streams, I have long wished for a means to
extract strings directly from the stream (input) buffer by means of a
(movable) marker that denoted the start of such extraction (from which
suitable offsets would apply, of course). if the marker was still in a
buffer, a new buffer would be allocated when needed, instead of just
replacing data in the old buffer. when the marker was moved or disabled,
the buffer(s) it held on to could be recycled. this would allow for much
reduced data copying when scanning input for tokens. (the Lisp reader
would be noticeably faster if it did not have to manage its own token
buffer, for instance.) I can implement this today with READ-SEQUENCE and
private buffer handling, but that still means copying data, although one
could imagine an implementation of READ-SEQUENCE that did not read into a
buffer first.
#:Erik
--
The year "98" was new 1900 years ago. | Help fight MULE in GNU Emacs 20!
Be year 2000 compliant, write "1998"! | http://sourcery.naggum.no/emacs/
In article <················@naggum.no>, Erik Naggum <······@naggum.no> wrote:
> | Is there a portable implementation? The wish to extend the existing
> | streams seems very natural for me - all kinds of application would
> | benefit from a portable implementation.
>
> I don't think there can be a portable implementation. the current
I meant "portable" like in "PCL". ;-)
> I think it would be useful to see if some of this could be handled by the
> EXTERNAL-FORMAT argument to existing streams functions. unfortunately,
> only the framework for this functionality is specified by the standard:
> all values are implementation-defined. a means of registering various
> external formats would be more useful to me than subclassing on streams
> because entirely new ways of opening files, etc, would be needed with
> those subclasses. use of EXTERNAL-FORMAT could, however, mean that
> mixins were added to the stream class at runtime. then we could also
> have a list of external formats, to be applied in order.
Otherwise you would modify streams (for example those you get
with WITH-OPEN-FILE) by combining streams in some way.
Usually what you do in Unix is pipe those streams around.
I would like to see an approach in CL, where you could
define filters, reducers, generators etc. for streams.
Why not being able to write:
(with-open-file (out-stream "foo.gz" :direction :output)
(with-compression (out-stream :method :gzip)
(with-encryption (out-stream :method :md5)
(with-encoding (out-stream :method :uuencode)
(with-open-file (in-stream "foo.text")
(copy-stream in-stream out-stream))))))
And the implementation should do the stream operations
in a way that this is not **overly** slow.
> private buffer handling, but that still means copying data, although one
> could imagine an implementation of READ-SEQUENCE that did not read into a
> buffer first.
You'll need a buffer protocol for streams, too.
From JCMA:
---start---
Here is the stream buffer protocol used in the LispM from my comments in
http:mcl;server;xmactcp.lisp You will find in the same directory a copy of
the mcl buffer stuff, which does not quite measure up to the lispm stuff because
it did not envision writing lots of around , before, and after methods to specialize
stream behavior, for example in the case of translating streams. See the
lispm TCP stream implementation in http:lispm;server;tcp-stream.lisp for
an example of how you can customize TCP streams for HTTP, including
character translation and chunking.
LWW really wants a competent stream buffer protocol because it makes
the tools much more powerful and flexible. It allows users to do lots of
neat stuff -- assuming they have source access.
;;
;; discard-current-output-buffer
;; discard-output-buffer
;; send-current-output-buffer
;; send-output-buffer
;; discard-output-buffer
;; setup-new-output-buffer
;; send-current-output-buffer
;; new-output-buffer
;; stream-output-buffer
;;
;; These are consumer operations
;;
;; read-input-buffer
;; setup-next-input-buffer
;; advance-input-buffer (consumer)
;; discard-current-input-buffer
;;
;; These are producer operations
;;
;; next-input-buffer
;; -- get buffer from tcp
;; discard-input-buffer
;; -- return buffer to tcp
;; discard-current-input-buffer
;; discard-input-buffer
;; setup-next-input-buffer
;; discard-current-input-buffer
;; next-input-buffer
;; stream-input-buffer
---end---
--
http://www.lavielle.com/~joswig/
From: Kent M Pitman
Subject: Re: Modifying stream printing ? Possible ? Portable ?
Date:
Message-ID: <sfw4t2y50rd.fsf@world.std.com>
······@lavielle.com (Rainer Joswig) writes:
> Kent, do you think this is something for your "next round of standardization"?
It might be a reasonable thing to do, but...
> Then ask implementors to support it?
1. Never count on standards to happen in the time frame you need them.
So don't wait to ask implementors.
2. STANDARDIZATION SHOULD NOT BE CONFUSED WITH DESIGN.
(Sorry to shout. It's just that it's important and I wanted
to take anyone up who was reading while also sleeping.)
Standardization groups, at their best, do design ONLY when
there are incompatible theories in different quarters of the
world and we can't decide between them OR there are important
problems that vendors haven't had time to work out a solution
for. It's better for vendors to unilaterally experiment and
then come to the standards table with years of experience so
that the "cast in concrete" process standardization is can be
a less stressful operation. Nothing more scary than making a
standard out of something untried.
So, especially where community consensus of users exists, get all
vendors you can to rally around a de facto standard because (a) that
insulates you from the slowness of real standards and (b) it keeps the
standards process from making mistakes by letting the mistakes be made
by individual vendors earlier while things are not yet formally fixed
and can be tweaked to get them right.
From: Howard R. Stearns
Subject: Re: Modifying stream printing ? Possible ? Portable ?
Date:
Message-ID: <34C9128C.4B13@elwood.com>
Rainer Joswig wrote:
>
> In article <············@news3.texas.net>, ·······@cyc.com (David Gadbois) wrote:
>
> Thanks to Kent Pitman and David Gadbois for their answers.
>
> > Rainer Joswig <······@lavielle.com> wrote:
> > >How can I modify the stream printing facilities. Is there any
> > >possibility to do this portable (and maybe elegant) in Common Lisp?
> > >Would I need an implementation of "Gray Streams"?
> >
> > There is no way to do it in vanilla ANS CL, but Gray's proposal
> > provide for this capability. There is a copy at
> > ftp://parcftp.xerox.com/pub/cl/cleanup/mail/stream-definition-by-user.mail.
> > A number of the commercial implementations provide support for it.
>
> I'll look into the proposal.
> ...
In addition, to the original "Gray streams proposal", the concept made
its way into the CLIM spec as "Appendix D: Common Lisp Streams." CLIM
implemenations are not "required" to support it, but those that do are
"encouraged" to conform. (It's your call as to how well CLIM
implementations conform to the spec.)
The CLIM spec is at ftp://ftp.franz.com/pub/clim/clim-spec/
For what its worth, Eclipse also supports this. The Eclipse CLOS
streams documentation, based closely on the CLIM spec, can be viewed at
http://www.elwood.com/eclipse/extensions.htm#streams
Two things that are not covered by any of this are:
1. buffering protocol
2. OPEN protocol
For example, in the current implementation of Eclipse, OPEN calls the
generic function eclipse::FILE-STREAM-CLASS (pathname truename direction
external-format element-type). One can imagine defining specialized
methods in order to get OPEN to use the stream of your dreams.
Unfortunately, we don't (yet) document this.
Similarly, we don't yet document the various buffering protocols.
From: Kent M Pitman
Subject: Re: Modifying stream printing ? Possible ? Portable ?
Date:
Message-ID: <sfwd8honif7.fsf@world.std.com>
······@lavielle.com (Rainer Joswig) writes:
> How can I modify the stream printing facilities.
> Is there any possibility to do this portable (and maybe elegant)
> in Common Lisp? Would I need an implementation of "Gray Streams"?
To really do it with stream operations requires Gray Streams or a
similar access to the low-level stream system on a per-vendor basis.
That is, CL doesn't provide a way to do this and all "Gray Streams"
are is David Gray's suggested protocol for modifying the stream
printing facilities in a portable way, just like you're asking to.
I think Harlequin's LispWorks 4.0 and higher has Gray Streams. I
would not be surprised to see them in other vendors' offerings as
well. Without this kind of thing, CLIM is a hassle to support.
Of course, if you don't have implementations which offer them,
you can sort of cheat sometimes with kludges like this (untested)
code below. Although I didn't do any testing of this particular
macro, I have many times used this technique and it goes pretty
far toward what you want...
(defmacro with-mime-encoding ((streamvar) &body forms)
(check-type streamvar
(and symbol (not (satisfies constantp)))
"a variable name")
`(write-string
(mime-encode-string
(with-output-to-string (,streamvar) ,@forms))
,streamvar))
(defun mime-format (stream format-string &rest args)
(let* ((uncoded (apply #'format nil format-string args))
(coded (mime-encode-string uncoded)))
(case stream
((nil) coded)
((t) (write-string coded))
(otherwise (write-string coded stream)))))
Of course, you have to write MIME-ENCODE-STRING, etc.
And if the operations involve boundary conditions,
like that stupid code for F at the start of a line, then
(with-mime-encoding (str) (terpri str))
(with-mime-encoding (str) (write-char #\F str))
may or may not figure out that F is in need of coding.
If you always assume the start is at the start of a line,
you may mostly win. As long as it's ok to use that funny
encoding mid-line by accident when you instead do:
(with-mime-encoding (str) (terpri str) (write-string "oops" str))
(with-mime-encoding (str) (write-char #\F str))
I'm not familiar enough with mime codes to know if this
is going to win for you. But in any case, it illustrates one of
the two major ways that this kind of kludge tends to lose.
The other way is if you do simultaneous output to the raw stream
and expect it to be synchronized, as in:
(let ((s stream))
(with-mime-encoding (stream)
(write-char #\a stream)
(write-char #\b s)
(write-char #\c stream)))
which will come out as bac, not abc.