From: ············@gmail.com
Subject: binary file upload with common lisp
Date: 
Message-ID: <1117625221.272583.261400@g44g2000cwa.googlegroups.com>
Hi,

how can I read a binary file from *standard-input* in common lisp? I
need this to support file upload in CGI scripts.

David

From: ············@gmail.com
Subject: Re: binary file upload with CLISP
Date: 
Message-ID: <1117625465.885614.289700@o13g2000cwo.googlegroups.com>
That's not what I wanted to ask, really, sorry. How can I do it with
CLISP; CLISP converts return+linefeed to newline with character
streams, and does not allow to switch to binary mode on standard-input.
From: Pascal Bourguignon
Subject: Re: binary file upload with CLISP
Date: 
Message-ID: <87mzqa9si4.fsf@thalassa.informatimago.com>
·············@gmail.com" <············@gmail.com> writes:

> That's not what I wanted to ask, really, sorry. How can I do it with
> CLISP; CLISP converts return+linefeed to newline with character
> streams, and does not allow to switch to binary mode on standard-input.

You could use:

(with-open-file (bin "/dev/stdin" :direction :input
                                  :element-type (quote (unsigned-byte 8)))
     (loop for byte = (read-byte bin nil nil)
           for count = 0 then (mod (1+ count) 16)
           while byte
           do (when (zerop count) (terpri)) (format t "~2,'0X " byte)))


$ clisp -q -ansi -x '(with-open-file (bin "/dev/stdin" :direction :input
                                 :element-type (quote (unsigned-byte 8)))
     (loop for byte = (read-byte bin nil nil)
           for count = 0 then (mod (1+ count) 16)
           while byte
           do (when (zerop count) (terpri)) (format t "~2,'\''0X " byte)))' \
  < /tmp/vm87963318553
[1]> 
47 49 46 38 39 61 0C 00 0C 00 D5 00 00 FF FF FF 
F7 F7 FF F7 F7 F7 EF EF F7 EF EF EF DE E6 F7 CE 
D6 EF CE D6 E6 C5 D6 E6 B5 C5 EF AD BD E6 A5 BD 
EF 9C B5 EF 94 AD EF 94 AD DE 94 A5 CE 8C A5 D6 
84 A5 E6 8C A5 CE 8C 9C C5 73 94 DE 73 8C C5 63 
8C E6 63 8C DE 63 84 D6 5A 84 DE 63 84 BD 52 84 
D6 5A 7B C5 52 7B D6 52 7B CE 52 7B C5 52 7B BD 
4A 7B CE 4A 73 CE 4A 73 C5 52 73 AD 4A 6B BD 42 
6B C5 42 6B BD 42 63 B5 3A 63 B5 3A 63 AD 3A 5A 
AD 3A 5A A5 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 00 2C 00 00 
00 00 0C 00 0C 00 00 08 8C 00 01 00 08 70 C0 C1 
85 0B 10 10 08 00 B0 90 80 83 0C 16 12 2C 68 D0 
C1 01 01 86 0F 33 0C 10 08 60 81 08 07 02 10 74 
B8 C0 A0 E3 85 81 21 4C 20 80 B0 21 43 04 03 14 
30 08 1C 61 02 02 87 0E 1E 42 E8 34 00 20 81 89 
13 20 38 7C F0 30 62 84 02 00 05 4A A0 48 01 42 
C2 89 12 27 50 F0 2C 91 62 E9 83 03 55 53 A8 D0 
50 41 85 D7 15 0A 1F B0 F0 7A 00 80 0A 16 2C 1E 
34 14 CB A2 2C DA 07 17 05 0A 38 30 81 04 89 09 
07 16 0A 08 08 00 3B 
NIL


You could even read an ASCII HTTP header with *standard-input*, then
open the binary stream to read the rest of the data, since they're
both unbuffered streams.

-- 
__Pascal Bourguignon__                     http://www.informatimago.com/
-----BEGIN GEEK CODE BLOCK-----
Version: 3.12
GCS d? s++:++ a+ C+++ UL++++ P--- L+++ E+++ W++ N+++ o-- K- w--- 
O- M++ V PS PE++ Y++ PGP t+ 5+ X++ R !tv b+++ DI++++ D++ 
G e+++ h+ r-- z? 
------END GEEK CODE BLOCK------
From: ············@gmail.com
Subject: Re: binary file upload with CLISP
Date: 
Message-ID: <1117629423.926513.283730@z14g2000cwz.googlegroups.com>
Pascal Bourguignon wrote:
> ·············@gmail.com" <············@gmail.com> writes:
>
> > That's not what I wanted to ask, really, sorry. How can I do it with
> > CLISP; CLISP converts return+linefeed to newline with character
> > streams, and does not allow to switch to binary mode on standard-input.
>
> You could use:
>
> (with-open-file (bin "/dev/stdin" :direction :input
>                                   :element-type (quote (unsigned-byte 8)))
>      (loop for byte = (read-byte bin nil nil)
>            for count = 0 then (mod (1+ count) 16)
>            while byte
>            do (when (zerop count) (terpri)) (format t "~2,'0X " byte)))

Great, thanks. That's what I wanted to know. Will use.

David
From: Joerg Hoehle
Subject: Re: binary file upload with CLISP
Date: 
Message-ID: <u3bs2owq9.fsf@users.sourceforge.net>
Pascal Bourguignon <···@informatimago.com> writes:
> ·············@gmail.com" <············@gmail.com> writes:
> > That's not what I wanted to ask, really, sorry. How can I do it with
> > CLISP; CLISP converts return+linefeed to newline with character
> > streams, and does not allow to switch to binary mode on standard-input.

> You could use:
> (with-open-file (bin "/dev/stdin" :direction :input
>                                   :element-type (quote (unsigned-byte 8)))

I thought such unportable Linux device tricks have been unnecessary
for some time now? While clisp would not allow to switch to binary
mode in an interactive terminal, it would recognize being connected
via special files and allow the switch then (like when you redirect
stdin from a file).

Did I miss something?

	Jorg Hohle
Telekom/T-Systems Technology Center
From: ············@gmail.com
Subject: Re: binary file upload with CLISP
Date: 
Message-ID: <1117649666.560123.208800@z14g2000cwz.googlegroups.com>
> > You could use:
> > (with-open-file (bin "/dev/stdin" :direction :input
> >                                   :element-type (quote (unsigned-byte 8)))

> I thought such unportable Linux device tricks have been unnecessary
> for some time now? While clisp would not allow to switch to binary
> mode in an interactive terminal, it would recognize being connected
> via special files and allow the switch then (like when you redirect
> stdin from a file).
>
> Did I miss something?

1) /dev/stdin, /dev/stdout are available under FreebSD, Mac OS X,
Linux.
2) isatty never says truth. Or almost never.
3) In particular, it lies to CLISP under Mac OS X.

RFC 2388 is ill-designed, that's right. The sad fact is that
stdin/stdout in most lisps are even more broken than RFC2388 is;
requiring to resort to hacks (:latin-1 :eol-style :lf). The best case
is, actually, CLISP, which would allow to switch between binary and
text modes, but it is too smart. It should not use isatty, it is
broken. 

David
From: ············@gmail.com
Subject: Re: binary file upload with CLISP
Date: 
Message-ID: <1117651453.691563.263070@o13g2000cwo.googlegroups.com>
>
> RFC 2388 is ill-designed, that's right. The sad fact is that
> stdin/stdout in most lisps are even more broken than RFC2388 is;
> requiring to resort to hacks (:latin-1 :eol-style :lf). The best case
> is, actually, CLISP, which would allow to switch between binary and
> text modes, but it is too smart. It should not use isatty, it is
> broken.

*** - (SETF STREAM-ELEMENT-TYPE) on #<IO TERMINAL-STREAM> is illegal
[Wed Jun  1 23:41:32 2005] [error] [client 127.0.0.1] Premature end of
script headers: /Users/dvd/Sites/cgi-test.cgi

from my Apache log. And the code that causes it is (setf
(stream-element-type *standard-output*) 'unsigned-byte).
From: Christopher C. Stacy
Subject: Re: binary file upload with CLISP
Date: 
Message-ID: <u8y1t3mgm.fsf@news.dtpq.com>
It seems that the problem is that *STANDARD-OUTPUT* is getting
bound to wrong kind of stream.  What you want is for Lisp to 
connect /dev/stdout to some other kind of stream.

You mention that clisp is calling isatty() on the channel to see
if it should do this, but apparently this gives the wrong answer.
In the case of an HTTP program, the stream should be a "bivalent"
stream from the "Gray Streams" or "Simple Streams" package.
But there's no way for Lisp to know the type of stream a priori.
Even if isatty() is not misleading, and this really is a terminal,
the default kind of terminal stream might not be what's desired.

One possibility is for your application to simply rebind
*STANDARD-OUTPUT* to an appropriate stream after Lisp starts up.
Can that be done without messing up your application?  
I can imagine there might be a problem if Lisp automatically 
goes and tries to do random OS calls on the channel.
Also, we are just as well talking about *STANDARD-INPUT* here,
so maybe that could be cause input buffering problems.

My suggestion is that when Lisp starts up, it should call a 
Lisp function (in the same binding context as the application 
toplevel will run) that receives the OS channels, so it can
set the standard stream variables to appropriate stream objects.
(It could also set other variables, or hack around with the
IO channels in other ways.) This Lisp function would be a
user-supplied hook that may or may not have been defined; 
if it's not there, a standard function is called by default 
and does whatever.  Notice that we expect the hook to be there
at startup; this has implications for Lisp startup/delivery.
This also naturally implies that the appropriate stream classes 
are defined and documented.  There may be some OS-specific functions
that are also available to the hook programmer (for closing channels, 
doing isatty(), certain ioctl(), etc.)

The details of what happens in this hook are highly implementation
dependant and could be very different across operating systems.
Since it doesn't exist now, it would be nice if they all called
it the same thing, but that's just one more #+WHATEVER if they don't.
From: ············@gmail.com
Subject: Re: binary file upload with CLISP
Date: 
Message-ID: <1117655082.411939.220930@g44g2000cwa.googlegroups.com>
 One possibility is for your application to simply rebind
> *STANDARD-OUTPUT* to an appropriate stream after Lisp starts up.

Under Mac OS X, I cannot re-open standard-output when it is open.
Lispworks allows to just
close the stream without closing the descriptor (what I am doing in
fact, rebinding the descriptor
to a bivalent stream). CLISP closes the descriptor, and there is no way
to rebind.
From: Marcin 'Qrczak' Kowalczyk
Subject: Re: binary file upload with CLISP
Date: 
Message-ID: <878y1tsuzq.fsf@qrnik.zagroda>
······@news.dtpq.com (Christopher C. Stacy) writes:

> One possibility is for your application to simply rebind
> *STANDARD-OUTPUT* to an appropriate stream after Lisp starts up.

No, it's the other way around: he wants to bind some possibly other
Lisp variable to raw standard output of the process. The problem is
in obtaining the stream.

Meaningfulness of binary standard I/O is OS-dependent. On Unix a
process usually receives 3 preopened byte streams (most processes
blindly assume that they are open, even though the program which has
called us might have closed them).

ISO/ANSI C doesn't provide access to these streams, even though it
provides binary I/O in general. It only provides standard character
streams.

Unix API for C does, because it states that text streams and binary
streams are the same.

CL assumptions are similar to ISO/ANSI C. The problem is that even on
Unix Lisp character streams and byte streams may be different (because
Lisp distinguishes characters from bytes, and because it might
automatically perform newline conversion on input which is not done
by Unix C).

Opening "/dev/stdin" as a binary file is not as portable as it could be,
because there are probably OSes with binary standard I/O but without
"/dev/*" device names (I guess Windows is an example).

-- 
   __("<         Marcin Kowalczyk
   \__/       ······@knm.org.pl
    ^^     http://qrnik.knm.org.pl/~qrczak/
From: ············@gmail.com
Subject: Re: binary file upload with CLISP
Date: 
Message-ID: <1117658033.968617.319020@g47g2000cwa.googlegroups.com>
> Opening "/dev/stdin" as a binary file is not as portable as it could be,
> because there are probably OSes with binary standard I/O but without
> "/dev/*" device names (I guess Windows is an example).


I would be happy with any non-portable, but reasonable way.
Unfortunately, there is not any,
short of hacking and misuse.
From: Christopher C. Stacy
Subject: Re: binary file upload with CLISP
Date: 
Message-ID: <u4qch3j2d.fsf@news.dtpq.com>
Marcin 'Qrczak' Kowalczyk <······@knm.org.pl> writes:

> ······@news.dtpq.com (Christopher C. Stacy) writes:
> 
> > One possibility is for your application to simply rebind
> > *STANDARD-OUTPUT* to an appropriate stream after Lisp starts up.
> 
> No, it's the other way around: he wants to bind some possibly other
> Lisp variable to raw standard output of the process. The problem is
> in obtaining the stream.

My proposed scheme lets you do that, too.

> [snipped: stuff about non-standards, C, CL, Unix, ...]

As I said this is all "implementation dependant".

> Opening "/dev/stdin" as a binary file is not as portable as it could be,
> because there are probably OSes with binary standard I/O but without
> "/dev/*" device names (I guess Windows is an example).

My proposal does not involve the user opening /dev/stdin.
From: ············@gmail.com
Subject: Re: binary file upload with CLISP
Date: 
Message-ID: <1117706298.981323.18300@g49g2000cwa.googlegroups.com>
I was too quick about isatty, it does say truth on Mac OS X. The
problem is, CLISP's diagnostics has nothing to do with terminal, except
that stdin and stdout are TERMINAL-IO streams, even if TERMINAL is not
a terminal.

And standard_input, standard_output being always created as streams of
type terminal_stream (regular_handle_p counts any non-file-system file
handle as a terminal, including pipes and sockets) are not allowed to
have their types changed.

That means that unless you rebind *standard-input*/*standard-output* in
a non-portable way, you can't get binary input in CLISP to work.

I've ended up cutting CRLF smartness off CLISP so that I don't have to
fix the terminal-io stuff.  The Right Thing is the worst thing ever
happening to software. I just want my CR in Latin-1/Unix, it is there,
in the encoding.


David
From: Bruno Haible
Subject: Re: binary file upload with CLISP
Date: 
Message-ID: <d7mqg6$g08$2@laposte.ilog.fr>
David Tolpin wrote:

> /dev/stdin, /dev/stdout are available under FreebSD, Mac OS X, Linux.

They are available not only on Linux, FreeBSD, MacOS X, but also on
Solaris, NetBSD and OpenBSD. I.e. on every Unix except HP-UX and Cygwin.

Also, some GNU programs accept /dev/stdin and /dev/stdout even on
systems that don't have it in the filesystem.  clisp could do the same.

Bruno
From: Thomas F. Burdick
Subject: Re: binary file upload with CLISP
Date: 
Message-ID: <xcvll5t5421.fsf@conquest.OCF.Berkeley.EDU>
Bruno Haible <·····@clisp.org> writes:

> David Tolpin wrote:
> 
> > /dev/stdin, /dev/stdout are available under FreebSD, Mac OS X, Linux.
> 
> They are available not only on Linux, FreeBSD, MacOS X, but also on
> Solaris, NetBSD and OpenBSD. I.e. on every Unix except HP-UX and Cygwin.
> 
> Also, some GNU programs accept /dev/stdin and /dev/stdout even on
> systems that don't have it in the filesystem.  clisp could do the same.

Wouldn't it be cleaner to provide these as binary streams in the EXT
package, as something like *BINARY-STDIN*, *BINARY-STDOUT*, and
*BINARY-STDERR* ?  That way the user doesn't have to figure out the
correct way to get them on that OS, and it wouldn't involve any
grossness like magical filenames that the implementation intercepts.

-- 
           /|_     .-----------------------.                        
         ,'  .\  / | Free Mumia Abu-Jamal! |
     ,--'    _,'   | Abolish the racist    |
    /       /      | death penalty!        |
   (   -.  |       `-----------------------'
   |     ) |                               
  (`-.  '--.)                              
   `. )----'                               
From: Marco Antoniotti
Subject: Re: binary file upload with CLISP
Date: 
Message-ID: <0GJne.52$mi7.84810@typhoon.nyu.edu>
Thomas F. Burdick wrote:
> Bruno Haible <·····@clisp.org> writes:
> 
> 
>>David Tolpin wrote:
>>
>>
>>>/dev/stdin, /dev/stdout are available under FreebSD, Mac OS X, Linux.
>>
>>They are available not only on Linux, FreeBSD, MacOS X, but also on
>>Solaris, NetBSD and OpenBSD. I.e. on every Unix except HP-UX and Cygwin.
>>
>>Also, some GNU programs accept /dev/stdin and /dev/stdout even on
>>systems that don't have it in the filesystem.  clisp could do the same.
> 
> 
> Wouldn't it be cleaner to provide these as binary streams in the EXT
> package,

OT.  The notion of the "EXT" package makes me cringe.

Cheers
--
Marco
From: Pascal Bourguignon
Subject: Re: binary file upload with CLISP
Date: 
Message-ID: <8764ww8pu5.fsf@thalassa.informatimago.com>
Marco Antoniotti <·······@cs.nyu.edu> writes:

> Thomas F. Burdick wrote:
>> Bruno Haible <·····@clisp.org> writes:
>> 
>>>David Tolpin wrote:
>>>
>>>
>>>>/dev/stdin, /dev/stdout are available under FreebSD, Mac OS X, Linux.
>>>
>>>They are available not only on Linux, FreeBSD, MacOS X, but also on
>>>Solaris, NetBSD and OpenBSD. I.e. on every Unix except HP-UX and Cygwin.
>>>
>>>Also, some GNU programs accept /dev/stdin and /dev/stdout even on
>>>systems that don't have it in the filesystem.  clisp could do the same.
>> Wouldn't it be cleaner to provide these as binary streams in the EXT
>> package,
>
> OT.  The notion of the "EXT" package makes me cringe.

Well, all implementations have extension packages.  Better put them in
these distinct packages and let us only (:use "COMMON-LISP") in our
portable packages.

The only thing is that you don't have to declare in defpackage what
packages are used directly in the "body" of the package:

(defpackage "EXAMPLE" (:use "COMMON-LISP")) ; me .oO(Nice a portable package!)
(in-package "EXAMPLE")

;; ...
;; 1000 lines of portable code
;; ...

;; suddenly:

(defun f ()
  (ext:non-portable-stuff)) ; me .oO[ Opps! Not portable :-( ]

;; ...
;; 1000 lines of portable code
;; ...

So you have to write tools that scan the whole sources for strange
packages.  On the other hand, since lisp is so dynamical, it can use
packages known only at run-time anyways, so it may not be so important...


To compare apples with oranges, compare the handling of modules in
Modula-3 (with the signaling of safe and unsafe stuff, etc), with
packages in Lisp.

-- 
__Pascal Bourguignon__                     http://www.informatimago.com/
The rule for today:
Touch my tail, I shred your hand.
New rule tomorrow.
From: Thomas F. Burdick
Subject: Re: binary file upload with CLISP
Date: 
Message-ID: <xcvis0v6gu6.fsf@conquest.OCF.Berkeley.EDU>
Marco Antoniotti <·······@cs.nyu.edu> writes:

> Thomas F. Burdick wrote:
> > Bruno Haible <·····@clisp.org> writes:
> > 
> > 
> >>David Tolpin wrote:
> >>
> >>
> >>>/dev/stdin, /dev/stdout are available under FreebSD, Mac OS X, Linux.
> >>
> >>They are available not only on Linux, FreeBSD, MacOS X, but also on
> >>Solaris, NetBSD and OpenBSD. I.e. on every Unix except HP-UX and Cygwin.
> >>
> >>Also, some GNU programs accept /dev/stdin and /dev/stdout even on
> >>systems that don't have it in the filesystem.  clisp could do the same.
> > 
> > 
> > Wouldn't it be cleaner to provide these as binary streams in the EXT
> > package,
> 
> OT.  The notion of the "EXT" package makes me cringe.

Why?  I don't like having a grab-bag of everything, like MCL's CCL
package, but for little miscelaneous extensions ... where would you
put these symbols?  Having a CLISP-IO package would make sense if
there were a lot of symbols to put in there, but if it's only three,
that sounds worse to me than the EXT package.

-- 
           /|_     .-----------------------.                        
         ,'  .\  / | Free Mumia Abu-Jamal! |
     ,--'    _,'   | Abolish the racist    |
    /       /      | death penalty!        |
   (   -.  |       `-----------------------'
   |     ) |                               
  (`-.  '--.)                              
   `. )----'                               
From: Marco Antoniotti
Subject: Re: binary file upload with CLISP
Date: 
Message-ID: <Tt_ne.53$mi7.85994@typhoon.nyu.edu>
Thomas F. Burdick wrote:
> Marco Antoniotti <·······@cs.nyu.edu> writes:
> 
> 
>>Thomas F. Burdick wrote:
>>
>>>Bruno Haible <·····@clisp.org> writes:
>>>
>>>
>>>
>>>>David Tolpin wrote:
>>>>
>>>>
>>>>
>>>>>/dev/stdin, /dev/stdout are available under FreebSD, Mac OS X, Linux.
>>>>
>>>>They are available not only on Linux, FreeBSD, MacOS X, but also on
>>>>Solaris, NetBSD and OpenBSD. I.e. on every Unix except HP-UX and Cygwin.
>>>>
>>>>Also, some GNU programs accept /dev/stdin and /dev/stdout even on
>>>>systems that don't have it in the filesystem.  clisp could do the same.
>>>
>>>
>>>Wouldn't it be cleaner to provide these as binary streams in the EXT
>>>package,
>>
>>OT.  The notion of the "EXT" package makes me cringe.
> 
> 
> Why?  I don't like having a grab-bag of everything, like MCL's CCL
> package, but for little miscelaneous extensions ... where would you
> put these symbols?  Having a CLISP-IO package would make sense if
> there were a lot of symbols to put in there, but if it's only three,
> that sounds worse to me than the EXT package.

Because it is never "just three symbols" and because eventually you end 
up with (* 3 N) totally unrelated things in it. :)  I understand that 
legacy demands the availability of a grab-bag package.  However, I would 
still build it up from smaller ones.  It would be an interesting 
exercise to take CMUCL/SBCL EXT package and break it up.  Then rebuild 
EXT by doing a bunch of :USE.

Cheers
--
Marco
From: Christophe Rhodes
Subject: Re: binary file upload with CLISP
Date: 
Message-ID: <sqk6lbmpsq.fsf@cam.ac.uk>
Marco Antoniotti <·······@cs.nyu.edu> writes:

> Because it is never "just three symbols" and because eventually you
> end up with (* 3 N) totally unrelated things in it. :)  I understand
> that legacy demands the availability of a grab-bag package.  However,
> I would still build it up from smaller ones.  It would be an
> interesting exercise to take CMUCL/SBCL EXT package and break it up.
> Then rebuild EXT by doing a bunch of :USE.

What would you actually gain from such an exercise?

Christophe
From: Marco Antoniotti
Subject: Re: binary file upload with CLISP
Date: 
Message-ID: <f25oe.57$mi7.87136@typhoon.nyu.edu>
Christophe Rhodes wrote:
> Marco Antoniotti <·······@cs.nyu.edu> writes:
> 
> 
>>Because it is never "just three symbols" and because eventually you
>>end up with (* 3 N) totally unrelated things in it. :)  I understand
>>that legacy demands the availability of a grab-bag package.  However,
>>I would still build it up from smaller ones.  It would be an
>>interesting exercise to take CMUCL/SBCL EXT package and break it up.
>>Then rebuild EXT by doing a bunch of :USE.
> 
> 
> What would you actually gain from such an exercise?

Clarity?  Modularity?  Better self-documenting features?  Why having 
packages if you don't use them when it is reasonable to?

Not that I advocate getting to the extreme of having one function or two 
per package, but just by eyeballing the list of 365 symbols in 
CMUCL(18e) EXT there are at least four categories of things that can be 
extracted: numerics, stream stuff, environment control switches, data 
structure thingies.  There may be more.

So, the bottom line is that you do not loose anything by doing the 
exercise and may gain something.  Unless you are of the school that a 
big cauldron is all you need, and therefore C and Scheme are all you 
really want (which may or may not be true depending on the 
circumstances).  But at that point why not stick all of EXT into CL?

Cheers
--
Marco
From: Christophe Rhodes
Subject: Re: binary file upload with CLISP
Date: 
Message-ID: <sqy89qczfm.fsf@cam.ac.uk>
Marco Antoniotti <·······@cs.nyu.edu> writes:

> Christophe Rhodes wrote:
>> What would you actually gain from such an exercise?
>
> Clarity?  Modularity?  Better self-documenting features?  Why having
> packages if you don't use them when it is reasonable to?

That is begging the question.  You say it's a priori reasonable, but
that's what I'm asking you to demonstrate.  I do not think multiple
extension packages bring any of your claimed benefits.

> Not that I advocate getting to the extreme of having one function or
> two per package, but just by eyeballing the list of 365 symbols in
> CMUCL(18e) EXT there are at least four categories of things that can
> be extracted: numerics, stream stuff, environment control switches,
> data structure thingies.  There may be more.

Right.  There may also be environmental control switches which affect
numerics, data structure thingies for streams, and other such
cross-product effects.  What package does SET-FLOATING-POINT-MODE live
in, environment switches or numerics?

> So, the bottom line is that you do not loose anything by doing the
> exercise and may gain something.  Unless you are of the school that a
> big cauldron is all you need, and therefore C and Scheme are all you
> really want (which may or may not be true depending on the
> circumstances).  

This characterization of my view is pretty inaccurate, for what it's
worth; there is clearly value in splitting off objectively
self-contained extensions (the MOP, Gray streams, and the like) into
their own namespace.  However, I claim that where the extension is not
self-contained, there is no gain at all in inventing a distinct
namespace for it.  And of course this is completely orthogonal to
providing the package facility to the user of the implementation,
which is unaffected by EXT's package discipline: so invokation of C
and Scheme's lack of namespacing is a simple strawman.

> But at that point why not stick all of EXT into CL?

Well, because it's not allowed, right now.  On the other hand, I think
that things which live in an implementation's EXT package, whatever
it's called, are likely to be plausible candidates for "common
extensions" in agreements between lisp vendors.

Christophe
From: lin8080
Subject: Re: binary file upload with CLISP
Date: 
Message-ID: <42A2366A.EED343AE@freenet.de>
Marco Antoniotti schrieb:

Hallo

> Not that I advocate getting to the extreme of having one function or two
> per package, but just by eyeballing the list of 365 symbols in
> CMUCL(18e) EXT there are at least four categories of things that can be
> extracted: numerics, stream stuff, environment control switches, data
> structure thingies.  There may be more.

Well in this case you can copy the java class-tree. Maybe some adds or
renames are requiered.

stefan
From: Pascal Bourguignon
Subject: Re: binary file upload with CLISP
Date: 
Message-ID: <87acm75m38.fsf@thalassa.informatimago.com>
Christophe Rhodes <·····@cam.ac.uk> writes:
> Marco Antoniotti <·······@cs.nyu.edu> writes:
>
>> Because it is never "just three symbols" and because eventually you
>> end up with (* 3 N) totally unrelated things in it. :)  I understand
>> that legacy demands the availability of a grab-bag package.  However,
>> I would still build it up from smaller ones.  It would be an
>> interesting exercise to take CMUCL/SBCL EXT package and break it up.
>> Then rebuild EXT by doing a bunch of :USE.
>
> What would you actually gain from such an exercise?

I had the tendency of making small packages (eg. one class, one
package).  This doesn't work.  You spend more time managing packages
than real programming.

Packages should be rather big.  
Almost: one programmer, one package. Or one application, one package.

CL exports 972 symbols.
A POSIX package would export around 1500 symbols.  This would be much
more tractable than making one package per .h for example...
(Even if 1500 exports sounds like a maximum for a package size).


-- 
__Pascal Bourguignon__                     http://www.informatimago.com/

Nobody can fix the economy.  Nobody can be trusted with their finger
on the button.  Nobody's perfect.  VOTE FOR NOBODY.
From: Bruno Haible
Subject: Re: binary file upload with CLISP
Date: 
Message-ID: <d7q1ib$fhi$1@laposte.ilog.fr>
Thomas F. Burdick wrote:
>> Also, some GNU programs accept /dev/stdin and /dev/stdout even on
>> systems that don't have it in the filesystem.  clisp could do the same.
>
> Wouldn't it be cleaner to provide these as binary streams in the EXT
> package, as something like *BINARY-STDIN*, *BINARY-STDOUT*, and
> *BINARY-STDERR* ?  That way the user doesn't have to figure out the
> correct way to get them on that OS, and it wouldn't involve any
> grossness like magical filenames that the implementation intercepts.

I like the idea. Thanks.

Would other Lisp implementors want to do the same (hi SBCL people!)?

            Bruno
From: Christopher C. Stacy
Subject: Re: binary file upload with CLISP
Date: 
Message-ID: <uvf4v2pqd.fsf@news.dtpq.com>
Bruno Haible <·····@clisp.org> writes:

> Thomas F. Burdick wrote:
> >> Also, some GNU programs accept /dev/stdin and /dev/stdout even on
> >> systems that don't have it in the filesystem.  clisp could do the same.
> >
> > Wouldn't it be cleaner to provide these as binary streams in the EXT
> > package, as something like *BINARY-STDIN*, *BINARY-STDOUT*, and
> > *BINARY-STDERR* ?  That way the user doesn't have to figure out the
> > correct way to get them on that OS, and it wouldn't involve any
> > grossness like magical filenames that the implementation intercepts.
> 
> I like the idea. Thanks.

This presupposes that a "raw" stream is a "binary" stream.
"Binary and binary, what is binary?"
From: Marcin 'Qrczak' Kowalczyk
Subject: Re: binary file upload with CLISP
Date: 
Message-ID: <87r7fjm881.fsf@qrnik.zagroda>
······@news.dtpq.com (Christopher C. Stacy) writes:

> This presupposes that a "raw" stream is a "binary" stream.
> "Binary and binary, what is binary?"

The CGI protocol relies on the existence of streams of 8-bit bytes
called stdin and stdout. A practical programming language must give
access to them on systems where they exist.

-- 
   __("<         Marcin Kowalczyk
   \__/       ······@knm.org.pl
    ^^     http://qrnik.knm.org.pl/~qrczak/
From: Christopher C. Stacy
Subject: Re: binary file upload with CLISP
Date: 
Message-ID: <uacm63l3i.fsf@news.dtpq.com>
Marcin 'Qrczak' Kowalczyk <······@knm.org.pl> writes:

> ······@news.dtpq.com (Christopher C. Stacy) writes:
> 
> > This presupposes that a "raw" stream is a "binary" stream.
> > "Binary and binary, what is binary?"
> 
> The CGI protocol relies on the existence of streams of 8-bit bytes
> called stdin and stdout. A practical programming language must give
> access to them on systems where they exist.

Are all Lisp applications that are not the REPL/listener
based on streams of octets?
From: Edi Weitz
Subject: Re: binary file upload with CLISP
Date: 
Message-ID: <u64wx2d0n.fsf@agharta.de>
On 2 Jun 2005 11:28:06 GMT, Bruno Haible <·····@clisp.org> wrote:

> They are available not only on Linux, FreeBSD, MacOS X, but also on
> Solaris, NetBSD and OpenBSD. I.e. on every Unix except HP-UX and
> Cygwin.

There are certainly more Unixes than those - AI/X, Irix, OSF/1,
Tru64, Reliant Unix, etc.  Here are some more:

  <http://www.levenez.com/unix/>

-- 

Lisp is not dead, it just smells funny.

Real email: (replace (subseq ·········@agharta.de" 5) "edi")
From: Pascal Bourguignon
Subject: Re: binary file upload with CLISP
Date: 
Message-ID: <87acm9agnb.fsf@thalassa.informatimago.com>
Joerg Hoehle <······@users.sourceforge.net> writes:
> Pascal Bourguignon <···@informatimago.com> writes:
>> ·············@gmail.com" <············@gmail.com> writes:
>> > That's not what I wanted to ask, really, sorry. How can I do it with
>> > CLISP; CLISP converts return+linefeed to newline with character
>> > streams, and does not allow to switch to binary mode on standard-input.
>
>> You could use:
>> (with-open-file (bin "/dev/stdin" :direction :input
>>                                   :element-type (quote (unsigned-byte 8)))
>
> I thought such unportable Linux device tricks have been unnecessary
> for some time now? While clisp would not allow to switch to binary
> mode in an interactive terminal, it would recognize being connected
> via special files and allow the switch then (like when you redirect
> stdin from a file).
>
> Did I miss something?

In an xterm:

[···@thalassa tmp]$ clisp -ansi -q -x ' (setf (STREAM-ELEMENT-TYPE *standard-input*) (quote (unsigned-byte 8)))' < /tmp/aa
[1]> 
*** - (SETF STREAM-ELEMENT-TYPE) on #<INPUT STRING-INPUT-STREAM> is illegal

[···@thalassa tmp]$ clisp --version
[1]> GNU CLISP 2.33.83 (2005-03-14) (built 3324433689) (memory 3324434889)
Software: GNU C 3.3 20030226 (prerelease) (SuSE Linux) 
gcc -W -Wswitch -Wcomment -Wpointer-arith -Wimplicit -Wreturn-type  -Wno-sign-compare -O2 -fexpensive-optimizations -DUNICODE -DDYNAMIC_FFI -DNO_SIGSEGV -I. -x none libcharset.a libavcall.a libcallback.a -lreadline -lncurses -ldl    -L/usr/X11R6/lib -lX11
SAFETY=0 HEAPCODES LINUX_NOEXEC_HEAPCODES SPVW_BLOCKS SPVW_MIXED TRIVIALMAP_MEMORY
Features: 
(WILDCARD CLX-MIT-R5 CLX-MIT-R4 XLIB CLX CLX-LITTLE-ENDIAN REGEXP SYSCALLS LOOP
 COMPILER CLOS MOP CLISP ANSI-CL COMMON-LISP LISP=CL INTERPRETER SOCKETS
 GENERIC-STREAMS LOGICAL-PATHNAMES SCREEN FFI GETTEXT UNICODE
 BASE-CHAR=CHARACTER PC386 UNIX)
Installation directory: /usr/local/languages/clisp-2.33.83/lib/clisp/
User language: ENGLISH
Machine: I686 (I686) thalassa.informatimago.com [62.93.174.79]

[···@thalassa tmp]$ clisp -ansi -q                                             
[1]> (setf (STREAM-ELEMENT-TYPE *standard-input*) (quote (unsigned-byte 8)))

*** - (SETF STREAM-ELEMENT-TYPE) on #<IO TERMINAL-STREAM> is illegal
The following restarts are available:
ABORT          :R1      ABORT
Break 1 [2]> (ext:quit)


In an inferior-lisp in emacs:

[1]> (setf (STREAM-ELEMENT-TYPE *standard-input*) (quote (unsigned-byte 8)))

*** - (SETF STREAM-ELEMENT-TYPE) on #<IO TERMINAL-STREAM> is illegal
The following restarts are available:
ABORT          :R1      ABORT
Break 1 [2]> 


-- 
__Pascal Bourguignon__                     http://www.informatimago.com/
Our enemies are innovative and resourceful, and so are we. They never
stop thinking about new ways to harm our country and our people, and
neither do we. -- Georges W. Bush
From: Bruno Haible
Subject: Re: binary file upload with CLISP
Date: 
Message-ID: <d7mptm$g08$1@laposte.ilog.fr>
Pascal Bourguignon tried:
>
> [···@thalassa tmp]$ clisp -ansi -q                                             
> [1]> (setf (STREAM-ELEMENT-TYPE *standard-input*) (quote (unsigned-byte 8)))
>
> *** - (SETF STREAM-ELEMENT-TYPE) on #<IO TERMINAL-STREAM> is illegal

There's no way that you could change the element type of *terminal-io*,
*standard-input*, *debug-io* etc. to binary.  That's because the Lisp
implementation relies on these streams for signalling errors and
similar.

Like other people, I recommend to not touch the predefined
*standard-input* or *terminal-io* stream, but at most rebind some
stream variable:

clisp -q -x '(let ((*standard-input*
                     (open "/dev/stdin" :direction :input
                                        :element-type (quote
                                                        (unsigned-byte 8)))))
               (loop
                 (let ((x (read-byte *standard-input* nil nil)))
                   (if x (format t "~2,VX " #\0 x) (return)))))'

Bruno
From: ············@gmail.com
Subject: Re: binary file upload with CLISP
Date: 
Message-ID: <1117712944.260429.49800@g14g2000cwa.googlegroups.com>
> > *** - (SETF STREAM-ELEMENT-TYPE) on #<IO TERMINAL-STREAM> is illegal
>
> There's no way that you could change the element type of *terminal-io*,
> *standard-input*, *debug-io* etc. to binary.  That's because the Lisp
> implementation relies on these streams for signalling errors and
> similar.

How does CLISP rely on *standard-input* for signalling errors? Why
CLISP reports that *standard-input* is a terminal stream when it is a
pipe?

>
> Like other people, I recommend to not touch the predefined
> *standard-input* or *terminal-io* stream, but at most rebind some
> stream variable:
>
> clisp -q -x '(let ((*standard-input*
>                      (open "/dev/stdin" :direction :input
>                                         :element-type (quote
>                                                         (unsigned-byte 8)))))
>                (loop
>                  (let ((x (read-byte *standard-input* nil nil)))
>                    (if x (format t "~2,VX " #\0 x) (return)))))'

This does not save from hacking, unfortunately; CLISP always calls
fresh-line on standard output, and never sets argv_verbose to zero if
used as a script interpreter. I've just had to fix spvw.d to make it
work (that is, to regard -q set twice despite of scripts etc., and to
not print newline on stdout on exit if argv_verbose is 0).

David
From: Sam Steingold
Subject: Re: binary file upload with CLISP
Date: 
Message-ID: <uacm3b8hv.fsf@gnu.org>
> * ············@gmail.com <············@tznvy.pbz> [2005-06-01 04:31:05 -0700]:
>
> That's not what I wanted to ask, really, sorry. How can I do it with
> CLISP; CLISP converts return+linefeed to newline with character
> streams, and does not allow to switch to binary mode on standard-input.

<http://www.podval.org/~sds/clisp/impnotes/stream-dict.html#bin-stdio>

-- 
Sam Steingold (http://www.podval.org/~sds) running w2k
<http://www.dhimmi.com/> <http://www.jihadwatch.org/>
<http://www.iris.org.il> <http://ffii.org/> <http://www.mideasttruth.com/>
char*a="char*a=%c%s%c;main(){printf(a,34,a,34);}";main(){printf(a,34,a,34);}
From: ············@gmail.com
Subject: Re: binary file upload with CLISP
Date: 
Message-ID: <1118088213.569209.18050@g44g2000cwa.googlegroups.com>
Sam Steingold wrote:
> > * ············@gmail.com <············@tznvy.pbz> [2005-06-01 04:31:05 -0700]:
> >
> > That's not what I wanted to ask, really, sorry. How can I do it with
> > CLISP; CLISP converts return+linefeed to newline with character
> > streams, and does not allow to switch to binary mode on standard-input.
>
> <http://www.podval.org/~sds/clisp/impnotes/stream-dict.html#bin-stdio>


Hi Sam,

thank you for the link; the documentation suggests that CLISP
determines that stdin is not a terminal-io stream and allows to setf
stream-element-type. The problem is that it does not. This, of course,
just means that I need to handle all of the input as binary by dup'ing
the handle through ext:make-stream in CGI scripts (instead of switching
between modes),

or just recreate behavior of all other implementations through Gray
streams; which I will do, just to bring it back to portability among
implementations. It is sad though that I cannot just use CLISP to read
CGI input as raw 8-bit encoding and have to resort to recreation of the
behavior through complex hooks impeding performance.

David Tolpin