From: Nathan Baum
Subject: CLISP + xinetd = Instant failure.
Date: 
Message-ID: <1167707961.666083.250300@h40g2000cwb.googlegroups.com>
I wanted to make a simple HTTP server in CLISP using xinetd as the
front end.

  (format t "HTTP/1.1 200 OK~%~%")

  (loop :for line = (remove #\Return (read-line) :from-end t :count 1)
        :until (zerop (length line)))

  (format t "Hello, world~%")

This example works well enough in SBCL via Firefox.

However, CLISP doesn't seem fond of finishing what it started, so when
the script ends, it just throws away whatever output it has pending, by
which Firefox is not amused.

No problem, I think, (finish-output) will take care of that.
Unfortunately, somebody had gotten there first and decided to liven
finish-output up a little. The result is

  *** - UNIX error 14 (EFAULT): Bad address

and, even if the error is ignored, no finished output.

I next try

  (setf *standard-output* (make-string :output :buffered nil))

reasoning that if I just stop CLISP buffering the output, it won't have
any output pending when it finishes. This, however, has the same
result, except it doesn't wait until the script has finished to signal
the error.

I wanted to use CLISP because it appears to have much better support
for internationalisation (for example, stripping the #\Return isn't
necessary in CLISP because it does it automatically), and because (in
my experience) it tends to have a shorter startup time, which is
important for a xinetd service.

The point: Is there a way to get CLISP to finish its output in this
scenario?

From: Kaz Kylheku
Subject: Re: CLISP + xinetd = Instant failure.
Date: 
Message-ID: <1167711022.576271.230900@i12g2000cwa.googlegroups.com>
On Jan 1, 7:19 pm, "Nathan Baum" <···········@btinternet.com> wrote:
> I wanted to make a simple HTTP server in CLISP using xinetd as the
> front end.
>
>   (format t "HTTP/1.1 200 OK~%~%")
>
>   (loop :for line = (remove #\Return (read-line) :from-end t :count 1)
>         :until (zerop (length line)))
>
>   (format t "Hello, world~%")
>
> This example works well enough in SBCL via Firefox.
>
> However, CLISP doesn't seem fond of finishing what it started, so when
> the script ends, it just throws away whatever output it has pending, by
> which Firefox is not amused.

You forgot:
- what version of CLISP, etc
- what is the exact thing you are launching via xinetd.

>   *** - UNIX error 14 (EFAULT): Bad address
>
> and, even if the error is ignored, no finished output.

That means CLISP passed a bad pointer to a C library function.

> I next try
>
>   (setf *standard-output* (make-string :output :buffered nil))

Surely, you mean make-string-stream?

Does make-string-stream have a :buffered keyword? I mean, a string is a
buffer, so things don't get much more buffered than that. :)

> reasoning that if I just stop CLISP buffering the output, it won't have
> any output pending when it finishes. This, however, has the same
> result, except it doesn't wait until the script has finished to signal
> the error.
>
> I wanted to use CLISP because it appears to have much better support
> for internationalisation (for example, stripping the #\Return isn't
> necessary in CLISP because it does it automatically),

Stripping the carriage return is /not/ internationalization! It's an
annoying behavior in CLISP. It recognizes DOS line endings, even when
it's running on Unix.

> The point: Is there a way to get CLISP to finish its output in this
> scenario?

Probably not if it's passing invalid addresses into the C library.
From: Nathan Baum
Subject: Re: CLISP + xinetd = Instant failure.
Date: 
Message-ID: <pan.2007.01.02.04.52.04.442774@btinternet.com>
On Mon, 01 Jan 2007 20:10:22 -0800, Kaz Kylheku wrote:

> On Jan 1, 7:19 pm, "Nathan Baum" <···········@btinternet.com> wrote:
>> I wanted to make a simple HTTP server in CLISP using xinetd as the
>> front end.
>>
>>   (format t "HTTP/1.1 200 OK~%~%")
>>
>>   (loop :for line = (remove #\Return (read-line) :from-end t :count 1)
>>         :until (zerop (length line)))
>>
>>   (format t "Hello, world~%")
>>
>> This example works well enough in SBCL via Firefox.
>>
>> However, CLISP doesn't seem fond of finishing what it started, so when
>> the script ends, it just throws away whatever output it has pending, by
>> which Firefox is not amused.
> 
> You forgot:
> - what version of CLISP, etc

GNU CLISP 2.38 (2006-01-24) (built 3376067987) (memory 3376068459)
Software: GNU C 4.1.1 (Gentoo 4.1.1-r1) gcc -W -Wswitch -Wcomment
-Wpointer-arith -Wimplicit -Wreturn-type -Wmissing-declarations
-Wno-sign-compare -O2 -fexpensive-optimizations -DUNICODE -DDYNAMIC_FFI
-I. -x none libcharset.a libavcall.a libcallback.a /usr/lib/libreadline.so
-lncurses -ldl   -L/usr/lib -lsigsegv -L/usr/lib -lc -lX11 SAFETY=0
HEAPCODES LINUX_NOEXEC_HEAPCODES GENERATIONAL_GC SPVW_BLOCKS SPVW_MIXED
TRIVIALMAP_MEMORY libreadline 5.1
Features:
(ASDF CLC-OS-DEBIAN COMMON-LISP-CONTROLLER ZLIB PCRE FASTCGI
CLX-ANSI-COMMON-LISP CLX RAWSOCK WILDCARD READLINE REGEXP SYSCALLS I18N
LOOP COMPILER CLOS MOP CLISP ANSI-CL
 COMMON-LISP LISP=CL INTERPRETER SOCKETS GENERIC-STREAMS LOGICAL-PATHNAMES
 SCREEN FFI GETTEXT UNICODE BASE-CHAR=CHARACTER PC386 UNIX)
C Modules: (clisp i18n syscalls regexp readline wildcard rawsock linux clx
fastcgi pcre zlib) Installation directory: /usr/lib/clisp/ User language:
ENGLISH
Machine: I686 (I686) localhost [127.0.0.1]

xinetd Version 2.3.14 libwrap loadavg

> - what is the exact thing you are launching via xinetd.

That with #!/usr/bin/clisp -q on the top.


>>   *** - UNIX error 14 (EFAULT): Bad address
>>
>> and, even if the error is ignored, no finished output.
> 
> That means CLISP passed a bad pointer to a C library function.
> 
>> I next try
>>
>>   (setf *standard-output* (make-string :output :buffered nil))
> 
> Surely, you mean make-string-stream?

Nearly. I actually meant make-stream. (A CLISP extension.)

> Does make-string-stream have a :buffered keyword? I mean, a string is a
> buffer, so things don't get much more buffered than that. :)
>
>> reasoning that if I just stop CLISP buffering the output, it won't have
>> any output pending when it finishes. This, however, has the same
>> result, except it doesn't wait until the script has finished to signal
>> the error.
>>
>> I wanted to use CLISP because it appears to have much better support
>> for internationalisation (for example, stripping the #\Return isn't
>> necessary in CLISP because it does it automatically),
> 
> Stripping the carriage return is /not/ internationalization!

Okay. It has much better support for portable non-ASCII character
encodings.

> It's an annoying behavior in CLISP.

Doing the Right Thing is annoying? Since when? ;-)

> It recognizes DOS line endings, even when it's running on Unix.

Rightly so. If your program needs to be able to distinguish between CR and
LF, then it should be using binary input. Otherwise it's not portable: if
you at some time decide to run it on Windows, suddenly (read-line) stops
working correctly on the same data.

>> The point: Is there a way to get CLISP to finish its output in this
>> scenario?
> 
> Probably not if it's passing invalid addresses into the C library.
From: Pascal Bourguignon
Subject: Re: CLISP + xinetd = Instant failure.
Date: 
Message-ID: <87ac12jd4x.fsf@thalassa.informatimago.com>
"Nathan Baum" <···········@btinternet.com> writes:

> I wanted to make a simple HTTP server in CLISP using xinetd as the
> front end.
>
>   (format t "HTTP/1.1 200 OK~%~%")
>
>   (loop :for line = (remove #\Return (read-line) :from-end t :count 1)
>         :until (zerop (length line)))
>
>   (format t "Hello, world~%")
>
> This example works well enough in SBCL via Firefox.
>
> However, CLISP doesn't seem fond of finishing what it started, so when
> the script ends, it just throws away whatever output it has pending, by
> which Firefox is not amused.

What protocol exactly are you implementing?
If you want to access to this server using firefox, you'd better
implement HTTP!

----(/home/pjb/src/lisp/encours/clisp-xinet-http)-----------------------
#!/usr/local/bin/clisp -q -ansi -Kfull -E utf-8

(defparameter *request*
  (loop :for line = (read-line t nil)
        :while (and line (< 0 (length line)))
        :collect line))

(format t "HTTP/1.1 200 OK~%")
(format t "Server: clisp ad-hoc server~%")
(format t "Content-Type: text/plain~%")
(format t "~%")

(format t "Hello world!~%")
(format t "Your request was: ~%~{   ~A~%~}~%" *request*)
(ext:quit 0)

----(/etc/xinet.d/clisp-http)-------------------------------------------
service clisp-http
{
    socket_type = stream
    server = /home/pjb/src/lisp/encours/clisp-xinet-http
    server_args =
    protocol = tcp
    user = nobody
    wait = no
    disable = no
}
----(Added to /etc/services)--------------------------------------------
clisp-http       20003/tcp # pjb
----(firefox output)----------------------------------------------------
Hello world!
Your request was: 
   GET / HTTP/1.1
   Host: thalassa:20003
   User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.0.5) Gecko/20060719 Firefox/1.5.0.5
   Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
   Accept-Language: en-us,en;q=0.5
   Accept-Encoding: gzip,deflate
   Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
   Keep-Alive: 300
   Connection: keep-alive
------------------------------------------------------------------------





-- 
__Pascal Bourguignon__                     http://www.informatimago.com/

WARNING: This product warps space and time in its vicinity.
From: Nathan Baum
Subject: Re: CLISP + xinetd = Instant failure.
Date: 
Message-ID: <1167711363.652282.121560@42g2000cwt.googlegroups.com>
On Jan 2, 3:55 am, Pascal Bourguignon <····@informatimago.com> wrote:
> "Nathan Baum" <···········@btinternet.com> writes:
> > I wanted to make a simple HTTP server in CLISP using xinetd as the
> > front end.
>
> >   (format t "HTTP/1.1 200 OK~%~%")
>
> >   (loop :for line = (remove #\Return (read-line) :from-end t :count 1)
> >         :until (zerop (length line)))
>
> >   (format t "Hello, world~%")
>
> > This example works well enough in SBCL via Firefox.
>
> > However, CLISP doesn't seem fond of finishing what it started, so when
> > the script ends, it just throws away whatever output it has pending, by
> > which Firefox is not amused.
> What protocol exactly are you implementing?
> If you want to access to this server using firefox, you'd better
> implement HTTP!

I already said it was HTTP. :P

Right now it's not very well-formed HTTP, but the fact that it works
flawlessly in SBCL implies the problem is not with the actual content
of the HTTP response itself.

> ----(/home/pjb/src/lisp/encours/clisp-xinet-http)-----------------------
> #!/usr/local/bin/clisp -q -ansi -Kfull -E utf-8
>
> (defparameter *request*
>   (loop :for line = (read-line t nil)
>         :while (and line (< 0 (length line)))
>         :collect line))
>
> (format t "HTTP/1.1 200 OK~%")
> (format t "Server: clisp ad-hoc server~%")
> (format t "Content-Type: text/plain~%")
> (format t "~%")
>
> (format t "Hello world!~%")
> (format t "Your request was: ~%~{   ~A~%~}~%" *request*)
> (ext:quit 0)
>
> ----(/etc/xinet.d/clisp-http)-------------------------------------------
> service clisp-http
> {
>     socket_type = stream
>     server = /home/pjb/src/lisp/encours/clisp-xinet-http
>     server_args =
>     protocol = tcp
>     user = nobody
>     wait = no
>     disable = no}
>----(Added to /etc/services)--------------------------------------------
> clisp-http       20003/tcp # pjb
> ----(firefox output)----------------------------------------------------
> Hello world!
> Your request was:
>    GET / HTTP/1.1
>    Host: thalassa:20003
>    User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.0.5) Gecko/20060719 Firefox/1.5.0.5
>    Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
>    Accept-Language: en-us,en;q=0.5
>    Accept-Encoding: gzip,deflate
>    Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
>    Keep-Alive: 300
>    Connection: keep-alive
> ------------------------------------------------------------------------

Tried it. No dice; same result.

Sometimes I get everything, sometimes I get just the first few lines of
the response, most times I get a "connection was reset" error.

You should be able to see for yourself at
http://mechanus.dyndns.org:8080/
From: Pascal Bourguignon
Subject: Re: CLISP + xinetd = Instant failure.
Date: 
Message-ID: <8764bqjbrz.fsf@thalassa.informatimago.com>
"Nathan Baum" <···········@btinternet.com> writes:

> On Jan 2, 3:55 am, Pascal Bourguignon <····@informatimago.com> wrote:
>> "Nathan Baum" <···········@btinternet.com> writes:
>> > I wanted to make a simple HTTP server in CLISP using xinetd as the
>> > front end.
>>
>> >   (format t "HTTP/1.1 200 OK~%~%")
>>
>> >   (loop :for line = (remove #\Return (read-line) :from-end t :count 1)
>> >         :until (zerop (length line)))
>>
>> >   (format t "Hello, world~%")
>>
>> > This example works well enough in SBCL via Firefox.
>>
>> > However, CLISP doesn't seem fond of finishing what it started, so when
>> > the script ends, it just throws away whatever output it has pending, by
>> > which Firefox is not amused.
>> What protocol exactly are you implementing?
>> If you want to access to this server using firefox, you'd better
>> implement HTTP!
>
> I already said it was HTTP. :P

But this is not what you implemented!
Time is important with network protocols.
You shouldn't write an answer before reading the request!


> Right now it's not very well-formed HTTP, but the fact that it works
> flawlessly in SBCL implies the problem is not with the actual content
> of the HTTP response itself.
> 
> [...]
> Tried it. No dice; same result.
>
> Sometimes I get everything, sometimes I get just the first few lines of
> the response, most times I get a "connection was reset" error.
>
> You should be able to see for yourself at
> http://mechanus.dyndns.org:8080/


Well, we get some thing:

    [···@thalassa pjb]$ telnet mechanus.dyndns.org 8080
    Trying 86.136.170.170...
    Connected to mechanus.dyndns.org.
    Escape character is '^]'.
    GET / HTTP/1.0


    HTTP/1.1 200 OKConnection closed by foreign host.
    [···@thalassa pjb]$


but where is the CR-LF after "OK"?


The next step would be indeed to check the version of clisp.  
My example ran well with 2.41.


-- 
__Pascal Bourguignon__                     http://www.informatimago.com/

"Specifications are for the weak and timid!"
From: Nathan Baum
Subject: Re: CLISP + xinetd = Instant failure.
Date: 
Message-ID: <pan.2007.01.02.05.10.24.569077@btinternet.com>
On Tue, 02 Jan 2007 05:25:04 +0100, Pascal Bourguignon wrote:

> "Nathan Baum" <···········@btinternet.com> writes:
> 
>> On Jan 2, 3:55 am, Pascal Bourguignon <····@informatimago.com> wrote:
>>> "Nathan Baum" <···········@btinternet.com> writes:
> But this is not what you implemented!
> Time is important with network protocols.
> You shouldn't write an answer before reading the request!

It seemed unlikely that Firefox would care, and I think I was right.

Besides, I think there are larger holes in my "implementation" of HTTP
than the response starting too early. It is clearly not complete. However,
as noted before, it works fine in SBCL. I think the reasonable conclusion
is that the content and/or timing of the message isn't the problem.

> Well, we get some thing:
> 
>     [···@thalassa pjb]$ telnet mechanus.dyndns.org 8080
>     Trying 86.136.170.170...
>     Connected to mechanus.dyndns.org.
>     Escape character is '^]'.
>     GET / HTTP/1.0
> 
> 
>     HTTP/1.1 200 OKConnection closed by foreign host.
>     [···@thalassa pjb]$
> 
> 
> but where is the CR-LF after "OK"?

Well, it seems to be AWOL. :)

> The next step would be indeed to check the version of clisp.  
> My example ran well with 2.41.

I'll upgrade and see what happens.
From: Nathan Baum
Subject: Re: CLISP + xinetd = Instant failure.
Date: 
Message-ID: <pan.2007.01.02.05.43.25.124410@btinternet.com>
On Tue, 02 Jan 2007 05:10:25 +0000, Nathan Baum wrote:
>> The next step would be indeed to check the version of clisp.  
>> My example ran well with 2.41.
> I'll upgrade and see what happens.

Updated to 2.41. No apparent change in behaviour.
From: Pascal Bourguignon
Subject: Re: CLISP + xinetd = Instant failure.
Date: 
Message-ID: <87r6uehsc5.fsf@thalassa.informatimago.com>
Nathan Baum <···········@btinternet.com> writes:

> On Tue, 02 Jan 2007 05:10:25 +0000, Nathan Baum wrote:
>>> The next step would be indeed to check the version of clisp.  
>>> My example ran well with 2.41.
>> I'll upgrade and see what happens.
>
> Updated to 2.41. No apparent change in behaviour.

Perhaps another difference is that I always use -ansi 
But I'd be surprised if it changed the buffering behavior...


-- 
__Pascal Bourguignon__                     http://www.informatimago.com/

"What is this talk of "release"?  Klingons do not make software
"releases".  Our software "escapes" leaving a bloody trail of
designers and quality assurance people in its wake."
From: Nathan Baum
Subject: Re: CLISP + xinetd = Instant failure.
Date: 
Message-ID: <pan.2007.01.02.06.28.24.64209@btinternet.com>
On Tue, 02 Jan 2007 07:10:18 +0100, Pascal Bourguignon wrote:

> Nathan Baum <···········@btinternet.com> writes:
> 
>> On Tue, 02 Jan 2007 05:10:25 +0000, Nathan Baum wrote:
>>>> The next step would be indeed to check the version of clisp. My
>>>> example ran well with 2.41.
>>> I'll upgrade and see what happens.
>>
>> Updated to 2.41. No apparent change in behaviour.
> 
> Perhaps another difference is that I always use -ansi But I'd be
> surprised if it changed the buffering behavior...

I'm still using your version of the code, so mine's using -ansi right now
anyway.

Google is vaguely mumbling at me about CLISP not quite working as expected
if *terminal-io* isn't a PTY. Possibly your version of xinetd gives the
server process a PTY. That could explain why it'd work. Or it may be a
difference in our kernels, or glibc, given that the message suggests the
fault isn't in CLISP per se.

I recall having similar trouble with socket streams created within CLISP.
Then, finish-output didn't signal an error, it was silently ignored.
From: Pascal Bourguignon
Subject: Re: CLISP + xinetd = Instant failure.
Date: 
Message-ID: <87mz51j2h5.fsf@thalassa.informatimago.com>
Nathan Baum <···········@btinternet.com> writes:

> On Tue, 02 Jan 2007 07:10:18 +0100, Pascal Bourguignon wrote:
>
>> Nathan Baum <···········@btinternet.com> writes:
>> 
>>> On Tue, 02 Jan 2007 05:10:25 +0000, Nathan Baum wrote:
>>>>> The next step would be indeed to check the version of clisp. My
>>>>> example ran well with 2.41.
>>>> I'll upgrade and see what happens.
>>>
>>> Updated to 2.41. No apparent change in behaviour.
>> 
>> Perhaps another difference is that I always use -ansi But I'd be
>> surprised if it changed the buffering behavior...
>
> I'm still using your version of the code, so mine's using -ansi right now
> anyway.

Next step would be tcpdump
and if it doesn't reveal anything significant, then strace of clisp...

> Google is vaguely mumbling at me about CLISP not quite working as expected
> if *terminal-io* isn't a PTY. Possibly your version of xinetd gives the
> server process a PTY. That could explain why it'd work. Or it may be a
> difference in our kernels, or glibc, given that the message suggests the
> fault isn't in CLISP per se.
>
> I recall having similar trouble with socket streams created within CLISP.
> Then, finish-output didn't signal an error, it was silently ignored.

I'm running a kernel Linux 2.6.15-c3 and an old xinetd Version 20030122.

-- 
__Pascal Bourguignon__                     http://www.informatimago.com/

ATTENTION: Despite any other listing of product contents found
herein, the consumer is advised that, in actuality, this product
consists of 99.9999999999% empty space.
From: Joerg Hoehle
Subject: Re: CLISP + xinetd = Instant failure.
Date: 
Message-ID: <usle33unv.fsf@users.sourceforge.net>
"Nathan Baum" <···········@btinternet.com> writes:

> On Jan 2, 3:55 am, Pascal Bourguignon <····@informatimago.com> wrote:
> > What protocol exactly are you implementing?
> > If you want to access to this server using firefox, you'd better
> > implement HTTP!
> I already said it was HTTP. :P

> Right now it's not very well-formed HTTP, but the fact that it works
> flawlessly in SBCL implies the problem is not with the actual content
> of the HTTP response itself.

No, I'm sorry. Non-observance of bugs using one possible configuration
is no proof of any kind of bugfreeness in a particular program.

Pascal is right about time issues. The HTTP protocol ought to be
respected.  Another difference may be the size of buffers supplied to
read() and write().

So far, all programs in this thread emit bogus HTTP.  MIME headers
require CRLF line endings, and none of the programs makes provision
for supplying a CRLF line-ending in the HTTP response.

It maybe that your favourite browser doesn't care about CRLF.  That
doesn't make your programs correct.

It maybe that a CGI-bin from Apache need not provide CRLF, because
Apache will fix the stream provided by tha careless programmer.  I
don't know, but this is quite possible.

I don't believe xinetd would perform such transformation. It does not
know about HTTP requirements.
 
Regards,
	Jorg Hohle
Telekom/T-Systems Technology Center
From: Kaz Kylheku
Subject: Re: CLISP + xinetd = Instant failure.
Date: 
Message-ID: <1167712262.797215.147770@h40g2000cwb.googlegroups.com>
On Jan 1, 7:55 pm, Pascal Bourguignon <····@informatimago.com> wrote:
> ----(/home/pjb/src/lisp/encours/clisp-xinet-http)-----------------------
> #!/usr/local/bin/clisp -q -ansi -Kfull -E utf-8

You want -E iso-8859-1 here. You don't want to be munging the HTTP
protocol through UTF-8.

Imagine what that does, say, to the body of a POST which is a binary
file. Can you be sure that the character string which comes out of that
will encode back, through utf-8, into a byte string which is identical
to the original bytes? Even if that's the case, the intermediate
character representation will likely have the wrong length, making it
troublesome to read the correct content-length amount.

iso-8859-1 is the right hack to getting all the octets rendered into
characters, in a 1:1 way. From a string of the resulting characters,
you can recover the binary data easily. Each character's char-code
gives you one byte.
From: Nathan Baum
Subject: Re: CLISP + xinetd = Instant failure.
Date: 
Message-ID: <pan.2007.01.02.04.59.32.946995@btinternet.com>
On Mon, 01 Jan 2007 20:31:02 -0800, Kaz Kylheku wrote:

> On Jan 1, 7:55 pm, Pascal Bourguignon <····@informatimago.com> wrote:
>> ----(/home/pjb/src/lisp/encours/clisp-xinet-http)-----------------------
>> #!/usr/local/bin/clisp -q -ansi -Kfull -E utf-8
> 
> You want -E iso-8859-1 here. You don't want to be munging the HTTP
> protocol through UTF-8.

A logical impossibility. The headers specify the encoding for the body, if
there is one, and the headers must only contain ASCII characters, which
have the same binary representation in UTF-8 as they do in ISO-8859-1.

> Imagine what that does, say, to the body of a POST which is a binary
> file. Can you be sure that the character string which comes out of that
> will encode back, through utf-8, into a byte string which is identical
> to the original bytes? Even if that's the case, the intermediate
> character representation will likely have the wrong length, making it
> troublesome to read the correct content-length amount.
> 
> iso-8859-1 is the right hack to getting all the octets rendered into
> characters, in a 1:1 way. From a string of the resulting characters,
> you can recover the binary data easily. Each character's char-code
> gives you one byte.

The right solution is to use bivalent streams. CLISP won't let you output
certain character sequences to an iso-8859-1, so you don't want to be
trying to send binary data to it.
From: Pascal Bourguignon
Subject: Re: CLISP + xinetd = Instant failure.
Date: 
Message-ID: <87y7omhv67.fsf@thalassa.informatimago.com>
Nathan Baum <···········@btinternet.com> writes:

> On Mon, 01 Jan 2007 20:31:02 -0800, Kaz Kylheku wrote:
>
>> On Jan 1, 7:55 pm, Pascal Bourguignon <····@informatimago.com> wrote:
>>> ----(/home/pjb/src/lisp/encours/clisp-xinet-http)-----------------------
>>> #!/usr/local/bin/clisp -q -ansi -Kfull -E utf-8
>> 
>> You want -E iso-8859-1 here. You don't want to be munging the HTTP
>> protocol through UTF-8.
>
> A logical impossibility. The headers specify the encoding for the body, if
> there is one, and the headers must only contain ASCII characters, which
> have the same binary representation in UTF-8 as they do in ISO-8859-1.

Yes, but Kaz is right, it's safer to use a 1-1 encoding, if we don't
use a binary stream for HTTP: an evil client could send a sequence of
bytes between 128 and 255 that'd make an invalid UTF-8 sequence, and
we'd get an ugly condition reading the input stream.


Perhaps clisp should allow changing the element-type of the stdin stream:


   C/USER[8]> (setf (stream-element-type *terminal-io*) '(unsigned-byte 8))

   *** - (SETF STREAM-ELEMENT-TYPE) on #<IO TERMINAL-STREAM> is illegal


and/or add an option to set stdin, stdout and/or stderr as binary streams...



-- 
__Pascal Bourguignon__                     http://www.informatimago.com/

HANDLE WITH EXTREME CARE: This product contains minute electrically
charged particles moving at velocities in excess of five hundred
million miles per hour.
From: Nathan Baum
Subject: Re: CLISP + xinetd = Instant failure.
Date: 
Message-ID: <pan.2007.01.02.05.41.29.749056@btinternet.com>
On Tue, 02 Jan 2007 06:09:04 +0100, Pascal Bourguignon wrote:

> Nathan Baum <···········@btinternet.com> writes:
> 
>> On Mon, 01 Jan 2007 20:31:02 -0800, Kaz Kylheku wrote:
>>
>>> On Jan 1, 7:55 pm, Pascal Bourguignon <····@informatimago.com> wrote:
>>>> ----(/home/pjb/src/lisp/encours/clisp-xinet-http)-----------------------
>>>> #!/usr/local/bin/clisp -q -ansi -Kfull -E utf-8
>>> 
>>> You want -E iso-8859-1 here. You don't want to be munging the HTTP
>>> protocol through UTF-8.
>>
>> A logical impossibility. The headers specify the encoding for the body, if
>> there is one, and the headers must only contain ASCII characters, which
>> have the same binary representation in UTF-8 as they do in ISO-8859-1.
> 
> Yes, but Kaz is right, it's safer to use a 1-1 encoding, if we don't
> use a binary stream for HTTP: an evil client could send a sequence of
> bytes between 128 and 255 that'd make an invalid UTF-8 sequence, and
> we'd get an ugly condition reading the input stream.

A good point.

> 
> Perhaps clisp should allow changing the element-type of the stdin stream:
> 
> 
>    C/USER[8]> (setf (stream-element-type *terminal-io*) '(unsigned-byte 8))
> 
>    *** - (SETF STREAM-ELEMENT-TYPE) on #<IO TERMINAL-STREAM> is illegal
> 
> 
> and/or add an option to set stdin, stdout and/or stderr as binary streams...

The CLISP documentation contains

  (EXT:MAKE-STREAM :INPUT :ELEMENT-TYPE '(UNSIGNED-BYTE 8))

which works for me, although it can't be assigned directly to
*standard-input* since CLISP (perhaps not unreasonably) demands that be a
character stream.

With a little bit of work, it's probably possible to wrap that in a Gray
stream which uses CLISP's trancoding features to switch encodings on the
fly.
From: sds
Subject: Re: CLISP + xinetd = Instant failure.
Date: 
Message-ID: <1167755472.205324.246420@v33g2000cwv.googlegroups.com>
Pascal Bourguignon wrote:
> Perhaps clisp should allow changing the element-type of the stdin stream:
>
>    C/USER[8]> (setf (stream-element-type *terminal-io*) '(unsigned-byte 8))
>
>    *** - (SETF STREAM-ELEMENT-TYPE) on #<IO TERMINAL-STREAM> is illegal
>
> and/or add an option to set stdin, stdout and/or stderr as binary streams...

http://clisp.cons.org/impnotes/stream-dict.html#stream-eltype
http://clisp.cons.org/impnotes/stream-dict.html#bin-stdio

Sam.
From: Joerg Hoehle
Subject: Re: CLISP + xinetd = Instant failure.
Date: 
Message-ID: <uwt3f3v3w.fsf@users.sourceforge.net>
Nathan Baum <···········@btinternet.com> writes:
> On Mon, 01 Jan 2007 20:31:02 -0800, Kaz Kylheku wrote:
> > You want -E iso-8859-1 here. You don't want to be munging the HTTP
> > protocol through UTF-8.

> > iso-8859-1 is the right hack to getting all the octets rendered into
> > characters, in a 1:1 way.

So you should recommend "-Efoo 1:1" instead (even though that's identical
to ISO-8859-1 in practice).  1:1 is the better name.

> The right solution is to use bivalent streams. CLISP won't let you output
> certain character sequences to an iso-8859-1, so you don't want to be
> trying to send binary data to it.
Huh? Output is not the problem. See Edi Weitz' cookbook on binary I/O.
You can faithfully output
(loop for i below 256 do (princ (code-char i) mysocket))
For every character in ISO-8859-1, its code is below 256.

Of course, you can't output #\EURO_SIGN or Kanji to ISO-8859-1.


Sam Steingold wrote:
>>   *** - UNIX error 14 (EFAULT): Bad address
>this might be related to
>https://sourceforge.net/tracker/index.php?func=detail&aid=1592343&group_id=1355&atid=101355
>which is actually a linux kernel bug
>http://bugzilla.kernel.org/show_bug.cgi?id=7635
Maybe it's related, maybe not.

Nathan, it would be interesting to the developers if you could confirm
or deny this assumption with your machine.

Regards,
	Jorg Hohle
Telekom/T-Systems Technology Center
From: sds
Subject: Re: CLISP + xinetd = Instant failure.
Date: 
Message-ID: <1167764031.762037.309250@a3g2000cwd.googlegroups.com>
Nathan Baum wrote:
> No problem, I think, (finish-output) will take care of that.
> Unfortunately, somebody had gotten there first and decided to liven
> finish-output up a little. The result is
>
>   *** - UNIX error 14 (EFAULT): Bad address
>
> and, even if the error is ignored, no finished output.

this might be related to
https://sourceforge.net/tracker/index.php?func=detail&aid=1592343&group_id=1355&atid=101355
which is actually a linux kernel bug
http://bugzilla.kernel.org/show_bug.cgi?id=7635

Sam.