Hi,
after some break I am playing again, this time with SBCL, and I have
a problem (as usual). I want to write server part of networking
application in SBCL (because it has native threads unlike CMUCL (my
hope is, that SBCL will support multiple processors (or cores) in close
future).
So far, I was using sb-bsd-sockets, but there is a problem :
1.) where is socket-shutdown? As I have checked the source of
socket-close, it looks like the sb-bsd-sockets API is working with
sockets as with file descriptors. The socket-close closes the
descriptor assigned with socket, but the socket itself should be
SHUTDOWNed with shutdown. This means, that when I close server listener
socket, blocked in socket-accept call, the listener thread is not
unblocked, but waits until another client tries to connect to him, and
then the sb-bsd-sockets API finds out that socket is closed and signals
that error with socket-condition. But this is far from elegant. For
example, I think, that in this code :
(defun stop-server ()
(when (and *listener-socket* *listener-thread*)
(terminate-thread *listener-thread*)
(socket-close *listener-socket*)
(setf *listener-socket* nil
*listener-thread* nil)
t))
there may be some resources leak, because shutdown is not called.
Anybody else tried to solve something similar?
2.) why sb-bsd-sockets works with any tcp socket using descriptor?
Could that cause some send/receive slowdown using this approach?
3.) I was checking some replacement for sb-bsd-sockets, but the 2 I
have checked :
- acl-compat :
acl-compat.socket:ACCEPT-CONNECTION
acl-compat.socket:DOTTED-TO-IPADDR
acl-compat.socket:IPADDR-TO-DOTTED
acl-compat.socket:IPADDR-TO-HOSTNAME
acl-compat.socket:LOCAL-HOST acl-compat.socket:LOCAL-PORT
acl-compat.socket:LOOKUP-HOSTNAME acl-compat.socket:MAKE-SOCKET
acl-compat.socket:REMOTE-HOST acl-compat.socket:REMOTE-PORT
acl-compat.socket:SOCKET-CONTROL
looks like does not solve the problem, the other one - trivial-sockets
- is according to the author's own description suitable for
"undemanding Internet applications".
Am I missing something?
Karol
"Karol Skocik" <············@gmail.com> writes:
> Am I missing something?
Free software is often created to satisfy the author's immediate
needs. In this specific case, the original author probably did not
need and thus did not make available the features you are asking
about.
The solution is to extend the software to suit your needs, or find
someone who can extend it for you.
Zach
Hi,
I am not complaining. You are right about F/OSS. My point here is :
is it all-right just to close socket? Imagine, you have a server
software and this *may* leak some resources. After serving for example
100000 clients in 1 week, the machine is going down because of this. I
have looked in the sb-bsd-sockets:socket-close source, and it is not
done there. I would like to know if it's OK, and if yes, maybe it would
be nice to write it in docs, somewhere near socket-close documentation.
Karol
Karol Skocik wrote:
> Hi,
> I am not complaining. You are right about F/OSS. My point here is :
> is it all-right just to close socket? Imagine, you have a server
> software and this *may* leak some resources. After serving for example
> 100000 clients in 1 week, the machine is going down because of this. I
> have looked in the sb-bsd-sockets:socket-close source, and it is not
> done there. I would like to know if it's OK, and if yes, maybe it would
> be nice to write it in docs, somewhere near socket-close documentation.
The only thing that can leak is the file descriptor. Memory is garbage
collected in Lisp (duh).
I don't know about the SBCL source. Are you saying it doesn't close the
file descriptor? In that case it should be enough (and correct and
unproblematic) to make that system call there.
--
The problems of the real world are primarily those you are left with
when you refuse to apply their effective solutions.
Edsger W. Dijkstra
It closes the descriptor, so it's not a problem. I have reread what
exactly shutdown does - it shutdowns in, out or both directions which
the data moves, so it is generally usefull when the communication is
going to end. So, things can be done without it of course, it is just
not that straightforward. Seems like no resouces leak to me...
Karol Skocik <············@gmail.com> wrote:
+---------------
| It closes the descriptor, so it's not a problem. I have reread what
| exactly shutdown does - it shutdowns in, out or both directions which
| the data moves, so it is generally usefull when the communication is
| going to end. So, things can be done without it of course, it is just
| not that straightforward. Seems like no resouces leak to me...
+---------------
NOTA BENE: Using a uni-directional "shutdown" instead of [or rather,
at some point *before*] a normal CLOSE is sometimes critical to
certain client/server applications, since the server may try to read
the socket until EOF before sending any response. I had this problem
once when writing a CGI script that needed to connect to another
server to forward certain POST transactions. I found that I had to
do a "shutdown(sock, SHUT_WR)" [in C] on the client side before the
server would reliably see an input EOF. One might *expect* that
the "Content-Length" in the request header would have been adequate,
but it wasn't. (*sigh*)
You have to use a "shutdown" rather than a "close" in this case,
since a "close" automatically shuts down *both* directions of the
TCP socket, but if you did that you wouldn't be able to read the
server's response. Hence the need for a uni-directional "shutdown"
in some cases.
AFAICT, CMUCL [and possibly SBCL, but I don't have one to check
that with] does not provide a built-in uni-directional "shutdown"
option on its networking streams [which are LISP::FD-STREAMs].
ISTR having to cons one up manually once using the UNIX::INT-SYSCALL
macro...
-Rob
-----
Rob Warnock <····@rpw3.org>
627 26th Avenue <URL:http://rpw3.org/>
San Mateo, CA 94403 (650)572-2607
Thanks for clarification. But does it matter when server is in Lisp and
client is Java applet? Since the applet can not store it's state other
than on server, I think, in this particular case it's OK. But shutdown
would be very nice in sb-bsd-sockets, as socket-send would be. I wish I
had enough skills to implement that myself, but 6 months Lisp
experience seems not enough to mess up with such important part of Lisp
community as SBCL is.
Karol
Karol Skocik <············@gmail.com> wrote:
+---------------
| Thanks for clarification. But does it matter when server is
| in Lisp and client is Java applet?
+---------------
No, the programming language really isn't an issue at all
[unless, of course, the language makes it hard to do a
"shutdown()" or equivalent]. The situation can arise with
*any* TCP-based protocol in which one end wants to "read
until EOF" before writing the last data it's going to send
before closing the socket. [Normally it's the client that
needs to send a uni-directional "shutdown()", but I've seen
occasional oddball situations where the reverse was true.]
My first experience with the issue was with both the client
and server being C programs, but I ran into it again later
with a C client and a Lisp server, and then yet again with
a Lisp client and a C server (Apache).
-Rob
-----
Rob Warnock <····@rpw3.org>
627 26th Avenue <URL:http://rpw3.org/>
San Mateo, CA 94403 (650)572-2607
····@rpw3.org (Rob Warnock) writes:
> Karol Skocik <············@gmail.com> wrote:
> +---------------
> | Thanks for clarification. But does it matter when server is
> | in Lisp and client is Java applet?
> +---------------
>
> No, the programming language really isn't an issue at all
> [unless, of course, the language makes it hard to do a
> "shutdown()" or equivalent]. The situation can arise with
> *any* TCP-based protocol in which one end wants to "read
> until EOF" before writing the last data it's going to send
> before closing the socket. [Normally it's the client that
> needs to send a uni-directional "shutdown()", but I've seen
> occasional oddball situations where the reverse was true.]
djb wrote about this:
http://cr.yp.to/tcpip/twofd.html
From that page:
All of this would have worked perfectly if a TCP connection, just
like a UNIX pipe, had been represented as two UNIX file
descriptors: a reading descriptor and a writing
descriptor. (Technically, a reading ofile and a writing ofile.) The
kernel sends FIN on the TCP connection when the writing descriptor
closes. It closes the reading descriptor when it receives FIN on
the TCP connection.
But the BSD socket designers instead decided to merge the two obvious
fds into one read-write fd. When the generate-data program finishes,
the same fd is still open in the consume-data program, so the kernel
has no idea that it should send a FIN.
The BSD socket designers introduced a shutdown(fd,1) system call to
send FIN through a TCP connection. This is exactly the sort of
device-specific garbage that UNIX fds were designed to avoid. The
generate-data program works with pipes but doesn't work with TCP
connections.
Zach
Zach Beane <····@xach.com> wrote:
+---------------
| >····@rpw3.org (Rob Warnock) writes:
| > No, the programming language really isn't an issue at all
| > [unless, of course, the language makes it hard to do a
| > "shutdown()" or equivalent]. The situation can arise with
| > *any* TCP-based protocol in which one end wants to "read
| > until EOF" before writing the last data it's going to send
| > before closing the socket.
|
| djb wrote about this:
| http://cr.yp.to/tcpip/twofd.html
+---------------
Thanks for the ref! Yes, that's exactly the issue I was trying to explain.
+---------------
| From that page:
| All of this would have worked perfectly if a TCP connection, just
| like a UNIX pipe, had been represented as two UNIX file descriptors...
+---------------
And Common Lisp implementors who added networking hooks to their
platforms *could* have also fixed it by, say, giving the routine
that converts an open Unix fd [called MAKE-FD-STREAM in CMUCL;
most CL other implementations have or do something similar] an
option to return a *pair* of uni-directional streams instead of
a TWO-WAY-STREAM. ...and of course arranging at the same time
that a CLOSE on one of them while the other was still open would
automatically perform only the appropriate-direction "shutdown()"
and not "close()" the underlying fd until *both* uni-directional
streams had been CLOSEd.
But I don't know of one that did. (Anybody...?)
-Rob
-----
Rob Warnock <····@rpw3.org>
627 26th Avenue <URL:http://rpw3.org/>
San Mateo, CA 94403 (650)572-2607
····@rpw3.org (Rob Warnock) writes:
> +---------------
> | From that page:
> | All of this would have worked perfectly if a TCP connection, just
> | like a UNIX pipe, had been represented as two UNIX file descriptors...
> +---------------
>
> And Common Lisp implementors who added networking hooks to their
> platforms *could* have also fixed it by, say, giving the routine
> that converts an open Unix fd [called MAKE-FD-STREAM in CMUCL;
> most CL other implementations have or do something similar] an
> option to return a *pair* of uni-directional streams instead of
> a TWO-WAY-STREAM. ...and of course arranging at the same time
> that a CLOSE on one of them while the other was still open would
> automatically perform only the appropriate-direction "shutdown()"
> and not "close()" the underlying fd until *both* uni-directional
> streams had been CLOSEd.
>
> But I don't know of one that did. (Anybody...?)
CMUCL couldn't do that with MAKE-FD-STREAM, because it does what its
name says it does: it makes a stream from an FD, and for non-socket
FDs that would be more than a little annoying. SBCL has
SOCKET-MAKE-STREAM, which is just a wrapper around MAKE-FD-STREAM, but
could be extended to support something along those lines.
--
/|_ .-----------------------.
,' .\ / | Free Mumia Abu-Jamal! |
,--' _,' | Abolish the racist |
/ / | death penalty! |
( -. | `-----------------------'
| ) |
(`-. '--.)
`. )----'
····@rpw3.org (Rob Warnock) writes:
> option to return a *pair* of uni-directional streams instead of
> a TWO-WAY-STREAM. ...and of course arranging at the same time
> that a CLOSE on one of them while the other was still open would
> automatically perform only the appropriate-direction "shutdown()"
> and not "close()" the underlying fd until *both* uni-directional
> streams had been CLOSEd.
FWIW, CLISP added socket-stream-shutdown after a discussion in
clisp-lisp from March to May 2002 and clisp-devel from May to July
2002. You may wish to consult the mailing list archives for the
discussion back then.
No separate Lisp-level streams, though:
http://clisp.cons.org/impnotes/socket.html#sost-shut
BTW, how would you implement the following behaviour (DWIM
bidirectional streams): automatically force-output prior to reading?
Many people forget FORCE-OUTPUT. Should we help them transparently or
see their software fail when run from interactive streams (pty, not
just socket) instead of pipes?
Regards,
Jorg Hohle
Telekom/T-Systems Technology Center
Joerg Hoehle <······@users.sourceforge.net> wrote:
+---------------
| ····@rpw3.org (Rob Warnock) writes:
| > option to return a *pair* of uni-directional streams instead of
| > a TWO-WAY-STREAM. ...and of course arranging at the same time
| > that a CLOSE on one of them while the other was still open would
| > automatically perform only the appropriate-direction "shutdown()"
| > and not "close()" the underlying fd until *both* uni-directional
| > streams had been CLOSEd.
|
| FWIW, CLISP added socket-stream-shutdown after a discussion in
| clisp-lisp from March to May 2002 and clisp-devel from May to July
| 2002. You may wish to consult the mailing list archives for the
| discussion back then.
+---------------
Thanks for the references! I'll check them out.
+---------------
| No separate Lisp-level streams, though:
| http://clisp.cons.org/impnotes/socket.html#sost-shut
+---------------
Hmmm... That looks like a reasonable API to copy for CMUCL, too,
though I'm not sure the CHANGE-CLASS part would work right on
the structures CMUCL uses for STREAMs. But if the stream doesn't
actually have to change class per se, but only change behavior
w.r.t. OUTPUT-STREAM-P or INPUT-STREAM-P (as appropriate), that
could be make to work quite easily.
+---------------
| BTW, how would you implement the following behaviour (DWIM
| bidirectional streams): automatically force-output prior to reading?
+---------------
I wouldn't. [See below.]
+---------------
| Many people forget FORCE-OUTPUT. Should we help them transparently or
| see their software fail when run from interactive streams (pty, not
| just socket) instead of pipes?
+---------------
Their software might also fail to perform as expected [especially if
one includes performance in the expectation] if FORCE-OUTPUTs were
"helpfully" inserted at random. (Think of pipelined protocols...)
-Rob
-----
Rob Warnock <····@rpw3.org>
627 26th Avenue <URL:http://rpw3.org/>
San Mateo, CA 94403 (650)572-2607
Joerg Hoehle wrote:
> BTW, how would you implement the following behaviour (DWIM
> bidirectional streams): automatically force-output prior to reading?
>
> Many people forget FORCE-OUTPUT. Should we help them transparently or
> see their software fail when run from interactive streams (pty, not
> just socket) instead of pipes?
Perhaps a note in the appropriate spot in the implementation's
documentation? One that describes the behaviour people will see if they
forget -- thus triggering their recall when they do. This isn't a Lisp
problem, and other languages (Perl, C) don't DWIM, either. Like Rob
Warnock says, the grief you cause to ex[perienced socket programmers may
well outweigh that which you save the newbies from.
Karol Skocik schrieb:
> Hi,
> after some break I am playing again, this time with SBCL, and I have
> a problem (as usual). I want to write server part of networking
> application in SBCL (because it has native threads unlike CMUCL (my
> hope is, that SBCL will support multiple processors (or cores) in close future).
What do you mean by "support multiple processors or cores"?
Say you have a AMD Dual-Core system. You start a new thread.
This one will be executed on the core which has less load.
Maybe this is what you mean by "support" - that is already in. You
don't have to think about it, your threads just get executed on the
right core/cpu (plus sbcl compiles to native 64 bit code).
André
--