I've had a ok grap on TCP/IP for a while now, but since I have not done
any real TCP/IP programming I have never before really encountered
sockets.
Having looked at some old Lisp code I realize that streams over networ
connections, raw IP packets and servers over sockets is all one big
happy circus. Sockets doesn't fit in the big fine shiny OSI stack, but
it seems to be the one thing everyone use for doing network
programming.
So how is it in the lisp world? When you are programming a network
server, do you start with some socket library, or do you try to build
TCP/IP packets by hand?
Anyone care to give me a rough overview of the present state of Lisp
network programming?
I once thought that doing a TCP/IP stack in Lisp would be much easier
than in C, but now I'm not so sure. What do you actually build upon? All
the lisp impelmentations I've investigated seems very preoccupied by
this "sockets" thing instead. Is there a TCP/IP implementation to look
at and investigate?
/Andreas
--
A: Because it fouls the order in which people normally read text.
Q: Why is top-posting such a bad thing?
A: Top-posting.
Q: What is the most annoying thing on usenet and in e-mail?
On Dec 10, 9:12 am, Andreas Davour <·······@updateLIKE.uu.HELLse>
wrote:
> I've had a ok grap on TCP/IP for a while now, but since I have not done
> any real TCP/IP programming I have never before really encountered
> sockets.
>
> Having looked at some old Lisp code I realize that streams over networ
> connections, raw IP packets and servers over sockets is all one big
> happy circus. Sockets doesn't fit in the big fine shiny OSI stack, but
> it seems to be the one thing everyone use for doing network
> programming.
>
> So how is it in the lisp world? When you are programming a network
> server, do you start with some socket library, or do you try to build
> TCP/IP packets by hand?
>
> Anyone care to give me a rough overview of the present state of Lisp
> network programming?
>
> I once thought that doing a TCP/IP stack in Lisp would be much easier
> than in C, but now I'm not so sure. What do you actually build upon? All
> the lisp impelmentations I've investigated seems very preoccupied by
> this "sockets" thing instead. Is there a TCP/IP implementation to look
> at and investigate?
>
> /Andreas
>
> --
> A: Because it fouls the order in which people normally read text.
> Q: Why is top-posting such a bad thing?
> A: Top-posting.
> Q: What is the most annoying thing on usenet and in e-mail?
As far as I can figure out the Common Lisp standard doesn't really
cover network programming and hence each CL implementation handles it
in a slightly different way. Therefore if you seriously wanted to
make a network server in Lisp you'd need to check-out the network
programming support of the implementation (CLISP, SBCL, etc) first or
be prepared to use a foreign-function interface to get the
functionality you need.
http://www.hackinghat.com/index.php/lisp/talking-to-donkeys
In general though whatever abstractions you chose to build will
probably need to sit on top of IP sockets somehow. Unless you were
using unusual hardware.
Steve
Steve-o <······@googlemail.com> writes:
> On Dec 10, 9:12 am, Andreas Davour <·······@updateLIKE.uu.HELLse>
> wrote:
>> Anyone care to give me a rough overview of the present state of Lisp
>> network programming?
>
> As far as I can figure out the Common Lisp standard doesn't really
> cover network programming and hence each CL implementation handles it
> in a slightly different way. Therefore if you seriously wanted to
> make a network server in Lisp you'd need to check-out the network
> programming support of the implementation (CLISP, SBCL, etc) first or
> be prepared to use a foreign-function interface to get the
> functionality you need.
>
> http://www.hackinghat.com/index.php/lisp/talking-to-donkeys
>
> In general though whatever abstractions you chose to build will
> probably need to sit on top of IP sockets somehow. Unless you were
> using unusual hardware.
Thanks for the link.
I just reread what I'd posted and I wonder why I have such a big problem
saying what I mean when I first post? Thanks for taking the time to
deceipher me.
/Andreas
--
A: Because it fouls the order in which people normally read text.
Q: Why is top-posting such a bad thing?
A: Top-posting.
Q: What is the most annoying thing on usenet and in e-mail?
P� Mon, 10 Dec 2007 08:12:26 +0100, skrev Andreas Davour
<·······@updateLIKE.uu.HELLse>:
>
> I've had a ok grap on TCP/IP for a while now, but since I have not done
> any real TCP/IP programming I have never before really encountered
> sockets.
>
> Having looked at some old Lisp code I realize that streams over networ
> connections, raw IP packets and servers over sockets is all one big
> happy circus. Sockets doesn't fit in the big fine shiny OSI stack, but
> it seems to be the one thing everyone use for doing network
> programming.
>
> So how is it in the lisp world? When you are programming a network
> server, do you start with some socket library, or do you try to build
> TCP/IP packets by hand?
>
> Anyone care to give me a rough overview of the present state of Lisp
> network programming?
>
> I once thought that doing a TCP/IP stack in Lisp would be much easier
> than in C, but now I'm not so sure. What do you actually build upon? All
> the lisp impelmentations I've investigated seems very preoccupied by
> this "sockets" thing instead. Is there a TCP/IP implementation to look
> at and investigate?
>
> /Andreas
>
This gives a decent intro:
http://cl-cookbook.sourceforge.net/sockets.html
--------------
John Thingstad
"John Thingstad" <·······@online.no> writes:
> P� Mon, 10 Dec 2007 08:12:26 +0100, skrev Andreas Davour
> <·······@updateLIKE.uu.HELLse>:
>
>>
>> I've had a ok grap on TCP/IP for a while now, but since I have not done
>> any real TCP/IP programming I have never before really encountered
>> sockets.
>>
>> Having looked at some old Lisp code I realize that streams over networ
>> connections, raw IP packets and servers over sockets is all one big
>> happy circus. Sockets doesn't fit in the big fine shiny OSI stack, but
>> it seems to be the one thing everyone use for doing network
>> programming.
>>
>> So how is it in the lisp world? When you are programming a network
>> server, do you start with some socket library, or do you try to build
>> TCP/IP packets by hand?
>>
>> Anyone care to give me a rough overview of the present state of Lisp
>> network programming?
>>
>> I once thought that doing a TCP/IP stack in Lisp would be much easier
>> than in C, but now I'm not so sure. What do you actually build upon? All
>> the lisp impelmentations I've investigated seems very preoccupied by
>> this "sockets" thing instead. Is there a TCP/IP implementation to look
>> at and investigate?
>>
>
> This gives a decent intro:
> http://cl-cookbook.sourceforge.net/sockets.html
Interesting. Thanks. Now I also got to know that there is *another*
socket library out there! This time from the CLOCC. Talk about to much
choice...
/Andreas
--
A: Because it fouls the order in which people normally read text.
Q: Why is top-posting such a bad thing?
A: Top-posting.
Q: What is the most annoying thing on usenet and in e-mail?
Den Mon, 10 Dec 2007 08:12:26 +0100 skrev Andreas Davour:
> So how is it in the lisp world? When you are programming a network
> server, do you start with some socket library, or do you try to build
> TCP/IP packets by hand?
Why on Earth would you build packets by hand? Last time I checked, high
level languages were about avoiding low-level pain. Not to mention it'd
be incredibly stupid, see below. Of course you use libraries.
> I once thought that doing a TCP/IP stack in Lisp would be much easier
> than in C, but now I'm not so sure. What do you actually build upon? All
> the lisp impelmentations I've investigated seems very preoccupied by
> this "sockets" thing instead. Is there a TCP/IP implementation to look
> at and investigate?
1) A good TCP/IP stack is amazingly hard to build. Most implementations
get it somewhat right within 5-15 years of usage. Building your own would
be the dumbest way to go about network programming possible.
2) A network stack is an OS thing. Apps have no business in, and no
rights to, building their own. How do you imagine it'd work, with every
app shipping a separate implementation of such a fundamental piece of
code, and requring direct access to the hardware to fiddle with bits?
It'd be like having each app ship with its own FS drivers.
The OS has a defined interface for file systems, and so it does for the
network. And in the Unix (BSD) world, those happen to be sockets, which
were an interface good enough to be ported / cloned by most other OS.
They are also fairly low-level and not portable between implementations
(because there's no standard mapping between sockets and a Lisp API), so
you usually want to use some library to smooth it out for you.
Right now I'm using IOlib, which is fairly usable, if docless (so you
will have to read a sockets manual alongside to figure out how to use the
socket-based streams, and poke around in source/exported symbols to see
what API is there)[1]. There are also various other libraries, like
trivial-sockets (since obsoleted by usockets), and others.
Cheers,
Maciej
[1] A common gotcha: In case you're writing a server (ie. you're gonna
listen for incoming connections), you need to remember about the
separation between the listening "passive" socket, and a separate
"active" socket resulting from accepting each connection. The former
stays open as long as you listen, the latter needs to be closed after
you're done with the connection, failure to do so results in puzzling
bugs. You can use WITH-ACCEPT-CONNECTION, which has been just added[2] to
the API to make it more apparent and as a shorthand for (with-open-stream
(accept-connection passive-socket) ...).
[2] You really should use the HEAD, which can be had from repos listed on
the project's download page.
Maciej Katafiasz <········@gmail.com> writes:
> Den Mon, 10 Dec 2007 08:12:26 +0100 skrev Andreas Davour:
> 1) A good TCP/IP stack is amazingly hard to build. Most implementations
> get it somewhat right within 5-15 years of usage. Building your own would
> be the dumbest way to go about network programming possible.
>
> 2) A network stack is an OS thing. Apps have no business in, and no
> rights to, building their own. How do you imagine it'd work, with every
> app shipping a separate implementation of such a fundamental piece of
> code, and requring direct access to the hardware to fiddle with bits?
> It'd be like having each app ship with its own FS drivers.
>
> The OS has a defined interface for file systems, and so it does for the
> network. And in the Unix (BSD) world, those happen to be sockets, which
> were an interface good enough to be ported / cloned by most other OS.
> They are also fairly low-level and not portable between implementations
> (because there's no standard mapping between sockets and a Lisp API), so
> you usually want to use some library to smooth it out for you.
OK, then I've learnt something new. Sockets is what you interface with,
the stack is the groundwork for the sockets. Got that. Lisp is a jungle
so you use a layer above the socket themselves. Got that.
Implementing bits of a stack would probably be easier i Lisp than in C,
I guessed. And it would teach the implementor quite a bit about
networking, but maybe I'll have to save that for when I want to write
routing software for a router, or something.
> Right now I'm using IOlib, which is fairly usable, if docless (so you
> will have to read a sockets manual alongside to figure out how to use the
> socket-based streams, and poke around in source/exported symbols to see
> what API is there)[1]. There are also various other libraries, like
> trivial-sockets (since obsoleted by usockets), and others.
It sounds like usockets is what I need now.
Thanks.
/Andreas
--
A: Because it fouls the order in which people normally read text.
Q: Why is top-posting such a bad thing?
A: Top-posting.
Q: What is the most annoying thing on usenet and in e-mail?
AD> Anyone care to give me a rough overview of the present state of Lisp
AD> network programming?
present state of Lisp network programming is pretty good, although there is
no standard for it.
there are, however, compatibility libraries like trivial-sockets and
usocket.
you just need to call one function to connect to server, and another one to
use socket like Lisp stream:
(let ((sock (usocket:socket-connect "common-lisp.net" 80)))
(unwind-protect
(progn
(format (usocket:socket-stream sock)
"GET / HTTP/1.0~A~A~A~A"
#\Return #\Newline #\Return #\Newline)
(force-output (usocket:socket-stream sock))
(read-line (usocket:socket-stream sock)))
(usocket:socket-close sock))
AD> I once thought that doing a TCP/IP stack in Lisp would be much easier
AD> than in C, but now I'm not so sure.
TCP/IP stack is already implemented in your OS. how exactly are you going to
implement it once more?
probably you can do it in Linux -- remove existing support, get direct
access to hardware, and try to implement your own.. but why?
TCP/Ip stack in Lisp would be pretty funny -- whenever Lisp goes garbage
collecting, no packets sent.
AD> All the lisp impelmentations I've investigated seems very preoccupied
AD> by this "sockets" thing instead.
why, for hell, you need something other than sockets? TCP/IP is essentially
a stream of octets, hence it's OK to use it as stream. you'll gain nothing
going low level.
you only need to go low level if you do some advanced stuff like ping
utility. in this case i think you'd need to use FFI and call OS functions
(different for each different OS). there's nothing about Lisp in this case.
On 2007-12-10, Alex Mizrahi <········@users.sourceforge.net> wrote:
>
> you only need to go low level if you do some advanced stuff like ping
> utility. in this case i think you'd need to use FFI and call OS functions
> (different for each different OS). there's nothing about Lisp in this case.
>
Not really, google "scotty nmicmpd" it allows you to treat it like
a binary stream. If I ever remember how to program again a wrapper
for CL would make a nice project.
marc
--
······@sdf.lonestar.org
SDF Public Access UNIX System - http://sdf.lonestar.org
"Alex Mizrahi" <········@users.sourceforge.net> writes:
> AD> Anyone care to give me a rough overview of the present state of Lisp
> AD> network programming?
>
> present state of Lisp network programming is pretty good, although
> there is no standard for it. there are, however, compatibility
> libraries like trivial-sockets and usocket.
>
> you just need to call one function to connect to server, and another one to
> use socket like Lisp stream:
[usocket example]
Thanks.
> AD> I once thought that doing a TCP/IP stack in Lisp would be much easier
> AD> than in C, but now I'm not so sure.
>
> TCP/IP stack is already implemented in your OS. how exactly are you
> going to implement it once more? probably you can do it in Linux --
> remove existing support, get direct access to hardware, and try to
> implement your own.. but why?
>
> TCP/Ip stack in Lisp would be pretty funny -- whenever Lisp goes
> garbage collecting, no packets sent.
Well, the reason I mentioned them is that they have been done in LispM
operating systems, and since all documentation I've found talks about
sockets like the TCP/IP stack don't even exist.
I think gc is handled intelligently in lisp when it's implemented. It
would probably be interesting to see how!
> AD> All the lisp impelmentations I've investigated seems very preoccupied
> AD> by this "sockets" thing instead.
>
> why, for hell, you need something other than sockets? TCP/IP is
> essentially a stream of octets, hence it's OK to use it as
> stream. you'll gain nothing going low level.
>
> you only need to go low level if you do some advanced stuff like ping
> utility. in this case i think you'd need to use FFI and call OS
> functions (different for each different OS). there's nothing about
> Lisp in this case.
The thing is (And this is not very clear from my original message, which
is a mess) that I've looked at a few books about networking and they all
talk about TCP and UDP and IP packets. Low level stuff. Text about
network programming always talks about sockets. I didn't understand how
they fit together.
So, in practical programming you use streams. I didn't know that. There
are many kind of streams, though. Gray? Simple? Is this where the
usocket library helps out by hiding the details? What would you do if
you'd like to implement, say, an extended FTP protocol you just
invented?
/Andreas
--
A: Because it fouls the order in which people normally read text.
Q: Why is top-posting such a bad thing?
A: Top-posting.
Q: What is the most annoying thing on usenet and in e-mail?
AD> The thing is (And this is not very clear from my original message,
AD> which is a mess) that I've looked at a few books about networking and
AD> they all talk about TCP and UDP and IP packets. Low level stuff. Text
AD> about network programming always talks about sockets. I didn't
AD> understand how they fit together.
there's certainly reason for this -- low level stuff is implemented once,
and now you have higher-level interface. book about web programming won't
even mention sockets -- they use only HTTP protocol. and many people doing
practical web programming do not even know how HTTP works..
of course lower-level knowdledge is not totally useless -- when you'll have
some problems, or would need to optimize your application it's always good
to know how it works on low level. but when you just write new programs you
need only high level stuff.
AD> So, in practical programming you use streams. I didn't know that.
that's why it's often better to do some experiments yourself rather than
read theory in books :)
i didn't ever read more than few chapters of any "computer" book..
AD> There are many kind of streams, though. Gray? Simple?
good thing about any streams is that standard Common Lisp functions like
format and princ work on them. so it doesn't matter how is it implemented.
what matters is if it is a binary or character stream (and what encoding
character has). i would recommend flexi-streams
http://weitz.de/flexi-streams/ if you need some transcoding.
AD> Is this where the usocket library helps out by hiding the details?
yup, it should
AD> What would you do if you'd like to implement, say, an extended FTP
AD> protocol you just
AD> invented?
i think example above with usockets will do at least text part of FTP. for
binary part you need to change some stuff..
"Alex Mizrahi" <········@users.sourceforge.net> writes:
> AD> There are many kind of streams, though. Gray? Simple?
>
> good thing about any streams is that standard Common Lisp functions like
> format and princ work on them. so it doesn't matter how is it implemented.
>
> what matters is if it is a binary or character stream (and what encoding
> character has). i would recommend flexi-streams
> http://weitz.de/flexi-streams/ if you need some transcoding.
So the main difference between oll the different kind of streams biols
down to if it's a character stream or a binary stream? That makes it
easier I guess. Reading the CMUCL/SBCL manual you can get confused by
the different stream implementations, you know.
/Andreas
--
A: Because it fouls the order in which people normally read text.
Q: Why is top-posting such a bad thing?
A: Top-posting.
Q: What is the most annoying thing on usenet and in e-mail?
Andreas Davour <·······@updateLIKE.uu.HELLse> wrote:
+---------------
| "Alex Mizrahi" <········@users.sourceforge.net> writes:
| > what matters is if it is a binary or character stream (and
| > what encoding character has). i would recommend flexi-streams
| > http://weitz.de/flexi-streams/ if you need some transcoding.
|
| So the main difference between oll the different kind of streams biols
| down to if it's a character stream or a binary stream? That makes it
| easier I guess. Reading the CMUCL/SBCL manual you can get confused by
| the different stream implementations, you know.
+---------------
Yes. If you're doing anything at all fancy with CMUCL streams you
probably want to "look under the covers" at the SYSTEM:FD-STREAM class.
For example, few people know that CMUCL actually does provide a form
of bivalent stream on FD-STREAMs (which includes Unix/Linux file
streams as well as sockets) with the little-known :CLASS option
to OPEN, e.g.:
> (open "foo" :class 'binary-text-stream :element-type :default)
#<Binary-text Stream for file "foo">
>
You can use both READ-CHAR and READ-BYTE on such a stream.
-Rob
-----
Rob Warnock <····@rpw3.org>
627 26th Avenue <URL:http://rpw3.org/>
San Mateo, CA 94403 (650)572-2607
····@rpw3.org (Rob Warnock) writes:
> Andreas Davour <·······@updateLIKE.uu.HELLse> wrote:
> +---------------
> | "Alex Mizrahi" <········@users.sourceforge.net> writes:
> | > what matters is if it is a binary or character stream (and
> | > what encoding character has). i would recommend flexi-streams
> | > http://weitz.de/flexi-streams/ if you need some transcoding.
> |
> | So the main difference between all the different kind of streams boils
> | down to if it's a character stream or a binary stream? That makes it
> | easier I guess. Reading the CMUCL/SBCL manual you can get confused by
> | the different stream implementations, you know.
> +---------------
>
> Yes. If you're doing anything at all fancy with CMUCL streams you
> probably want to "look under the covers" at the SYSTEM:FD-STREAM class.
> For example, few people know that CMUCL actually does provide a form
> of bivalent stream on FD-STREAMs (which includes Unix/Linux file
> streams as well as sockets) with the little-known :CLASS option
> to OPEN, e.g.:
>
> > (open "foo" :class 'binary-text-stream :element-type :default)
>
> #<Binary-text Stream for file "foo">
> >
>
> You can use both READ-CHAR and READ-BYTE on such a stream.
Very interesting. I'm just now trying to understand if I can have this
functionality from FLEXI-STREAMS, and now I hear it's included in the
depths of CMUCL. If only I knew where to look for in it SBCL.
It started with sockets and now I'm trying to learn all there is to know
about streams as well! At least I have written code in order to try out
my newfound knowledge...
/Andreas
--
A: Because it fouls the order in which people normally read text.
Q: Why is top-posting such a bad thing?
A: Top-posting.
Q: What is the most annoying thing on usenet and in e-mail?
AD> So the main difference between oll the different kind of streams biols
AD> down to if it's a character stream or a binary stream? That makes it
AD> easier I guess. Reading the CMUCL/SBCL manual you can get confused by
AD> the different stream implementations, you know.
_implementations_. do you know the difference between implementation and the
interface?
you only need interface to do actual programming, and they tried their best
interface to be same no matter what underlying implementation you're using.
same thing for TCP/IP -- TCP/IP stack is an implementation, but you need
just a socket interface to do programming.
"Alex Mizrahi" <········@users.sourceforge.net> writes:
> AD> So the main difference between oll the different kind of streams biols
> AD> down to if it's a character stream or a binary stream? That makes it
> AD> easier I guess. Reading the CMUCL/SBCL manual you can get confused by
> AD> the different stream implementations, you know.
>
> _implementations_. do you know the difference between implementation and the
> interface?
> you only need interface to do actual programming, and they tried their best
> interface to be same no matter what underlying implementation you're using.
>
> same thing for TCP/IP -- TCP/IP stack is an implementation, but you need
> just a socket interface to do programming.
I meant implementations. The manuals mentioned talks about
implementations, and far less about interfaces. I guess they expect the
reader to know hot to find the interfaces and only bother by
implementations if they know what they are doing. It fooled me into
thinking the different implementations mattered. I guess they don't,
usually.
/Andreas
--
A: Because it fouls the order in which people normally read text.
Q: Why is top-posting such a bad thing?
A: Top-posting.
Q: What is the most annoying thing on usenet and in e-mail?
In article <···············@Psilocybe.Update.UU.SE>,
Andreas Davour <·······@updateLIKE.uu.HELLse> wrote:
> "Alex Mizrahi" <········@users.sourceforge.net> writes:
>
> > AD> Anyone care to give me a rough overview of the present state of Lisp
> > AD> network programming?
> >
> > present state of Lisp network programming is pretty good, although
> > there is no standard for it. there are, however, compatibility
> > libraries like trivial-sockets and usocket.
> >
> > you just need to call one function to connect to server, and another one to
> > use socket like Lisp stream:
> [usocket example]
>
> Thanks.
>
> > AD> I once thought that doing a TCP/IP stack in Lisp would be much easier
> > AD> than in C, but now I'm not so sure.
> >
> > TCP/IP stack is already implemented in your OS. how exactly are you
> > going to implement it once more? probably you can do it in Linux --
> > remove existing support, get direct access to hardware, and try to
> > implement your own.. but why?
> >
> > TCP/Ip stack in Lisp would be pretty funny -- whenever Lisp goes
> > garbage collecting, no packets sent.
>
> Well, the reason I mentioned them is that they have been done in LispM
> operating systems, and since all documentation I've found talks about
> sockets like the TCP/IP stack don't even exist.
Here is a TCP/IP stack in Lisp (from the LMI Lambda Lisp Machine):
http://jrm-code-project.googlecode.com/svn/trunk/lambda/network/ip-tcp/
>
> I think gc is handled intelligently in lisp when it's implemented. It
> would probably be interesting to see how!
>
> > AD> All the lisp impelmentations I've investigated seems very preoccupied
> > AD> by this "sockets" thing instead.
> >
> > why, for hell, you need something other than sockets? TCP/IP is
> > essentially a stream of octets, hence it's OK to use it as
> > stream. you'll gain nothing going low level.
> >
> > you only need to go low level if you do some advanced stuff like ping
> > utility. in this case i think you'd need to use FFI and call OS
> > functions (different for each different OS). there's nothing about
> > Lisp in this case.
>
> The thing is (And this is not very clear from my original message, which
> is a mess) that I've looked at a few books about networking and they all
> talk about TCP and UDP and IP packets. Low level stuff. Text about
> network programming always talks about sockets. I didn't understand how
> they fit together.
>
> So, in practical programming you use streams. I didn't know that. There
> are many kind of streams, though. Gray? Simple? Is this where the
> usocket library helps out by hiding the details? What would you do if
> you'd like to implement, say, an extended FTP protocol you just
> invented?
>
> /Andreas
--
http://lispm.dyndns.org/
Rainer Joswig <······@lisp.de> writes:
> In article <···············@Psilocybe.Update.UU.SE>,
> Andreas Davour <·······@updateLIKE.uu.HELLse> wrote:
>
>> "Alex Mizrahi" <········@users.sourceforge.net> writes:
>> > AD> I once thought that doing a TCP/IP stack in Lisp would be much easier
>> > AD> than in C, but now I'm not so sure.
>> >
>> > TCP/IP stack is already implemented in your OS. how exactly are you
>> > going to implement it once more? probably you can do it in Linux --
>> > remove existing support, get direct access to hardware, and try to
>> > implement your own.. but why?
>> >
>> > TCP/Ip stack in Lisp would be pretty funny -- whenever Lisp goes
>> > garbage collecting, no packets sent.
>>
>> Well, the reason I mentioned them is that they have been done in LispM
>> operating systems, and since all documentation I've found talks about
>> sockets like the TCP/IP stack don't even exist.
>
> Here is a TCP/IP stack in Lisp (from the LMI Lambda Lisp Machine):
>
> http://jrm-code-project.googlecode.com/svn/trunk/lambda/network/ip-tcp/
Interesting. It also reminds me of another thing. Are there any
implementation of Flavors out there that one can use to play with old
code like this? I've found a few, but they seems to take quite a bit of
work to compile on a modern lisp. They all seems to be for the CL of
CLtL1.
/Andreas
--
A: Because it fouls the order in which people normally read text.
Q: Why is top-posting such a bad thing?
A: Top-posting.
Q: What is the most annoying thing on usenet and in e-mail?
Andreas Davour <·······@updateLIKE.uu.HELLse> writes:
> All
> the lisp impelmentations I've investigated seems very preoccupied by
> this "sockets" thing instead. Is there a TCP/IP implementation to look
> at and investigate?
Sure.
http://fresh.homeunix.net/~luke/misc/repo/slitch/
http://common-lisp.net/cgi-bin/viewcvs.cgi/movitz/losp/lib/net/?root=movitz
Not that you'd want to, if your goal is to do application programming.
--
Juho Snellman
Juho Snellman <······@iki.fi> writes:
> Andreas Davour <·······@updateLIKE.uu.HELLse> writes:
>> All
>> the lisp impelmentations I've investigated seems very preoccupied by
>> this "sockets" thing instead. Is there a TCP/IP implementation to look
>> at and investigate?
>
> Sure.
>
> http://fresh.homeunix.net/~luke/misc/repo/slitch/
> http://common-lisp.net/cgi-bin/viewcvs.cgi/movitz/losp/lib/net/?root=movitz
>
> Not that you'd want to, if your goal is to do application programming.
Well, my goal is to learn how to do application programming, and after
that how to write an operating system. Maybe I'll die of old age before
the latter.
It's good to know where to check, anyway.
How on earth could I forget to check movitz!?
/Andreas
--
A: Because it fouls the order in which people normally read text.
Q: Why is top-posting such a bad thing?
A: Top-posting.
Q: What is the most annoying thing on usenet and in e-mail?
Andreas Davour <·······@updateLIKE.uu.HELLse> writes:
> [...]
> So how is it in the lisp world?
It's implementation dependant.
> When you are programming a network
> server, do you start with some socket library, or do you try to build
> TCP/IP packets by hand?
You start with a library.
> Anyone care to give me a rough overview of the present state of Lisp
> network programming?
The Implementation Notes of clisp contains a section about its SOCKET
package, with an example:
http://clisp.cons.org/impnotes/socket.html
clisp also provides a RAW-SOCKET package when you need more a lower
level socket API.
--
__Pascal Bourguignon__
·························@anevia.com
http://www.anevia.com
···@anevia.com (Pascal J. Bourguignon) writes:
> Andreas Davour <·······@updateLIKE.uu.HELLse> writes:
>> Anyone care to give me a rough overview of the present state of Lisp
>> network programming?
>
> The Implementation Notes of clisp contains a section about its SOCKET
> package, with an example:
>
> http://clisp.cons.org/impnotes/socket.html
>
> clisp also provides a RAW-SOCKET package when you need more a lower
> level socket API.
Interesting pointer. Thanks.
/Andreas
--
A: Because it fouls the order in which people normally read text.
Q: Why is top-posting such a bad thing?
A: Top-posting.
Q: What is the most annoying thing on usenet and in e-mail?
On Mon, 10 Dec 2007 08:12:26 +0100, Andreas Davour
<·······@updateLIKE.uu.HELLse> wrote:
>
>I've had a ok grap on TCP/IP for a while now, but since I have not done
>any real TCP/IP programming I have never before really encountered
>sockets.
Then you really *don't* have a grasp of TCP/IP.
>Having looked at some old Lisp code I realize that streams over networ
>connections, raw IP packets and servers over sockets is all one big
>happy circus. Sockets doesn't fit in the big fine shiny OSI stack, but
>it seems to be the one thing everyone use for doing network
>programming.
??? Sockets fit very neatly into the OSI model - they are layer 3
devices. Streams as in CL are much higher in the OSI model - they
are layer 5 (or 6, depending on who you talk to) concepts.
>So how is it in the lisp world? When you are programming a network
>server, do you start with some socket library, or do you try to build
>TCP/IP packets by hand?
Depends on what you want to do. Some libraries don't support "raw"
packets so if you need them you may have to use FFI to access the OS
API. However, only applications that need to monitor or control
networks at the link layer need to use raw packets. Virtually all
other applications can make due with IP datagrams or TCP streams.
>Anyone care to give me a rough overview of the present state of Lisp
>network programming?
For most application uses, most CL implementations have perfectly good
socket libraries available.
>I once thought that doing a TCP/IP stack in Lisp would be much easier
>than in C, but now I'm not so sure. What do you actually build upon? All
>the lisp impelmentations I've investigated seems very preoccupied by
>this "sockets" thing instead. Is there a TCP/IP implementation to look
>at and investigate?
You don't really mean "doing" a TCP/IP stack - you mean "using" one.
Actually implementing a TCP/IP stack requires system level and device
level programming. Based on your comments, I don't think you are in a
position to even begin such a project.
If you really care to see it, source (in C) is available for the
TCP/IP implementations used by BSD Unix, Linux and several other
operating systems. Almost all existing TCP/IP implementations are
based in large part on BSD's implementation.
George
--
for email reply remove "/" from address
George Neuner <·········@/comcast.net> writes:
> On Mon, 10 Dec 2007 08:12:26 +0100, Andreas Davour
> <·······@updateLIKE.uu.HELLse> wrote:
>
>>
>>I've had a ok grap on TCP/IP for a while now, but since I have not done
>>any real TCP/IP programming I have never before really encountered
>>sockets.
>
> Then you really *don't* have a grasp of TCP/IP.
Let's just say 'ok' is a few steps below 'good'. I did take a course
once about TCP/IP and read a few books. After that I knew a lot about
how an IP header is formed and in what OSI layer they fit. I always
wondered how they were justed for anything except ftp and telnet which
is what everyone used as examples. I still know about those bits in a
UDP datagram, I have an 'ok' grasp on that.
There's *a lot* I don't know about TCP/IP!
>>Having looked at some old Lisp code I realize that streams over networ
>>connections, raw IP packets and servers over sockets is all one big
>>happy circus. Sockets doesn't fit in the big fine shiny OSI stack, but
>>it seems to be the one thing everyone use for doing network
>>programming.
>
> ??? Sockets fit very neatly into the OSI model - they are layer 3
> devices. Streams as in CL are much higher in the OSI model - they
> are layer 5 (or 6, depending on who you talk to) concepts.
Very interesting. For some reason I've never seen this mentioned
before. Maybe I've found just the bad books, and took a bad course.
So sockets actually bypass the transport layer? It almost sounds like
they were implemented by someone who either didn't know about or didn't
like the rest of the stack, or are there any good reason not to utilize
TCP for the functionality that sockets provide?
Geat fun to finally hear someone mention something happening at level 5
and 6. The litterature I've read have been very occupied with layer 2-4
for some reason.
>>So how is it in the lisp world? When you are programming a network
>>server, do you start with some socket library, or do you try to build
>>TCP/IP packets by hand?
>
> Depends on what you want to do. Some libraries don't support "raw"
> packets so if you need them you may have to use FFI to access the OS
> API. However, only applications that need to monitor or control
> networks at the link layer need to use raw packets. Virtually all
> other applications can make due with IP datagrams or TCP streams.
That would be the reason to have the underlaying level written in
something more sensible than C, to not be dependant on FFI.
But, I guess I will not use raw packets soon anyway.
Inventing new protocols on the packet/datagram level you would actually
get by with "raw" packets?
>>I once thought that doing a TCP/IP stack in Lisp would be much easier
>>than in C, but now I'm not so sure. What do you actually build upon? All
>>the lisp impelmentations I've investigated seems very preoccupied by
>>this "sockets" thing instead. Is there a TCP/IP implementation to look
>>at and investigate?
>
> You don't really mean "doing" a TCP/IP stack - you mean "using" one.
>
> Actually implementing a TCP/IP stack requires system level and device
> level programming. Based on your comments, I don't think you are in a
> position to even begin such a project.
>
> If you really care to see it, source (in C) is available for the
> TCP/IP implementations used by BSD Unix, Linux and several other
> operating systems. Almost all existing TCP/IP implementations are
> based in large part on BSD's implementation.
I have, after reading those books about datagram bits, actually read a
bit of the bsd source, but lisp is far easier to read, I think.
You are perfectly correct to suggest that actually implement a stack
myself is way beyond my present capabilities! My problem is I want to
know how things work, all the way down, and sometimes don't know when to
stop. You know how they say that most programmers most often start
projects to "scratch an itch"? My itch is the operating system!
Thanks for giving me good feedback. I know a lot more now than before
posting my original post.
--
A: Because it fouls the order in which people normally read text.
Q: Why is top-posting such a bad thing?
A: Top-posting.
Q: What is the most annoying thing on usenet and in e-mail?
On Wed, 12 Dec 2007 05:52:40 +0100, Andreas Davour wrote:
> Let's just say 'ok' is a few steps below 'good'. I did take a course
> once about TCP/IP and read a few books. After that I knew a lot about
> how an IP header is formed and in what OSI layer they fit. I always
> wondered how they were justed for anything except ftp and telnet which
> is what everyone used as examples. I still know about those bits in a
> UDP datagram, I have an 'ok' grasp on that.
TCP/IP doesn't exactly fit the OSI layer model in many respects, and
you'll find plenty of people who will argue that that's because TCP/IP
works in practice, and OSI doesn't. (I'm not one of those people: OSI
was essentially gone before I got to do much network programming, so I
never got a good look at it being used in practice.)
> So sockets actually bypass the transport layer?
Sockets abstract part of the transport layer. Vis you can have several
different kinds of sockets (IPv4, IPv6, Unix Domain, etc) that let you
communicate using different transport protocols.
> It almost sounds like
> they were implemented by someone who either didn't know about or didn't
> like the rest of the stack, or are there any good reason not to utilize
> TCP for the functionality that sockets provide?
Yes. There are applications (particularly real-time) where it's better
to lose a packet than to delay delivery of subsequent packets while you
ask for a re-transmit. Just by way of example. There are lots of other
protocols that use/prefer UDP and the like. These still use sockets
(unless you're using STREAMS (the shouted, AT&T/Solaris variety)).
>> Depends on what you want to do. Some libraries don't support "raw"
>> packets so if you need them you may have to use FFI to access the OS
>> API. However, only applications that need to monitor or control
>> networks at the link layer need to use raw packets. Virtually all
>> other applications can make due with IP datagrams or TCP streams.
>
> That would be the reason to have the underlaying level written in
> something more sensible than C, to not be dependant on FFI.
>
> But, I guess I will not use raw packets soon anyway.
>
> Inventing new protocols on the packet/datagram level you would actually
> get by with "raw" packets?
There's also the issue of multiplexing/demultiplexing packets to
applications, in a multi-tasking operating system. Below the socket
layer you really want to be in the operating system, so that you can pass
packets to the applications that are waiting for them. (You might want
to have a look at how the Minix3 network stack works: it doesn't use the
Berkeley Sockets model: network connections appear in the filesystem name
space, where regular file operations work on them.)
> You are perfectly correct to suggest that actually implement a stack
> myself is way beyond my present capabilities! My problem is I want to
> know how things work, all the way down, and sometimes don't know when to
> stop. You know how they say that most programmers most often start
> projects to "scratch an itch"? My itch is the operating system!
There have been Lisp operating systems before (from the silicon up), and
there are projects to build new ones. These will probably/necessarily
have network stacks implemented in Lisp. There's no reason why they
couldn't be (unless your Lisp is running on top of another OS).
Cheers,
--
Andrew
Andrew Reilly <···············@areilly.bpc-users.org> writes:
> On Wed, 12 Dec 2007 05:52:40 +0100, Andreas Davour wrote:
>>> Depends on what you want to do. Some libraries don't support "raw"
>>> packets so if you need them you may have to use FFI to access the OS
>>> API. However, only applications that need to monitor or control
>>> networks at the link layer need to use raw packets. Virtually all
>>> other applications can make due with IP datagrams or TCP streams.
>>
>> That would be the reason to have the underlaying level written in
>> something more sensible than C, to not be dependant on FFI.
>>
>> But, I guess I will not use raw packets soon anyway.
>>
>> Inventing new protocols on the packet/datagram level you would actually
>> get by with "raw" packets?
>
> There's also the issue of multiplexing/demultiplexing packets to
> applications, in a multi-tasking operating system. Below the socket
> layer you really want to be in the operating system, so that you can pass
> packets to the applications that are waiting for them. (You might want
> to have a look at how the Minix3 network stack works: it doesn't use the
> Berkeley Sockets model: network connections appear in the filesystem name
> space, where regular file operations work on them.)
I knew there was interesting things going on in Minix3. Thanks for
reminding me of that.
/Andreas
--
A: Because it fouls the order in which people normally read text.
Q: Why is top-posting such a bad thing?
A: Top-posting.
Q: What is the most annoying thing on usenet and in e-mail?
On Wed, 12 Dec 2007 05:52:40 +0100, Andreas Davour
<·······@updateLIKE.uu.HELLse> wrote:
>George Neuner <·········@/comcast.net> writes:
>
>> On Mon, 10 Dec 2007 08:12:26 +0100, Andreas Davour
>> <·······@updateLIKE.uu.HELLse> wrote:
>
>>>Having looked at some old Lisp code I realize that streams over networ
>>>connections, raw IP packets and servers over sockets is all one big
>>>happy circus. Sockets doesn't fit in the big fine shiny OSI stack, but
>>>it seems to be the one thing everyone use for doing network
>>>programming.
>>
>> ??? Sockets fit very neatly into the OSI model - they are layer 3
>> devices. Streams as in CL are much higher in the OSI model - they
>> are layer 5 (or 6, depending on who you talk to) concepts.
>
>Very interesting. For some reason I've never seen this mentioned
>before. Maybe I've found just the bad books, and took a bad course.
>
>So sockets actually bypass the transport layer? It almost sounds like
>they were implemented by someone who either didn't know about or didn't
>like the rest of the stack, or are there any good reason not to utilize
>TCP for the functionality that sockets provide?
Your first mistake is assuming that the ISO model was designed to be
used from the top - there are valid reasons for applications to access
it at several levels.
The only real stipulation of the ISO model is that layer N software
should depend solely on layer N-1 software and not skip over
intermediate layers. However, strict layering gives really lousy
performance - in a good network stack a *lot* of effort goes into
minimizing copying of data between logical layers.
Reality is that the ISO layering model is routinely violated in
practice. All NICs implement layer 1 in hardware and part of layer 2
in firmware. Software stacks such as TCP/IP, SPX/IPX, TLI, NetBIOS,
etc. all implement the remainder of layer 2, and layers 3 and 4. TLI
is notable because it also implements layer 5.
AFA sockets are concerned: TCP is a very heavyweight layer 4 protocol
designed for unreliable wide area networks. On a LAN, the odds of a
packet being lost or arriving out of sequence are virtually
non-existent - some other layer 4 protocols like NBT and SPX take
advantage of that to gain performance, but they don't work on a WAN.
TCP on a LAN is serious overkill - many LAN applications can make do
using the simpler layer 3 IP protocol, either by itself or with a
custom layer 4 protocol over IP.
>Geat fun to finally hear someone mention something happening at level 5
>and 6. The litterature I've read have been very occupied with layer 2-4
>for some reason.
Layers 5..7 are application specific - you won't find ANY general
network literature that does much more than mention them.
George
--
for email reply remove "/" from address
George Neuner <·········@/comcast.net> wrote:
+---------------
| AFA sockets are concerned: TCP is a very heavyweight layer 4
| protocol designed for unreliable wide area networks. On a LAN,
| the odds of a packet being lost or arriving out of sequence are
| virtually non-existent ...
+---------------
**BULL!!** Sorry for shouting, Sir, but this assertiong shows an
almost complete lack of real-world experience. Yes, sending/receiving
*isolated* packets at a *slow* rate on an totally *idle* LAN tends
to be quite reliable, but throw in a few bridges, switches, routers,
VLANs, NAT boxes, flaky NICs, poorly-constructed cables & patch
panels -- all the stuff you find in any *realistic* LAN -- and
the picture is far, far different!! Lost, delayed, duplicated, and
out-of-order packets are all too common on real production LANs.
+---------------
| TCP on a LAN is serious overkill ...
+---------------
Again, rubbish. With a single TCP session any modern OS/CPU/NIC
combination can easily saturate even a gigabit Ethernet [~117 MB/s
of user data end-to-end, within 1% of "perfect"].
+---------------
| ... many LAN applications can make do using the simpler layer 3 IP
| protocol, either by itself or with a custom layer 4 protocol over IP.
+---------------
Yeah, right. But by the time you finish adding the error-detection,
timeouts [for the errors that *do* happen], and retransmission
policies you'll need for a *real* LAN, you usually would have done
better to just use TCP [or perhaps UDP for VoIP] from the beginning.
I can't count the number of times that people have tried to code up
a "simpler alternative to TCP", only to have it blow up in their faces
and bring their LAN to its knees because they didn't understand
sequencing, flow control, backoff during retransmission, etc.
(*sigh*) This whole "TCP-on-a-LAN Woozle" seems to need re-slaying
every few years [which says something frightening about the (lack of)
community memory among high-tech folk]. Please recall RFC 872 [only
8 pages], which was already old news when it was published in 1982:
ftp://ftp.rfc-editor.org/in-notes/rfc872.txt
http://www.faqs.org/rfcs/rfc872.html
TCP-ON-A-LAN September 1982
M. A. Padlipsky M82-48
The Mitre Corporation
Bedford, Massachusetts
Abstract
The sometimes-held position that the DoD Standard Transmission
Control Protocol (TCP) and Internet Protocol (IP) are inappropriate
for use "on" a Local Area Network (LAN) is shown to be fallacious.
The paper is a companion piece to M82-47, M82-49, M82-50, and M82-51.
...
The footnote to reference [3] in the above is most enlightening:
* In all honesty, as far as I know I started the rumor that TCP
might be overkill for a LAN at that meeting. At the next TCP
design meeting, however, they separated IP out from TCP, and
everything's been alright for about three years now--except
for getting the rumor killed. (I'd worry about Woozles turning
into roosting chickens if it weren't for the facts that:
1. People tend to ignore their local guru; 2. I was trying
to encourage the IP separation; and 3. All I ever wanted
was some empirical data.)
Unfortunately, it appears that, despite the best of intentions,
MAP *still* hasn't managed to kill the rumor... (*sigh*)
-Rob
p.s. A slight update of RFC 872 [plus a prefacetory afterword]
can be found as the chapter "Slaying the TCP-on-a-LAN Woozle"
in this little classic:
Padlipsky, M. A., "The Elements of Networking Style and other
essays and animadversions on the art of intercomputer networking",
Prentice-Hall (Englewood Cliffs, NJ), 1985,
ISBN 0-13-268129-3, 0-13-268111-0 (paper).
This self-claimed "world's only known Constructively Snotty computer
science book" is now back in print as ISBN 0-595-08879-1, $19.95, from
<http://www.iuniverse.com/bookstore/book_detail.asp?isbn=0-595-08879-1>.
[Also see <http://world.std.com/~wdr/map-bibl.html>.] The "slogans
suitable for framing" in the back are worth the price all by themselves!
-----
Rob Warnock <····@rpw3.org>
627 26th Avenue <URL:http://rpw3.org/>
San Mateo, CA 94403 (650)572-2607
····@rpw3.org (Rob Warnock) writes:
> George Neuner <·········@/comcast.net> wrote:
> +---------------
> | AFA sockets are concerned: TCP is a very heavyweight layer 4
> | protocol designed for unreliable wide area networks. On a LAN,
> | the odds of a packet being lost or arriving out of sequence are
> | virtually non-existent ...
> +---------------
>
> **BULL!!** Sorry for shouting, Sir, but this assertiong shows an
> almost complete lack of real-world experience.
Heh. Sounds a bit like
(sublis '((c-or-fortran-program . ip-application)
(common-lisp . tcp))
(tenth *greenspun-rules*))
Note well that this was almost surely my first (and is hopefully to be
my only) use of CL:TENTH ever ... I realize that suggesting those
rules be implemented as a list pessimizes their access, but neither
(nth 9 *greenspun-rules*) nor (aref *greenspun-rules* 9) looked
perspicuous... :(
(See http://en.wikipedia.org/wiki/Greenspun's_Tenth_Rule for more details.)
Kent M Pitman <······@nhplace.com> wrote:
+---------------
| ····@rpw3.org (Rob Warnock) writes:
| > George Neuner <·········@/comcast.net> wrote:
| > +---------------
| > | AFA sockets are concerned: TCP is a very heavyweight layer 4
| > | protocol designed for unreliable wide area networks. On a LAN,
| > | the odds of a packet being lost or arriving out of sequence are
| > | virtually non-existent ...
| > +---------------
| >
| > **BULL!!** Sorry for shouting, Sir, but this assertiong shows an
| > almost complete lack of real-world experience.
|
| Heh. Sounds a bit like
| (sublis '((c-or-fortran-program . ip-application)
| (common-lisp . tcp))
| (tenth *greenspun-rules*))
+---------------
Exactly so! [...even if it took me a minute to parse your SUBLIS.] ;-}
And that was also basically one of the main points of RFC 872:
ftp://ftp.rfc-editor.org/in-notes/rfc872.txt
...
So if all you want is the ability to demultiplex data
streams from your Host-Host protocol, you use UDP, not TCP.
("You" is usually supposed to be a Packetized Speech
protocol, but doesn't have to be.) (And we'll worry
about Flow Control some other time.)
...
A word of caution, though: those applications probably
will need protocols of some sort--and they'll probably
need some sort of Host-Host protocol under them, so
unless you relish maintaining "parallel" suites of
protocols.... that is, you really would be better off
with TCP most of the time locally anyway, because you've
got to have it to talk to the catenet [read: "Internet"]
and it's a nuisance to have "something else" to talk
over the LAN--when, of course, what you're talking requires
a Host-Host protocol.
...
In fact, about the only thing that has changed since RFC 872
was written in 1982 is that TCP implementations have gotten
*so* much faster that it is very difficult to find a NIC and
LAN that *won't* run at "wire speed" on a LAN. As I said before,
most OS/CPU/NIC combos will saturate a GbE [~117 MB/s user data]
with a single TCP connection. Many higher-speed LANs have reported
TCP running at a substabtial fraction of wire speed: (1) SGI's
MIPS/Irix TCP ran at 550 MB/s over its 800 MB/s GSN [a.k.a.
ANSI HIPPI-6400]; (2) I've personally seen 380 MB/s of
TCP-over-Infiniband (4xDDR/Linux/Opteron), and subsequently
been told that that was "slow". ;-}
So, yes, IMHO the flavor, at least, of your SUBLIS is correct.
[Modulo the bit about IP/UDP sometimes being the "right" answer...]
-Rob
-----
Rob Warnock <····@rpw3.org>
627 26th Avenue <URL:http://rpw3.org/>
San Mateo, CA 94403 (650)572-2607
On Fri, 14 Dec 2007 00:08:00 -0600, ····@rpw3.org (Rob Warnock) wrote:
>George Neuner <·········@/comcast.net> wrote:
>+---------------
>| AFA sockets are concerned: TCP is a very heavyweight layer 4
>| protocol designed for unreliable wide area networks. On a LAN,
>| the odds of a packet being lost or arriving out of sequence are
>| virtually non-existent ...
>+---------------
>
>**BULL!!** Sorry for shouting, Sir, but this assertiong shows an
>almost complete lack of real-world experience.
20 years is "lacking experience" ... I'll have to make a note of that.
>Yes, sending/receiving
>*isolated* packets at a *slow* rate on an totally *idle* LAN tends
>to be quite reliable, but throw in a few bridges, switches, routers,
>VLANs, NAT boxes, flaky NICs, poorly-constructed cables & patch
>panels -- all the stuff you find in any *realistic* LAN -- and
>the picture is far, far different!! Lost, delayed, duplicated, and
>out-of-order packets are all too common on real production LANs.
I've written applications for the worst LANs you can imagine. Every
kind of hardware: Ethernets, token rings, token buses, point-to point
RS-422, HPIB ... running real time monitoring and control apps in
factories settings with flaky power, overheating equipment, trucks
driving over floor cable runs, cabling hung on fluorescent lights,
unshielded equipment putting out enough EMF to crank automobiles,
people who buy used equipment to save a few pennies ...
>+---------------
>| TCP on a LAN is serious overkill ...
>+---------------
>
>Again, rubbish. With a single TCP session any modern OS/CPU/NIC
>combination can easily saturate even a gigabit Ethernet [~117 MB/s
>of user data end-to-end, within 1% of "perfect"].
I don't know what you think you're measuring, but 117 MB/s user data
isn't even possible without compression. For networks, giga means
10^9, not 2^30. Ethernet is async requiring 10 bits/byte. 10^8 B/s =
95.37 * 2^20 bytes. And that total must include packet preambles,
headers and trailers.
The theoretical maximum for GB Ethernet may be 99.1%, but it is only
achievable on an otherwise quiet LAN. It takes very little other
traffic to screw it up.
Add to that the fact that exponential back off from CSMA/CD on
Ethernet begins to severely impact performance when usages drives
collision rates above 40%. For high usage rates, DDI rings are far
more performant.
TCP was designed for WANs and by comparison with other LAN streaming
protocols, it is quite heavy weight. That's why IBM, Novell and DEC
spent millions developing NetBIOS, SPX and DECnet ... because TCP was
so brilliantly performant on LANs. I never used DECnet even though I
personally know Stuart Wecker who led the team that designed it. I
have used NetBIOS and SPX extensively - both are measurably faster
than TCP on the same hardware. TCP can be tuned to the environment,
but that is beyond most users. Most stacks today perform some self
tuning within certain ranges of parameters, but they can't handle
everything they may encounter.
George
--
for email reply remove "/" from address
George Neuner <·········@/comcast.net> wrote:
+---------------
| ····@rpw3.org (Rob Warnock) wrote:
| >+---------------
| >| TCP on a LAN is serious overkill ...
| >+---------------
| >
| >Again, rubbish. With a single TCP session any modern OS/CPU/NIC
| >combination can easily saturate even a gigabit Ethernet [~117 MB/s
| >of user data end-to-end, within 1% of "perfect"].
|
| I don't know what you think you're measuring, but 117 MB/s user data
| isn't even possible without compression.
+---------------
Of course it is!! GbE is *125* MB/s peak user data rate; 118.66 MB/s
is the theoretical maximum sustained user data rate you get when you
take out link-level framing, IP, and TCP overheads. 117+ MB/s -- a
little less than that -- is readily attainable on commodity hardware.
+---------------
| For networks, giga means 10^9, not 2^30.
+---------------
Of course. And by 117 MB/s I meant 117,000,000 (decimal) bytes per second.
+---------------
| Ethernet is async requiring 10 bits/byte.
+---------------
***BZZZTTT!!*** Sorry, back to school. Ethernet was *never* "async",
always *synchronous* [once the initial preamble has locked up the
receive PLL]. Transmission on Gigabit Ethernet (GbE) runs at 125 MBaud
(8 ns/symbol), using PAM-5 coding on each pair, with the link-level
symbols encoded on all four twisted pairs at the same time. PAM-5 times
4 pairs = (expt 5 4) = 625 different symbols possible per signaling
interval. Of those, only the "best" 512 symbols are used for user
data [some of the others are used for framing], alternating between
two sets of 256 so as to maintain clocking and also shaping the
radiated EMI to stay within FCC limits. That means that for each
signaling interval there's really only a choice of 256 different
symbols, which are used to encode 8 user data bits. 125 MBaud times
8 bits/symbol = 1000 Mbit/s (125 MB/s) of *user* data (peak).
[By the way, this is the *same* 125 MBaud rate used for 100BASE-TX.
GbE runs 10 times faster than 100BASE-TX because GbE: (1) uses PAM-5
instead of MLT-3 (which was really only a binary code, under the hood);
(2) *doesn't* use the FDDI 8b/10b precoding for clock recovery, but
encodes across all four pairs so that there are fewer wasted symbols,
enough to allow clock recovery to be achieved by flipping between symbol
sets; and (3) uses all four pairs at once in both directions instead
of only one pair in each direction. All of which means that GbE gets
8 bits/Baud of user data instead of 100BASE-TX's 0.8 bit/Baud.]
The maximum frame size [excluding "jumbo frames"] is 1538 bytes,
comprised of preamble (8 bytes), Ethernet link-level header
(6 + 6 + 2 = 14 bytes), Ethernet payload (1500 bytes), CRC-32 (4 bytes),
and interpacket recovery time (12 bytes). [The interpacket gap is
actually silence on 10BASE-T or a stream of IDLEs on 100BASE-T
and GbE, but in any case it's 12 bytes worth of time.] Since the
Ethernet payload *includes* the IP (20 bytes) and TCP (20 bytes)
headers, the maximum user data per TCP packet is only 1460 bytes,
and thus the theoretical maximum sustained TCP user data rate is
(* 125e6 (/ 1460 1538)) ==> 118.66 MB/s [118,660,598.18 bytes/s
if you want to be picky], or 94.93% of wire speed.
As I've said before and will absolutely stand by, you can easily
get 98.6% of *that*, or 117 MB/s, on commodity hardware these days.
+---------------
| 10^8 B/s = 95.37 * 2^20 bytes.
+---------------
I have no idea what these number refer to here. I mentioned no
powers of 2 at all. My numbers show that the theoretical maximum
sustained TCP user data rate on GbE is ~94.93% of the wire speed,
or 949,284,785+ bit/s, and that one can actually achieve 98.6% of
*that* in practice, or 93.6% of peak wire speed = 117 MB/s from
the user process memory of one machine to the user process memory
of another machine. [With or without a GbE switch in the path, by
the way, provided that it's a reasonably performant switch.]
+---------------
| And that total must include packet preambles, headers and trailers.
+---------------
I included all of that, see above.
+---------------
| The theoretical maximum for GB Ethernet may be 99.1% ...
+---------------
It's not; it's only 94.93% (see above).
+---------------
| ...but it is only achievable on an otherwise quiet LAN.
| It takes very little other traffic to screw it up.
+---------------
But since in practice all GbE LANs are *switched*, if you buy
non-blocking switches traffic from one path won't interfere
with another path at all.
+---------------
| Add to that the fact that exponential back off from CSMA/CD on
| Ethernet begins to severely impact performance when usages drives
| collision rates above 40%.
+---------------
Oh, boy. Now you're just embarrassing yourself with that old myth.
Look, an Ethernet collision only costs at most one slot time (64 bytes)
plus an interpacket recovery time (12 bytes). A 40% collision rate
would therefore only add (* 0.4 (+ 64 12)) 30 bytes to the average
frame overhead, lowering the efficiency from 94.93% (1460/1538)
to 93% (1460/1568). No way can you consider that to be a "severe
performance impact"!!
Back at SGI, even on ancient half-duplex 10BASE-T, we routinely
got >1160 KB/s in our benchmarks. (Which, I'll admit, is slightly
less than the fraction of wire speed I've seen on GbE, but I did
say "half-duplex", and even with "delayed ACKs" turned on ACKs do
take up some bandwidth.)
Besides, there are no collisions at all on switched Gigabit Ethernet!!!
[Buffer congestion in switches, maybe, but no collisions.]
+---------------
| For high usage rates, DDI rings are far more performant.
+---------------
Sounds like someone with a product to sell.
+---------------
| TCP was designed for WANs and by comparison with other LAN streaming
| protocols, it is quite heavy weight. That's why IBM, Novell and DEC
| spent millions developing NetBIOS, SPX and DECnet ... because TCP was
| so brilliantly performant on LANs.
+---------------
Actually, they spent millions developing NetBIOS, SPX and DECnet
because they hoped they could lock customers into using *their*
proprietary protocols instead of open, standards-conforming,
*free* protocols. They lost.
+---------------
| I never used DECnet even though I personally know Stuart Wecker
| who led the team that designed it.
+---------------
Yes, yes, Stu's a really nice guy. His protocol description of
DDCMP was a marvel of clear writing. One could easily implement
the protocol directly from the spec -- and some non-DEC folks did!
By the way, did I ever mention that a little company I worked
for called DCA in Atlanta was shipping a networking product for
revenue that used DDCMP as the link-level protocol two years
*before* DEC shipped anything with it?!? We even used Stu's
software CRC-16 algorithm in our driver, this one:
Wecker, S., "A Table-Lookup Algorithm for Software
Computation of Cyclic Redundancy Check (CRC),"
Digital Equipment Corporation memorandum, 1974.
Now that we're done name-dropping, back to GbE... ;-} ;-}
+---------------
| I have used NetBIOS and SPX extensively - both are measurably faster
| than TCP on the same hardware. TCP can be tuned to the environment,
| but that is beyond most users. Most stacks today perform some self
| tuning within certain ranges of parameters, but they can't handle
| everything they may encounter.
+---------------
All I can say, George, is that you don't seem to have kept current.
TCP readily runs at wire speed on GbE these days, using commodity
hardware and readily-available operating systems (Linux, FreeBSD, etc.).
117 MB/s of "ttcp" or "netperf" is just what we routinely expect
to see. If we *don't* get it, we go look to see what's broken.
In conclusion: There are occasionally valid reasons one might choose
*not* to use TCP. The fact that it's "on a LAN" isn't one of them.
-Rob
-----
Rob Warnock <····@rpw3.org>
627 26th Avenue <URL:http://rpw3.org/>
San Mateo, CA 94403 (650)572-2607
George Neuner <·········@/comcast.net> writes:
> On Wed, 12 Dec 2007 05:52:40 +0100, Andreas Davour
>>> ??? Sockets fit very neatly into the OSI model - they are layer 3
>>> devices. Streams as in CL are much higher in the OSI model - they
>>> are layer 5 (or 6, depending on who you talk to) concepts.
>>
>>Very interesting. For some reason I've never seen this mentioned
>>before. Maybe I've found just the bad books, and took a bad course.
>>
>>So sockets actually bypass the transport layer? It almost sounds like
>>they were implemented by someone who either didn't know about or didn't
>>like the rest of the stack, or are there any good reason not to utilize
>>TCP for the functionality that sockets provide?
>
> Your first mistake is assuming that the ISO model was designed to be
> used from the top - there are valid reasons for applications to access
> it at several levels.
>
> The only real stipulation of the ISO model is that layer N software
> should depend solely on layer N-1 software and not skip over
> intermediate layers. However, strict layering gives really lousy
> performance - in a good network stack a *lot* of effort goes into
> minimizing copying of data between logical layers.
I did assume that the model was intended to be used from the top. My
bad.
> Reality is that the ISO layering model is routinely violated in
> practice. All NICs implement layer 1 in hardware and part of layer 2
> in firmware. Software stacks such as TCP/IP, SPX/IPX, TLI, NetBIOS,
> etc. all implement the remainder of layer 2, and layers 3 and 4. TLI
> is notable because it also implements layer 5.
I guess it's the old truism about what happens to well made plans when
they encounter reality. It was fun that you mention TLI, which made me
check it out on wikipedia, and thus I learned that the old Mac OS
OpenTransport thing is a SystemV'ism. Interesting.
/andreas
--
A: Because it fouls the order in which people normally read text.
Q: Why is top-posting such a bad thing?
A: Top-posting.
Q: What is the most annoying thing on usenet and in e-mail?