What would be the best way to do networking using both TCP *and* UDP
protocols using CL ?
I'm wondering why the hell this has not been put into the CL spec. CL
offers abilities to handle a huge amount of filesystem types but just
ignores networking. Looks like I'll have to use a CL implementation on
Plan 9 to achieve this ;)
-Nicolas
On Feb 8, 2:52 pm, ············@gmail.com wrote:
> What would be the best way to do networking using both TCP *and* UDP
> protocols using CL ?
http://www.sbcl.org/manual/Networking.html ..?
--
Lars Rune Nøstdal
On 8 fév, 17:17, "Lars Rune Nøstdal" <···········@gmail.com> wrote:
> On Feb 8, 2:52 pm, ············@gmail.com wrote:
>
> > What would be the best way to do networking using both TCP *and* UDP
> > protocols using CL ?
>
> http://www.sbcl.org/manual/Networking.html..?
>
> --
> Lars Rune Nøstdal
Ok, i just figure out sbcl offers both TCP and UDP protocls. Good
news ;)
Do you know if multiplexing (as with posix poll function) is also
available ?
On Feb 8, 2:52 pm, ············@gmail.com wrote:
> What would be the best way to do networking using both TCP *and* UDP
> protocols using CL ?
> I'm wondering why the hell this has not been put into the CL spec. CL
> offers abilities to handle a huge amount of filesystem types but just
> ignores networking. Looks like I'll have to use a CL implementation on
> Plan 9 to achieve this ;)
>
> -Nicolas
CL doesn't have networking (among other things) as part of the
specification because it is stuck in the eighties. Unlike its data
structures the language specification and the community around it seem
to be immutable. So, from someone who has just asked some of these
questions and also gone in the search of networking, it is platform
specific and you should really just spend a bunch of time looking for
libraries that don't suck rather than using well tested ones that
should be built-in. I recommend using sbcl because it is open source
and active, while also including a bsd-sockets layer that is easy to
use and solid. I can send you example code of a simple threaded
client/server echo test program if you want.
Cheers,
Jeff
On Feb 8, 4:57 pm, Jeff <······@gmail.com> wrote:
>
> CL doesn't have networking (among other things) as part of the
> specification because it is stuck in the eighties. Unlike its data
> structures the language specification and the community around it seem
> to be immutable.
The specification is mutable, but would require a ANSI standards
process to mutate, for which no one is interested in spending the
(large amount of) money, time, and effort required.
················@gmail.com" <············@gmail.com> writes:
> On Feb 8, 4:57 pm, Jeff <······@gmail.com> wrote:
>
> > CL doesn't have networking (among other things) as part of the
> > specification because it is stuck in the eighties. Unlike its data
> > structures the language specification and the community around it seem
> > to be immutable.
Stuck in the 80's? Should you ever want to be a consensus builder,
one who forges standard, I would say that leading with this kind of
non-constructive remark is not your strongest card.
Also, you don't need to change the CL standard to get a standard for
networks. It can be done as a separate standard. So blaming it on
the fact that CL is stable is silly.
> The specification is mutable, but would require a ANSI standards
> process to mutate, for which no one is interested in spending the
> (large amount of) money, time, and effort required.
Indeed. Those things matter a lot, and more than people often think.
But I'll add a few other remarks in case that's helpful...
For a while I was an individual paying member of X3J13, but that was a
very pricey proposition. It's hard to be a member without being
funded by a money-making organization. Standards used to be funded
off the revenue of companies making or using Lisp. It isn't that
there are no such companies now, but margins are certainly tighter.
And, importantly, even where there is cash being made, the making of
the standard may not improve that cash position--it could be that
after a lot of investment, the market is the same, and the money spent
on the standard was a waste. The same money invested into other
things might be very important to those same companies. So the real
cost in a tight market is "opportunity cost".
Incidentally, we had network technology in the 80's. What we did not
have was agreement on details. At the time (and probably still
today), there were subtle but important differences between the
different approaches taken such that vendors were not keen on being
pushed to take some else's approach.
I myself tried (I'm pretty sure it was around 1992, since it was
around the time I was laid off from Symbolics) to get vendors to agree
on some of these matters, which it seemed like a lot of people needed.
I ultimately concluded that the heavyweight mechanism of a formal
standard was way more than they were prepared to pay, and not because
they were being obstructionist. Just because the cost didn't make
sense. The truth was that the cost to vendors of disagreeing was not
much, and the cost to users was not much either. So it played well to
say "we need this", but in the end, no one needed it enough to really
do anything. The status quo was, and I suspect still is, pretty
tolerable.
Then there's the issue of what to standardize on. Standards should
begin with a near-consensus. They are not a proper forum for design,
they are a way to bless (or put finishing touches on) a de facto
standard. If you don't start with something very much like agreement
at the outset, you don't know which way things will go. And if you
don't know you'll like the outcome, why are you wanting to cast that
outcome into stone? Better to get an informal agreement FIRST, and
THEN standardize--once you have a de facto standard, the fact of
standardization is just an administrative detail, and an unimportant
one at that.
In truth, a standard is really just not needed. (I'll be misquoted
later for that I'm sure--someone will say I said it would be bad to
have a standard, or that I was obstructing a standard. Neither is
true. I meant literally what I said, and the use of "need" is
central. When push comes to shove, the subjective question of what
is a "need" and what is a "want" gets sorted out objectively by
observing where money is allocated in a tight-money system... and since
money has not been allocated to this task, I conclude as a matter of
objective fact that "need" is not there.)
If a standard were what stood in the way of success, a standard would
have been done a long time ago. But the pretty obvious truth is that,
in most cases, just tossing together code with a few #+'s in it is
enormously faster, enormously cheaper, and not going to cause
community disruption for those who don't need the change. And if
there's a need for a common one of these, it can be packaged into a
library and shared.
And most applications that need more than the #+/#- approach are not
portable anyway. Commercial applications are largely ones that are
going to pick an implementation and then use every detail of it.
Often they'll rely on enough things that are particular to that
implementation that worrying about making the application portable
will be the last thing on their mind.
None of what I've written should be taken to say "you've got to do it
my way". I'm not even saying I wouldn't like a networking standard;
in fact, I'd be very happy to see one. I'm just saying the reason
it's not there cannot be usefully or productively summed up by blaming
the stability of the standard or by inventing a rigidity of the
community.
············@gmail.com wrote:
...
> I'm wondering why the hell this has not been put into the CL spec. CL
> offers abilities to handle a huge amount of filesystem types but just
> ignores networking. ...
Most languages don't consider networking a fundamental part of the
language. Niklaus Wirth didn't even consider standard I/O a fundamental
part of Pascal, but an extension to the language. [1]
Networking is system dependent, and just a library call away.
BTW, Niklaus E. Wirth === Turing Award
--
[1] Pascal User Manual and Report, Appendix D
············@gmail.com writes:
> What would be the best way to do networking using both TCP *and* UDP
> protocols using CL ?
> I'm wondering why the hell this has not been put into the CL spec.
Because the Internet Protocol and the Transport Control Protocol were
defined in January 1980, and the User Datagram Protocol in August
1980, while works on Common Lisp started "in April 1981, after a
DARPA-sponsored meeting concerning the splintered Lisp community,
Symbolics, the SPICE project, the NIL project, and the S-1 Lisp
project joined together to define Common Lisp.".
Between 1981 and 1986, there was a number of network protocols, any
one of them could have become dominant, AppleTalk, DECnet, IPX/SPX, etc.
> CL
> offers abilities to handle a huge amount of filesystem types but just
> ignores networking. Looks like I'll have to use a CL implementation on
> Plan 9 to achieve this ;)
Indeed, there is absolutely no provision for networking in CL.
No more than in C, by the way.
That said, you may be lucky and be using an implementation that
provides some API to the networking services of your OS. Perhaps.
--
__Pascal Bourguignon__
ne> I'm wondering why the hell this has not been put into the CL spec.
because there is no good abstraction for different networking stuff.
best abstraction you can get is streams, and you actually get it.
ne> CL offers abilities to handle a huge amount of filesystem types but
because all filesystems have most parts common
Den Fri, 08 Feb 2008 20:18:22 +0200 skrev Alex Mizrahi:
> ne> CL offers abilities to handle a huge amount of filesystem types
> but
>
> because all filesystems have most parts common
When was the last time you looked at CL's filesystem handling? Compared
to pretty much any other language it's horrifically complex, and that's
precisely the result of there being almost nothing in common back when CL
was in the making.
Cheers,
Maciej
ne>>> CL offers abilities to handle a huge amount of filesystem types
??>> but
??>>
??>> because all filesystems have most parts common
MK> When was the last time you looked at CL's filesystem handling?
i'm using it all the time
MK> Compared to pretty much any other language it's horrifically complex,
i don't see any complexities with it. most languages have rudimentary APIs,
and people have to build their own libraries to work with pathnames then.
once i've noticed that C++ application i was working with had something like
3 different classes acting like pathname designators -- different
programmers in different ways made their own implementations (informally
specified, bug ridden..) of merge-pathnames and logical pathnames.
MK> and that's precisely the result of there being almost nothing in common
MK> back when CL was in the making.
you know, \ or / as a path separator and stuff like that is not *that* big
difference. some file systems had extended information like versions, and
what?
the common thing in file systems is that there are file names, and you can
access files via that names. and also in most cases file names are
hierarchical.
you can deal with pathanames as with some opaque tokens, and this is already
enough for 99% of functionality.
On 8 fév, 19:18, "Alex Mizrahi" <········@users.sourceforge.net>
wrote:
> ne> I'm wondering why the hell this has not been put into the CL spec.
>
> because there is no good abstraction for different networking stuff.
> best abstraction you can get is streams, and you actually get it.
>
Streams may be the best abstraction for TCP-like protocols (maybe this
is why you pass a 'STREAM' argument in the POSIX 'socket' function ;)
but I don't think its fits the datagram model.
> ne> CL offers abilities to handle a huge amount of filesystem types but
>
> because all filesystems have most parts common
And you think network protocols don't ?
ne>>> I'm wondering why the hell this has not been put into the CL spec.
??>> because there is no good abstraction for different networking stuff.
??>> best abstraction you can get is streams, and you actually get it.
ne> Streams may be the best abstraction for TCP-like protocols (maybe this
ne> is why you pass a 'STREAM' argument in the POSIX 'socket' function ;)
ne> but I don't think its fits the datagram model.
ne>>> CL offers abilities to handle a huge amount of filesystem types but
??>>
??>> because all filesystems have most parts common
ne> And you think network protocols don't ?
yes, i think so. as you see, abstraction that is good for TCP (streams) is
not good even for it's socket brother UDP.
what's about other network protocols and architectures?
also note that different people want to work with networking at different
abstraction level: some people are satisfied with streams, other people
would like to modify IP headers on packet level.
and all this stuff is highly system-dependent.. event demultiplexing, for
example.
should they make all networking doomed to be synchonous, or require
implementors to implement ``select'' on systems which do not support it?
IMHO it's better for us that networking is not in the standard and
implementations can have API that is adequate for given system.
would you be happy if standard included some weird baroque networking ideas?
"Alex Mizrahi" <········@users.sourceforge.net> writes:
> yes, i think so. as you see, abstraction that is good for TCP
> (streams) is not good even for it's socket brother UDP. what's
> about other network protocols and architectures?
Streams are actually a lousy abstraction for reading from a network
socket. The abstraction does work cleanly for writing, however.
When reading, you really want to say to the system "please start
reading data from this socket until a fully formed data blob has been
read, and then call this function with the blob."
Of course the meaning of "fully formed data blob" is dependent on the
protocol you're reading. For HTTP it's \r\n\r\n or something like
that. You might have a fixed length message coming at you. Or one
can imagine a header/body protocol with a fixed length header telling
you how many bytes are in the body. Etc.
You can do all of this with a stream interface, except then you're
forced to use threads, because for many protocols you don't know how
much data you need to read, so you have to block[1]. I find debugging
threaded programs much more difficult than non-threaded programs, so I
think this is a serious disadvantage for using streams to read from
sockets.
Also, one thread per client has kind of sucky resource requirements,
compared to just using poll / kqueue / epoll / IOCP or what have you.
I mention all of the above, because I'm just finishing up an interface
to the poll(2) system call for sbcl. Let me know if you want to see
it...
> also note that different people want to work with networking at
> different abstraction level: some people are satisfied with streams,
> other people would like to modify IP headers on packet level.
Or maybe threads are just too hard, let's go shopping...
> and all this stuff is highly system-dependent.. event
> demultiplexing, for example. should they make all networking doomed
> to be synchonous, or require implementors to implement ``select'' on
> systems which do not support it?
Keep in mind that this type of IO is only available on MacOS, BSD,
Linux, Solaris, AIX, and Windows. Oh wait, I'm probably forgetting a
few.
-russ
[1] I'm aware of CMUCL's recursive event loop that hides the blocking.
It's a nice hack, but IMHO it's a heroic attempt to pound a square peg
into a round hole.
RM> When reading, you really want to say to the system "please start
RM> reading data from this socket until a fully formed data blob has been
RM> read, and then call this function with the blob."
it depends on what you're doing. what if you're writting stuff to file, or
you implement a proxy that sends data over network?
you don't need blobbing in this case.
??>> and all this stuff is highly system-dependent.. event
??>> demultiplexing, for example. should they make all networking doomed
??>> to be synchonous, or require implementors to implement ``select'' on
??>> systems which do not support it?
RM> Keep in mind that this type of IO is only available on MacOS, BSD,
RM> Linux, Solaris, AIX, and Windows. Oh wait, I'm probably forgetting a
RM> few.
all these operating systems inherit BSD socket interface and implement POSIX
to some extent (yes, Windows NT was certified as fully POSIX compliant, with
some compat packages).
it wasn't clear that POSIX would be dominant 20+ years ago, when CL was
standardtized.
"Alex Mizrahi" <········@users.sourceforge.net> writes:
> RM> When reading, you really want to say to the system "please
> RM> start reading data from this socket until a fully formed data
> RM> blob has been read, and then call this function with the blob."
>
> it depends on what you're doing. what if you're writting stuff to
> file, or you implement a proxy that sends data over network? you
> don't need blobbing in this case.
Of course it matters what you're doing. My comments were specifically
in regards to network programming. Also I stated that the stream
abstraction works fine for writing; it's easy to queue up the data
written to the stream and send it at the right time.
> ??>> and all this stuff is highly system-dependent.. event
> ??>> demultiplexing, for example. should they make all networking doomed
> ??>> to be synchonous, or require implementors to implement ``select'' on
> ??>> systems which do not support it?
>
> RM> Keep in mind that this type of IO is only available on MacOS, BSD,
> RM> Linux, Solaris, AIX, and Windows. Oh wait, I'm probably forgetting a
> RM> few.
>
> all these operating systems inherit BSD socket interface and implement POSIX
> to some extent (yes, Windows NT was certified as fully POSIX compliant, with
> some compat packages).
>
> it wasn't clear that POSIX would be dominant 20+ years ago, when CL was
> standardtized.
I agree that these facilities don't belong in the CL standard. My
point is that just about every operating system around today provides
scalable event-driven mechanisms for reading from many network
connections, but these facilities are hiding behind the wrong
abstractions in today's Lisp implementations.
-russ
On Mon, 11 Feb 2008 13:35:44 +0200, Alex Mizrahi wrote:
> RM> Keep in mind that this type of IO is only available on MacOS, BSD,
> RM> Linux, Solaris, AIX, and Windows. Oh wait, I'm probably forgetting
> a RM> few.
>
> all these operating systems inherit BSD socket interface and implement
> POSIX to some extent (yes, Windows NT was certified as fully POSIX
> compliant, with some compat packages).
>
> it wasn't clear that POSIX would be dominant 20+ years ago, when CL was
> standardtized.
While that's true, those same POSIX networking standards aren't part of
the C or C++ standards, and C predates BSD sockets. BSD sockets aren't
even the only way that networking is done on Unix-like platforms.
Earlier AT&T Unix had STREAMS, and Minix-3 manages it all through magic
file-system device nodes. POSIX (and subsequently SUS) standardised the
BSD interfaces, and so C programs for those platforms get to be able to
rely on their existence and behaviour. There might be command-line
incantations required, to specify include paths or libraries, but the
source code gets to be portable. Maybe (a) goal could be not so much a
standard for networking, but a standard way to import or support external
standards. That seems to be what Kent has been suggesting recently. My
guess is that FFI is as close as that gets, at the moment. That's almost
certainly a poor match for CL, so you're still back to the problem of
wanting a *good* standard for networking. I just wanted to make the
point that arguments to age might also be historically accurate, but
aren't persuasively strong.
Cheers,
--
Andrew
On Mon, 11 Feb 2008 13:35:44 +0200, "Alex Mizrahi"
<········@users.sourceforge.net> wrote:
> RM> When reading, you really want to say to the system "please start
> RM> reading data from this socket until a fully formed data blob has been
> RM> read, and then call this function with the blob."
>
>it depends on what you're doing. what if you're writting stuff to file, or
>you implement a proxy that sends data over network?
>you don't need blobbing in this case.
But Russell's point about reading is well taken - virtually all
application level protocols are message based regardless of the
underlying network transport.
I've long thought that message queues were a much better abstraction
than streams for generalized network programming.
George
--
for email reply remove "/" from address
"Alex Mizrahi" <········@users.sourceforge.net> writes:
> IMHO it's better for us that networking is not in the standard and
> implementations can have API that is adequate for given system.
> would you be happy if standard included some weird baroque
> networking ideas?
As with the "religion in the schools" question, the answer is probably
that it sounds good until you find it's not the particular wording one
meant when one was advocating it. :)
Which leads directly to my related question, asked again recently and
not really answered:
Why would it not be adequate to simply have many layered standards for
dealing with each of the things you do need to talk to?
To understand the issues in a one-size-fits-all standard, look at
pathnames. They try to be the bridge between all possible file system
syntaxes. And they don't do badly given how weird the entire space
is. But often it's easier to presume a smaller space of possibilities
and standardize on that.
The problem that perplexed the designers of CL was that if you make
rules saying "you can't do x" for anything where "x" is outside the
control of CL, you are basically not forcing the world to have a
different file system or network system or whatever. All you are
saying is "it's not allowed to make something called CL that programs
any system that contains x as a fixed constraint". That seemed silly,
so CL sought not to create rules of that sort.
Again, sticking to pathnames as the example, nothing keeps users of,
say, posix from writing a library that handles posix filenames better
than CL can handle "all possible filenames". The right fix there is
not to say, "CL should also embrace posix" but rather "implementations
of a standard for posix should be hostable within CL". And it should
be quite obvious that they aren't going to get there by being written
in portable code, since that pretty much leads to circular
reasoning... That's why there is #+/#-, at least for now, so you can
call out to the implementation-specific things you need to get a
foothold for things that are not truly portable but that allow you to
write a "sufficiently portable" library that spans the implementations
that share a common model of some outside-world phenomenon.
For users who want to run CL on some other file system entirely, a
layered standard they didn't have to load or care about would be fine.
Whereas a change to the core language could be fatal.
The politics of pluralism requires a certain mindset of tolerance, and
a practice of people deliberately making of room for things that they
might not personally want, just because someone else might. If
practioners of Common Lisp learn only those lessons of tolerance and
inclusion from the experience, and ultimately end up using none of the
code they write, I expect they'll have come away with something of
great value.
Shifting examples, much has been made of the issues of case
sensitivity in Lisp, and the Clash of Absolutes involved in fighting
over who gets to be right, but much less is made of the extraordinary
degree of caring thought that has gone into the design of *PRINT-CASE*
and READTABLE-CASE, which are the technological embodiments of great
sensitivity on the part of the "victors" toward the needs of the
"losers", in a conflict that matters a great deal on both sides and
that could have only one winner. Those who "won" could have just
stopped there, but the fact that they went back and tried to carry as
many others along as they could is really quite extraordinary, and
would be (if not for its nerdy and esoteric nature, making it too
opaque for study by an Outsider) a model for what can come of careful
thought by people with strong technical knowledge and a willigness to
care.
On Feb 8, 3:52 pm, ············@gmail.com wrote:
> What would be the best way to do networking using both TCP *and* UDP
> protocols using CL ?
IIRC, iolib[1] package supplies socket & networking stuff via FFI
calls on top of POSIX. OTOH, you can (IMHO, you should) ping usocket
developers to speed up UDP support efforts[2,3].
[1] http://common-lisp.net/project/iolib/
[2] http://common-lisp.net/pipermail/usocket-devel/2008-February/000105.html
[3] http://common-lisp.net/pipermail/usocket-devel/2008-February/000110.html