From: Eric Merritt
Subject: Does a non-binding READ function exist?
Date: 
Message-ID: <19130ae.0304190918.1ae21b4f@posting.google.com>
Hello everyone,

 I am a long time lurker on this list. So far every piece of
information I have needed I have found in the archives with a quick
google search. However at this point I have managed to stump myself.

 I am writing a pretty simple server in cmucl on redhat 8.0 making use
of cmucl's SERVE-EVENT functionality. After a little reading on the
safety of the read function in the archives (there are two or three
good threads on this) I felt that using sexprs and lisp's built in
READ function would be a nice simple way to go for this particular
project. However, the READ function will block if no data is available
(obviously a bad thing for a single threaded server). I was wondaring
if there is some type of READ function out there that will read and
parse as much data as is available and then suspend and return (with
the ability to be restarted when more data comes online, of course). I
am doubting that this exists at the moment.

If this non-blocking read doesn't would one of you be willing to offer
suggestions about how to implement it efficiently? I am still pretty
new to lisp so my experience is lacking, though I do have plenty of
experiance in other functional and imperative languages.


Off the top of my head I can think of a few ways to do this, probably
the easiest is to do a non-parsing scan of the list as it comes in
till it finds what looks like full sexpr and then pass the resulting
string to the read function. The only problem is, that I hate to look
at each element in the list, once for the initial scan and once more
for the READ. This approach seams woefully inefficient.

In any case, I hope I have explained myself well. 

Thanks,
Eric

From: Nils Goesche
Subject: Re: Does a non-binding READ function exist?
Date: 
Message-ID: <87znmmqtr0.fsf@darkstar.cartan>
·········@yahoo.com (Eric Merritt) writes:

> However, the READ function will block if no data is available
> (obviously a bad thing for a single threaded server). I was
> wondaring if there is some type of READ function out there that
> will read and parse as much data as is available and then
> suspend and return (with the ability to be restarted when more
> data comes online, of course). I am doubting that this exists
> at the moment.

Why does it have to be a single-threaded server?  And what would
you do when READ ``suspends�� having read half a sexp, anyway?

Regards,
-- 
Nils G�sche
Ask not for whom the <CONTROL-G> tolls.

PGP key ID #xD26EF2A0
From: Bulent Murtezaoglu
Subject: Re: Does a non-binding READ function exist?
Date: 
Message-ID: <87ptni9y5p.fsf@acm.org>
>>>>> "NG" == Nils Goesche <···@cartan.de> writes:
[...]
    NG> Why does it have to be a single-threaded server?  

Because some people like writing state machines (sometimes for performance 
reasons, sometimes because they are too old and grizzly to consider spinning 
up new threads w/o good reason).  I thought you were one of us?  
 
    NG> And what
    NG> would you do when READ ``suspends�� having read half a sexp,
    NG> anyway?

Maybe the OP was taling about something like read(2) from Unix, rather 
than the CL read.

cheers,

BM
From: Nils Goesche
Subject: Re: Does a non-binding READ function exist?
Date: 
Message-ID: <87n0imqs8i.fsf@darkstar.cartan>
Bulent Murtezaoglu <··@acm.org> writes:

> >>>>> "NG" == Nils Goesche <···@cartan.de> writes:
> [...]
>     NG> Why does it have to be a single-threaded server?  
> 
> Because some people like writing state machines (sometimes for
> performance reasons, sometimes because they are too old and
> grizzly to consider spinning up new threads w/o good reason).
> I thought you were one of us?

I am, but mainly when writing C code :-) In Lisp, I tend to use
threads a lot these days...  In C you have to write everything
yourself, anyway, so it doesn't matter there, but if I have a
choice between rewriting READ or simply using threads, I'd go for
the thread solution.

Regards,
-- 
Nils G�sche
Ask not for whom the <CONTROL-G> tolls.

PGP key ID #xD26EF2A0
From: Eric Merritt
Subject: Re: Does a non-binding READ function exist?
Date: 
Message-ID: <25637N357@web2news.com>
> Why does it have to be a single-threaded server?  And what would
> you do when READ ``suspends�� having read half a sexp, anyway?

 Multi threaded servers tend to die under heavy load (not that this will
every be under that type of load of course). Not only that but the
context switching is pure overhead and sucks up cpu cycles needlessly.
Also lisp makes it pretty easy to hide the state switching. In fact in
my current implementation there isn't really state visible in the app.
Its all done using returned functions and some nice lexical context
stuff, god I love lisp.


Hmm, I just finished reading the paragraph above and I seem to be
optimizing. Its a bit early for that. perhaps I should do something with
threads and a thread pool. 


In any case, as to what read could do with half a sexp. It could do a
bunch of things I guess. The easiest would be to return it to the user
with an indicator that its not complete would probably be the easiest
and most strait forward. Then the calling function would just pass it
back in to 'resume'.

Just some thoughts in any case.
-- 
Direct access to this group with http://web2news.com
http://web2news.com/?comp.lang.lisp
From: Kenny Tilton
Subject: Re: Does a non-binding READ function exist?
Date: 
Message-ID: <3EA226CC.405@nyc.rr.com>
Eric Merritt wrote:
>>Why does it have to be a single-threaded server?  And what would
>>you do when READ ``suspends�� having read half a sexp, anyway?
> 
> 
>  Multi threaded servers tend to die under heavy load ...

So solve that, the Real Problem(tm). Not with single-threading -- an 
indiscriminate, too large, too "dumb" fix that simply creates other 
problems (no non-blocking read, or having to parse the input yourself) 
-- but by limiting the number of threads spawned.

-- 

  kenny tilton
  clinisys, inc
  http://www.tilton-technology.com/
  ---------------------------------------------------------------
"Everything is a cell." -- Alan Kay
From: Eric Merritt
Subject: Re: Does a non-binding READ function exist?
Date: 
Message-ID: <25727N154@web2news.com>
> Eric Merritt wrote:
>>>Why does it have to be a single-threaded server?  And what would
>>>you do when READ ``suspends�� having read half a sexp, anyway?
>>
>>  Multi threaded servers tend to die under heavy load ...
>
> So solve that, the Real Problem(tm). Not with single-threading -- an
> indiscriminate, too large, too "dumb" fix that simply creates other
> problems (no non-blocking read, or having to parse the
> input yourself)


True, the other option is thread pooling and I have started to look into
this. I personally think thread pooling offers as many problems as
single threading and a few disadvantages but it is viable alternative. I
have started looking into the multi_proc.lisp in the cmucl source and it
doesn't look like it would be to hard to create. I will probably still
look into the single threading piece for a bit though, if nothing more
to satisfy myself. 

Thanks,
Eric
-- 
Direct access to this group with http://web2news.com
http://web2news.com/?comp.lang.lisp
From: Thomas F. Burdick
Subject: Re: Does a non-binding READ function exist?
Date: 
Message-ID: <xcvadeknz8e.fsf@apocalypse.OCF.Berkeley.EDU>
"Eric Merritt" <······················@web2news.net> writes:

> > Eric Merritt wrote:
> >>>Why does it have to be a single-threaded server?  And what would
> >>>you do when READ ``suspends�� having read half a sexp, anyway?
> >>
> >>  Multi threaded servers tend to die under heavy load ...
> >
> > So solve that, the Real Problem(tm). Not with single-threading -- an
> > indiscriminate, too large, too "dumb" fix that simply creates other
> > problems (no non-blocking read, or having to parse the
> > input yourself)
> 
> 
> True, the other option is thread pooling and I have started to look into
> this. I personally think thread pooling offers as many problems as
> single threading and a few disadvantages but it is viable alternative. I
> have started looking into the multi_proc.lisp in the cmucl source and it
> doesn't look like it would be to hard to create. I will probably still
> look into the single threading piece for a bit though, if nothing more
> to satisfy myself. 

Well, if you're CMUCL, you might as well use SERVE-EVENT.  Instead of
a dumb fix, it's a nice event-server architecture.  Now, as for using
READ -- I think you'll need to call READ-FROM-STRING, and curry the
available input into a buffer on which to call this.  But that's not
exactly a dumb fix.

If you're using a potentially SMP-using threaded Lisp (I'm looking
with anticipation in the direction of SBCL), then thread pools have an
advantage.  The best of both worlds is to won the server, spawn as
many threads[*] as it has CPUs, and use event-serving architecture within
each thread.

[*] Or processes

-- 
           /|_     .-----------------------.                        
         ,'  .\  / | No to Imperialist war |                        
     ,--'    _,'   | Wage class war!       |                        
    /       /      `-----------------------'                        
   (   -.  |                               
   |     ) |                               
  (`-.  '--.)                              
   `. )----'                               
From: Eric Merritt
Subject: Re: Does a non-binding READ function exist?
Date: 
Message-ID: <25802N871@web2news.com>
> Well, if you're CMUCL, you might as well use SERVE-EVENT. Instead of
> a dumb fix, it's a nice event-server architecture. Now, as for using
> READ -- I think you'll need to call READ-FROM-STRING, and curry the
> available input into a buffer on which to call this. But that's not
> exactly a dumb fix.
>
> If you're using a potentially SMP-using threaded Lisp (I'm looking
> with anticipation in the direction of SBCL), then thread pools have an
> advantage. The best of both worlds is to won the server, spawn as
> many threads[*] as it has CPUs, and use event-serving architecture 
> within each thread.

Well, I am using CMUCL and SERVE-EVENT ;) 

 What I ended up doing is sending two pieces of information with a
message instead of one. The first piece is a 4 byte integer that gives
me the length of the following sexpr and the second in the sexpr itself.
Since I now know the length of both pieces of information I can simply
read them into the buffer using read-char-no-hold and then use
READ-FROM-STRING on the buffer after reading is complete. There are
other ways to do it, some perhaps more efficient like non-binding
streams etc. but this  seemed to be among the easiest and most strait
forward. I still hate touching every char as it comes in but I will have
to live with that I guess.
-- 
Direct access to this group with http://web2news.com
http://web2news.com/?comp.lang.lisp
From: Timothy Moore
Subject: Re: Does a non-binding READ function exist?
Date: 
Message-ID: <b7tdr7$4k1$0@216.39.145.192>
"Eric Merritt" <······················@web2news.net> writes:

> > Why does it have to be a single-threaded server?  And what would
> > you do when READ ``suspends�� having read half a sexp, anyway?
...
> In any case, as to what read could do with half a sexp. It could do a
> bunch of things I guess. The easiest would be to return it to the user
> with an indicator that its not complete would probably be the easiest
> and most strait forward. Then the calling function would just pass it
> back in to 'resume'.
> 
> Just some thoughts in any case.

For another view, see Kent Pitman's article "Ambitious Evaluation",
http://www.nhplace.com/kent/PS/Ambitious.html.  This describes input
editing as implemented on Lisp machines and in CLIM.  This isn't quite
the same situation as what the OP called for, but it does give some
insight on what to do while reading if the user backs up and edits
some earlier input.

Tim