Hi,
I'm looking for an option to trampoline into a long-running Lisp process
with a small CGI handler. Ideally, the trampoline would simply speak the
mod_lisp protocol, which would allow servers that already know mod_lisp
(ie. almost all of them) to be run unmodified. The usage scenario is
having Lisp serve content on hosts where I have neither privileges to
install Apache modules nor FastCGI available, which describes very many of
the cheaper shared hosting options.
One fairly obvious solution would be to have a CGI handler version of
mod_lisp.c, but I haven't found anything like that, and looking at the
code, it scares me with Apache API which I have no clue about. So, is
anyone aware of existence / capable of producing a standalone mod_lisp?
Also, google and #lisp tell me Rob Warnock has previously mentioned using
a "tiny C trampoline that speaks a mod_lisp-like protocol", how much
mod_lisp-like is it (and if sufficiently so, would you be willing to
share)?
Cheers,
Maciej
Maciej Katafiasz <········@gmail.com> wrote:
+---------------
| I'm looking for an option to trampoline into a long-running Lisp process
| with a small CGI handler. Ideally, the trampoline would simply speak the
| mod_lisp protocol, which would allow servers that already know mod_lisp
| (ie. almost all of them) to be run unmodified. The usage scenario is
| having Lisp serve content on hosts where I have neither privileges to
| install Apache modules nor FastCGI available, which describes very many of
| the cheaper shared hosting options.
|
| One fairly obvious solution would be to have a CGI handler version of
| mod_lisp.c, but I haven't found anything like that, and looking at the
| code, it scares me with Apache API which I have no clue about. So, is
| anyone aware of existence / capable of producing a standalone mod_lisp?
| Also, google and #lisp tell me Rob Warnock has previously mentioned using
| a "tiny C trampoline that speaks a mod_lisp-like protocol", how much
| mod_lisp-like is it (and if sufficiently so, would you be willing to
| share)?
+---------------
Well, *now* you've gone & done it!! ;-} You've *finally* forced me
to rummage around in my "nonexistent spare time" junkbox and find one
of them "round tuit" thingamabobs and add an MIT-style copyright notice
to my "cgi_sock.c" program, which is now available to all & sundry here:
http://rpw3.org/hacks/lisp/cgi_sock.c
As it says there, provided as-is, no warranty, no support. [And no
free consulting, either, so please *DON'T* send me email about it.
It'll just make me grumpy, and won't get you any help.] Though if
you have questions/comments/suggestions and you post them *here*,
somebody [maybe even me?] might respond. Criticisms of the "Why did
you do it *that* way?!?" kind will probably all be answered with this
bolierplate: "When I wrote this I was in a hurry, and it worked
well enough, so I haven't messed with it since." [It's still in use,
unchanged, on several lightly-loaded production web servers out there.]
Since you're not using my idiosyncratic CMUCL-based application server
[which is the *next* thing I need to clean up enough to release!],
you will almost certainly want to change the hardcoded location of
the application server's listening socket, which you do by defining
SOCK_PATH on the compile command line, e.g.:
cc -O -DSOCK_PATH=/var/run/my_app.sock cgi_sock.c -o cgi_sock
Install with:
cp -p cgi_sock /path/to/your/cgi-bin/
or:
cp -p cgi_sock /usr/local/www/htdocs/...some_path.../sock.cgi
Consult the source for the Apache "AddHandler" directives needed
for those two cases.
If you want to maintain protocol compatibility with with "mod_lisp",
you probably don't want to change any of the other "#define"s in the
source [but if not, observe the effect of changing SOCK_PROTOCOL to
PROTOCOL_GETENV instead of PROTOCOL_MODLISP].
And as a bonus... [sorry, no knives, no orange peelers, just more
software]... there's also a tiny/sample/debugging stub app server
that serves up a plaintext HTTP response showing you what all the
"mod_lisp" a.k.a. CGI variables were:
http://rpw3.org/hacks/lisp/cgi_sockd.c
Caveat: I haven't used this since the first week I wrote "cgi_sock.c",
so the socket paths it uses will *NOT* match the current "cgi_sock.c"
paths at all. [You have been warned.]
Enjoy!
-Rob
-----
Rob Warnock <····@rpw3.org>
627 26th Avenue <URL:http://rpw3.org/>
San Mateo, CA 94403 (650)572-2607
Den Thu, 08 Nov 2007 05:31:14 -0600 skrev Rob Warnock:
> Well, *now* you've gone & done it!! ;-} You've *finally* forced me to
> rummage around in my "nonexistent spare time" junkbox and find one of
> them "round tuit" thingamabobs and add an MIT-style copyright notice to
> my "cgi_sock.c" program, which is now available to all & sundry here:
>
> http://rpw3.org/hacks/lisp/cgi_sock.c
Ah, jolly good. Now you fell for my trick, I'm gonna try it and promptly
disregard your disclaimers by complaining loudly it doesn't work.
> If you want to maintain protocol compatibility with with "mod_lisp", you
> probably don't want to change any of the other "#define"s in the source
> [but if not, observe the effect of changing SOCK_PROTOCOL to
> PROTOCOL_GETENV instead of PROTOCOL_MODLISP].
A bit of investigation shows that it's not a drop-in replacement for
actual mod_lisp. After writing a socket_connect() version for AF_INET, it
seems that Hunchentoot hates the values cgi_sock feeds it with, as they
are named quite differently than what mod_lisp provides. I was afraid of
that, and it was actually the part I wanted to avoid dealing with...
Oh well, I'll have to find some intelligent way to deal with it, but
probably not today.
Cheers,
Maciej
Maciej Katafiasz <········@gmail.com> wrote:
+---------------
| skrev Rob Warnock:
| > If you want to maintain protocol compatibility with with "mod_lisp", you
| > probably don't want to change any of the other "#define"s in the source
| > [but if not, observe the effect of changing SOCK_PROTOCOL to
| > PROTOCOL_GETENV instead of PROTOCOL_MODLISP].
|
| A bit of investigation shows that it's not a drop-in replacement for
| actual mod_lisp. After writing a socket_connect() version for AF_INET, it
| seems that Hunchentoot hates the values cgi_sock feeds it with, as they
| are named quite differently than what mod_lisp provides. I was afraid of
| that, and it was actually the part I wanted to avoid dealing with...
+---------------
Uh... Did you read the "Known bugs/errata/limitations:" section?
Where it says this?
- Current "mod_lisp" protocol explicitly downcases the names of all
environment variables before sending them; this program does not.
(Hint, hint...)
-Rob
-----
Rob Warnock <····@rpw3.org>
627 26th Avenue <URL:http://rpw3.org/>
San Mateo, CA 94403 (650)572-2607
Den Fri, 09 Nov 2007 23:26:23 -0600 skrev Rob Warnock:
> Uh... Did you read the "Known bugs/errata/limitations:" section? Where
> it says this?
>
> - Current "mod_lisp" protocol explicitly downcases the names of all
> environment variables before sending them; this program does not.
>
> (Hint, hint...)
Yes. I even wrote code to fix that. However, I think the fact that
mod_lisp sends, for example, a variable called "url", whereas for a CGI
request it's "request_uri", is even more of a problem. And Hunchentoot
refusing to work unfortunately confirms that.
Cheers,
Maciej
Maciej Katafiasz <········@gmail.com> wrote:
+---------------
| Rob Warnock:
| > Uh... Did you read the "Known bugs/errata/limitations:" section? Where
| > it says this?
| > - Current "mod_lisp" protocol explicitly downcases the names of all
| > environment variables before sending them; this program does not.
|
| Yes. I even wrote code to fix that. However, I think the fact that
| mod_lisp sends, for example, a variable called "url", whereas for a CGI
| request it's "request_uri", is even more of a problem.
+---------------
De gustibus non est disputandum. I found it more of a problem that
"mod_lisp" (seemingly gratuitously) changed the names of some of
the standard CGI variables, which is why "cgi_sock.c" passes them
through unchanged. You may, of course, choose to hack up the source
of "cgi_sock.c" to make the same changes that "mod_lisp" does.
+---------------
| And Hunchentoot refusing to work unfortunately confirms that.
+---------------
"Use the source, Luke..."
-Rob
p.s. As long as we're listing problematic(?) features of "cgi_sock.c",
note that it doesn't support "keep-alive" connections, though ISTR
that later versions of "mod_lisp" do. My app server doesn't need/want
"keep-alive", so I didn't implement it.
-----
Rob Warnock <····@rpw3.org>
627 26th Avenue <URL:http://rpw3.org/>
San Mateo, CA 94403 (650)572-2607
Den Sat, 10 Nov 2007 05:13:48 -0600 skrev Rob Warnock:
> +---------------
> | And Hunchentoot refusing to work unfortunately confirms that.
> +---------------
>
> "Use the source, Luke..."
Oh, but I did. http://mathrick.org/blog/archives/2007/11/10/cgi-lisp/
Besides variable names translation (which I agree is silly and
gratitious), there was a genuine bug in mod_lisp response handling. You
were copying the output verbatim, whereas in order to comply with CGI
response format, headers ought to be changed from Header\nvalue to
Header: value.
> p.s. As long as we're listing problematic(?) features of "cgi_sock.c",
> note that it doesn't support "keep-alive" connections, though ISTR that
> later versions of "mod_lisp" do. My app server doesn't need/want
> "keep-alive", so I didn't implement it.
Can you even implement keep-alive in a CGI handler?
Cheers,
Maciej
Den Sat, 10 Nov 2007 15:19:03 +0000 skrev Maciej Katafiasz:
> Oh, but I did. http://mathrick.org/blog/archives/2007/11/10/cgi-lisp/
Updated now not to require GLib. I blame Xach.
Cheers,
Maciej
Maciej Katafiasz <········@gmail.com> wrote:
+---------------
| ... http://mathrick.org/blog/archives/2007/11/10/cgi-lisp/
| Besides variable names translation (which I agree is silly and
| gratitious), there was a genuine bug in mod_lisp response handling. You
| were copying the output verbatim, whereas in order to comply with CGI
| response format, headers ought to be changed from Header\nvalue to
| Header: value.
+---------------
Ahh... I see the light. You were looking for an *EXACT* replacement
for "mod_lisp", whereas that was never a design requirement or even
a desire of mine for "cgi_sock.c". Rather, my app server code had
initially started out *as* a classic CGI app[1], and "cgi_sock.c" was
just a trivial way to transition it from the model of "run a new Lisp
process every time you get a web hit" to "run a single persistent
Lisp process at boot time and only run a *tiny* CGI program when
you get a web hit".
Given that, you can then understand why "cgi_sock.c" "copies the
output verbatim", because the Lisp side -- originally written *as*
as CGI program -- was already emitting the proper "Header: Value"
format. In fact, it was initially sendin* classical CGI environment
variable format ("NAME=value") request headers across the socket
as well, and doing the splitting on the Lisp side. [That is, using
the C code that's now under the "#if SOCK_PROTOCOL == PROTOCOL_GETENV"
tests.] Only later did I add the "PROTOCOL_MODLISP" flavor and switch
from splitting the CGI vars in the Lisp app to splitting them in
"cgi_sock", since that slightly lessens the amount of per-request
garbage generated on the Lisp side. But I dodn't mess with the
response direction, since that was already fine was it was.
I noticed that you're now using the "PROTOCOL_MODLISP" conditional to
also control the response format, which IMHO is probably a mistake
in the long run. What we should probably both do is keep the request
and response formats orthogonal, and switch to using conditional names
that make that clear, e.g., rename PROTOCOL_GETENV to PROTOCOL_CGI, and
split SOCK_PROTOCOL into SOCK_REQUEST_PROTOCOL and SOCK_RESPONSE_PROTOCOL.
That way, you can use:
#define SOCK_REQUEST_PROTOCOL PROTOCOL_MODLISP
#define SOCK_RESPONSE_PROTOCOL PROTOCOL_MODLISP
while I continue to use my preferred hybrid:
#define SOCK_REQUEST_PROTOCOL PROTOCOL_MODLISP
#define SOCK_RESPONSE_PROTOCOL PROTOCOL_CGI
and anyone who wants to split the environment strings on the Lisp side
could use:
#define SOCK_REQUEST_PROTOCOL PROTOCOL_CGI
#define SOCK_RESPONSE_PROTOCOL PROTOCOL_CGI
+---------------
| > p.s. As long as we're listing problematic(?) features of "cgi_sock.c",
| > note that it doesn't support "keep-alive" connections...
|
| Can you even implement keep-alive in a CGI handler?
+---------------
You're right, you can't, which is partly why I didn't bother to try . ;-}
One more thing... In your blog you wrote:
I dont think it will be very useful without handling POST anyway.
What do you mean by "without handling POST"? *All* of my web apps use
POST very heavily, and "cgi_sock" handles it just fine. [For example,
see <http://rpw3.org/hacks/lisp/appsrv-demo.lhp> which uses POST.]
Did you actually try it? From looking at the "mod_lisp.c" code, I don't
see any obvious reason that it *shouldn't* work. Marc did roughly the
same thing that I did, namely, right after sending the "end\n", check
to see if there's a "request entity" ["mod_lisp.c" calls Apache's
"ap_should_client_block()" whereas "cgi_lisp.c" looks to see if the
environment variable "REQUEST_METHOD" is "POST"], and if so, copy from
the web client to the socket. It really, really *ought* to work the same.
-Rob
[1] Note that by using a Lisp image with all of the CGI stuff
pre-compiled/loaded, on a modern O.S. with decent file caching
(such as FreeBSD or Linux) the execution of a whole new Lisp
image for each CGI call costs well under 50 ms (>20 req/s,
for trivial requests), which is fast enough for a really
low-volume site. Still, with almost exactly the same Lisp
code, the average trivial request through "cgi_sock" takes
only ~6 ms (~165 req/s) [measureed woth "ab -n100"]. Plus,
it's actually a lot easier to debug a persistent daemon than
a "real" CGI app.
-----
Rob Warnock <····@rpw3.org>
627 26th Avenue <URL:http://rpw3.org/>
San Mateo, CA 94403 (650)572-2607
Den Sat, 10 Nov 2007 21:52:17 -0600 skrev Rob Warnock:
> Maciej Katafiasz <········@gmail.com> wrote:
> +---------------
> | ... http://mathrick.org/blog/archives/2007/11/10/cgi-lisp/
> | Besides variable names translation (which I agree is silly and
> | gratitious), there was a genuine bug in mod_lisp response handling. You
> | were copying the output verbatim, whereas in order to comply with CGI
> | response format, headers ought to be changed from Header\nvalue to
> | Header: value.
> +---------------
>
> Ahh... I see the light. You were looking for an *EXACT* replacement for
> "mod_lisp", whereas that was never a design requirement or even a desire
> of mine for "cgi_sock.c".
Yes, I thought it was clear from my initial post, apologies if not.
> I noticed that you're now using the "PROTOCOL_MODLISP" conditional to
> also control the response format, which IMHO is probably a mistake in
> the long run. What we should probably both do is keep the request and
> response formats orthogonal, and switch to using conditional names that
> make that clear, e.g., rename PROTOCOL_GETENV to PROTOCOL_CGI, and split
> SOCK_PROTOCOL into SOCK_REQUEST_PROTOCOL and SOCK_RESPONSE_PROTOCOL.
And there I go, declaring 0.5 mostly final... but I didn't have the
net over the weekend, so I couldn't read your post. Will do that and
release 0.6 tomorrow.
> One more thing... In your blog you wrote:
>
> I dont think it will be very useful without handling POST anyway.
>
> What do you mean by "without handling POST"? *All* of my web apps use
> POST very heavily, and "cgi_sock" handles it just fine. [For example,
> see <http://rpw3.org/hacks/lisp/appsrv-demo.lhp> which uses POST.]
>
> Did you actually try it? From looking at the "mod_lisp.c" code, I don't
> see any obvious reason that it *shouldn't* work. Marc did roughly the
> same thing that I did, namely, right after sending the "end\n", check to
> see if there's a "request entity" ["mod_lisp.c" calls Apache's
> "ap_should_client_block()" whereas "cgi_lisp.c" looks to see if the
> environment variable "REQUEST_METHOD" is "POST"], and if so, copy from
> the web client to the socket. It really, really *ought* to work the
> same.
Yep, it did. Just that at the time I tried some POST action, the
variables actually useful for it weren't translated to their mod_lisp
names, so it didn't work.
> [1] Note that by using a Lisp image with all of the CGI stuff
> pre-compiled/loaded, on a modern O.S. with decent file caching (such
> as FreeBSD or Linux) the execution of a whole new Lisp image for
> each CGI call costs well under 50 ms (>20 req/s, for trivial
> requests), which is fast enough for a really low-volume site. Still,
> with almost exactly the same Lisp code, the average trivial request
> through "cgi_sock" takes only ~6 ms (~165 req/s) [measureed woth "ab
> -n100"]. Plus, it's actually a lot easier to debug a persistent
> daemon than a "real" CGI app.
Yes, from my informal testing, it seems to have more or less the same
performance as mod_proxy. Which is a good result I think :). And
consistent with your findings. And I do care about speed, as I might
end up handling on the order of 100 req/s per image. So now it's time
to profile Weblocks, which brings these nice numbers down to 20 req/s
again...
Cheers,
Maciej
PS. Your fancy quotes surely are fun when responding :)
Maciej Katafiasz <········@gmail.com> wrote:
+---------------
| And I do care about speed, as I might end up handling on the
| order of 100 req/s per image.
+---------------
In that case, you should really consider using the real "mod_lisp",
not "cgi_sock". The latter was never intended for loads in the
100 req/s per image range -- more like the 1-10 req/s range! ;-}
[Though I have seen it run at >100 req/s, for trivial requests.]
But "mod_lisp" has been documented[1] as running well in excess
of 1000 req/s!! So if you really care about speed...
-Rob
[1] http://groups.google.com/group/comp.lang.lisp/msg/154184cd54885460
Marc Battyani, "Re: Thoughts on mod_lisp", 2004-10-08
-----
Rob Warnock <····@rpw3.org>
627 26th Avenue <URL:http://rpw3.org/>
San Mateo, CA 94403 (650)572-2607
MK> know mod_lisp (ie. almost all of them) to be run unmodified. The usage
MK> scenario is having Lisp serve content on hosts where I have neither
MK> privileges to install Apache modules nor FastCGI available, which
MK> describes very many of the cheaper shared hosting options.
since when "cheaper shared hosting options" would allow you to install lisp
executable and run it persistently?
and even if so, do they really allow CGI in C? i'd better invest into PHP
thing