I've found lots of (e.g. open-source) lisp packages that:
1. help you write CGI scripts in lisp
2. let you run a lisp web server
3. help with generating HTML
But I haven't found any that provide the relatively trivial functionality
of acting as a web _browser_, e.g. here's a URL, go and get the content.
Does anyone know of any such package? Something like a lisp version of cURL:
http://curl.haxx.se/
Thanks,
-- Don
_______________________________________________________________________________
Don Geddis http://don.geddis.org/ ···@geddis.org
I guess one of my pet peeves is that sometimes I have no idea what people are
talking about.
-- Deep Thoughts, by Jack Handey [1999]
Don Geddis <···@geddis.org> wrote in message news:<··············@sidious.geddis.org>...
> I've found lots of (e.g. open-source) lisp packages that:
> 1. help you write CGI scripts in lisp
> 2. let you run a lisp web server
> 3. help with generating HTML
>
> But I haven't found any that provide the relatively trivial functionality
> of acting as a web _browser_, e.g. here's a URL, go and get the content.
>
> Does anyone know of any such package? Something like a lisp version of cURL:
> http://curl.haxx.se/
Some very basic client functions are in [portable] allegroserve.
Enough to fetch URLs and test your webserver.
Cheers, Gabor
Don Geddis <···@geddis.org> writes:
> I've found lots of (e.g. open-source) lisp packages that:
> 1. help you write CGI scripts in lisp
> 2. let you run a lisp web server
> 3. help with generating HTML
>
> But I haven't found any that provide the relatively trivial functionality
> of acting as a web _browser_, e.g. here's a URL, go and get the content.
>
> Does anyone know of any such package? Something like a lisp version of cURL:
> http://curl.haxx.se/
For all the protocols that curl suports not that I know of, but
there is a http client package called client.cl in alegroserv/
portable alegroserv on sourceforge.
Good luck
marc
>
> Thanks,
>
> -- Don
> _______________________________________________________________________________
> Don Geddis http://don.geddis.org/ ···@geddis.org
> I guess one of my pet peeves is that sometimes I have no idea what people are
> talking about.
> -- Deep Thoughts, by Jack Handey [1999]
Don Geddis <···@geddis.org> writes:
> I've found lots of (e.g. open-source) lisp packages that:
> 1. help you write CGI scripts in lisp
> 2. let you run a lisp web server
> 3. help with generating HTML
>
> But I haven't found any that provide the relatively trivial functionality
> of acting as a web _browser_, e.g. here's a URL, go and get the content.
I don't think that even just downloading the content is a terribly
trivial task; negotiating with servers, some of which will have
creative interpretations of what constitutes HTTP, will probably
necessitate a fair amount of experimentation. Nevertheless...
> Does anyone know of any such package? Something like a lisp version of cURL:
> http://curl.haxx.se/
... this is perhaps overkill, but have a look at
<http://www.stud.uni-karlsruhe.de/~unk6/closure/>
(for a command-line download client, you'll be able to ignore all the
heuristics dealing with broken HTML :-)
Christophe
--
http://www-jcsu.jesus.cam.ac.uk/~csr21/ +44 1223 510 299/+44 7729 383 757
(set-pprint-dispatch 'number (lambda (s o) (declare (special b)) (format s b)))
(defvar b "~&Just another Lisp hacker~%") (pprint #36rJesusCollegeCambridge)
Don Geddis <···@geddis.org> writes:
> But I haven't found any that provide the relatively trivial
> functionality of acting as a web _browser_, e.g. here's a URL, go
> and get the content.
There's W4, part of CL-HTTP. We've got similar functionality in a web
robot that we've built.
I guess the real question is what you're trying to achieve. If all
you want to do is to fetch the content at a given URI, see url.lisp in
cllib, part of CLOCC, http://clocc.sourceforge.net/.
In practice, you often need much more smarts than that -- the ability
to detect loops, the ability to update indices based on 3xx HTTP
codes, state management, etc. I'm sure that the CLOCC folks would not
take offense if you were to extend their code to handle these kinds of
things, or to add new code for these purposes. :-)
--
Matt Curtin, CISSP, IAM, INTP. Keywords: Lisp, Unix, Internet, INFOSEC.
Founder, Interhack Corporation +1 614 545 HACK http://web.interhack.com/
Author of /Developing Trust: Online Privacy and Security/ (Apress, 2001)
"Don Geddis" <···@geddis.org> wrote in message
···················@sidious.geddis.org...
> I've found lots of (e.g. open-source) lisp packages that:
> 1. help you write CGI scripts in lisp
> 2. let you run a lisp web server
> 3. help with generating HTML
>
> But I haven't found any that provide the relatively trivial functionality
> of acting as a web _browser_, e.g. here's a URL, go and get the content.
Is it really that trivial? Sure I can get the content in about 10 lines of
code, but don't you want to see it in some reasonable form? Graphics,
client-side scripts, good formatting, all that stuff...There are surely some
devils in those details.
--
Coby Beck
(remove #\Space "coby 101 @ big pond . com")
On 21 Jul 2003 16:23:55 -0700, Don Geddis <···@geddis.org> wrote:
> But I haven't found any that provide the relatively trivial functionality
> of acting as a web _browser_, e.g. here's a URL, go and get the content.
CLOCC provides such a tool as part of cllib. See the file:
src/cllib/url.lisp
and in particular CLLIB:GET-URL.
Paolo
--
Paolo Amoroso <·······@mclink.it>
Don Geddis <···@geddis.org> writes:
> I've found lots of (e.g. open-source) lisp packages that:
> 1. help you write CGI scripts in lisp
> 2. let you run a lisp web server
> 3. help with generating HTML
>
> But I haven't found any that provide the relatively trivial functionality
> of acting as a web _browser_, e.g. here's a URL, go and get the content.
For the non-trivial functionality of acting as a real web browser,
there's Closure.
For just implementing a simple GET query ... it never occured to me to
release my code that does this. [looks for code] ... Looks like it's
not actually mine, but a client's. For the trivial stuff, it's, well,
trivial, to reimplement.
--
/|_ .-----------------------.
,' .\ / | No to Imperialist war |
,--' _,' | Wage class war! |
/ / `-----------------------'
( -. |
| ) |
(`-. '--.)
`. )----'
"Thomas F. Burdick" <···@famine.OCF.Berkeley.EDU> wrote in message
····················@famine.OCF.Berkeley.EDU...
> Don Geddis <···@geddis.org> writes:
>
> > I've found lots of (e.g. open-source) lisp packages that:
*snip*
> > But I haven't found any that provide the relatively trivial
functionality
> > of acting as a web _browser_, e.g. here's a URL, go and get the content.
>
> For just implementing a simple GET query ... it never occured to me to
> release my code that does this. [looks for code] ... Looks like it's
> not actually mine, but a client's. For the trivial stuff, it's, well,
> trivial, to reimplement.
Yeah, I don't know what the real problem is. Thankfully for many uses, most
servers support HTTP/1.0, which is vastly more simple than HTTP/1.1.
Since all of the optional bits with HTTP are, well, optional, there's no
protocol necessity to actually implement those details.
For a basic HTTP/1.0 GET, the two "hard parts" are formulating the URL (if
it has any query parameters), and then scraping the headers off of the
reply. Since the connection closes automatically when complete, read to EOF
and viola, done.
For a POST, it gets lightly more difficult in that you need to formulate the
actual POST data. The "challenge" there is ensure you have the
Content-Length correct.
Also, make sure you send (and read) CR/LF. It's part of the protocol.
If you want to make things like automatically following redirects and what
not, then that simply leverages on the original GET/POST code. It's an
incremental leap.
HTTP/1.1 gets more sophisticated with its continuous connections, chunked
reads and such. Then you can throw SSL on top of that (whee).
But, basic HTTP/1.0 is, well, basic.
Regards,
Will Hartung
(·····@msoft.com)
In article <··············@sidious.geddis.org>,
Don Geddis <···@geddis.org> wrote:
> I've found lots of (e.g. open-source) lisp packages that:
> 1. help you write CGI scripts in lisp
> 2. let you run a lisp web server
> 3. help with generating HTML
>
> But I haven't found any that provide the relatively trivial functionality
> of acting as a web _browser_, e.g. here's a URL, go and get the content.
>
> Does anyone know of any such package? Something like a lisp version of cURL:
> http://curl.haxx.se/
>
> Thanks,
>
> -- Don
> ______________________________________________________________________________
> _
> Don Geddis http://don.geddis.org/
> ···@geddis.org
> I guess one of my pet peeves is that sometimes I have no idea what people are
> talking about.
> -- Deep Thoughts, by Jack Handey [1999]
Cl-HTTP has a client. It also has a web walker for doing more
complicated stuff on web pages (like enumerating, etc.).
What I also regularly use is CL-HTTP's HTTP:COPY-FILE .
With that you can copy to and from web pages.
something like
(http:copy-file "http://foo.bar.com/baz.html" "/tmp/baz.html")
is fine.
but if you have configured PUT on a web server you also
can do it the other direction (!!):
(http:copy-file "/tmp/baz.html" "http://foo.bar.com/baz.html")