I am working on this project (undergraduate) and need some advice for
the overall architecture. I know, you guys may think this is too
complicated; but I am just throwing out ideas.
Basically an agent is an entity that responds to some receptors,
outside influences [1].
I want to build an online agent server that will respond to my
database of indexed web data.
Here is where it gets a little complicated. I want to use Django for
the web frontend (yes, I could use a lisp server; but I really want to
use some existing Django code); I want to use the python server to
upload the lisp scripts. The lisp scripts contain code to control the
agents. The code will use a collection of different libraries calls
and hopefully I can restrict volatile calls like read/write file
operations, etc.
A user submits their lisp agent script to the web server. At this
point, the request is "waiting to be processed". I don't expect the
user to wait until everything is done. The webserver then puts the
lisp code on the queue to be processed. I want to use RabbitMQ or
some other Queuing library to process agent scripts on a FIFO order.
My Question:
This is where I ask the lisp community for help; How do you think I
should process the scripts off the queue. The scripts won't take too
long to execute; for example, I might have a request:
Intelligent Agent Script on the Queue:
"Give me the top web-pages from the top websites in the last month. I
am going to count the top keywords from that resultset"
How should I poll the queue for scripts and how should I process the
scripts once I know I need to process them.
I was thinking of having a long running lisp server or process
(possibly a web server?) that will poll the queue every 5 minutes and
then extract the scripts off the queue and then handle accordingly.
What kind of lisp server? I don't know.
And I would also have to evaluate the agent scripts within the context
of that running server. The scripts could potentially hose the state
of the server. Also, I am not 100% sure how to do this.
My other approach;
Have a cron job that polls the queue and then when a script is
available on the queue; launch a lisp process to handle the script and
exit when it is complete.
This has issues for obvious reasons; you aren't sharing resources.
There could be 1000 lisp processes running at one time.
It is a simpler approach and the lisp scripts can't effect the state
of the other lisp processes.
Like I said, just a little project. I really want to architect some
lisp server or go that route. It is more complicated but I see more
benefits if I go that way.
References
[1] http://aima.cs.berkeley.edu/
From: colarte
Subject: Re: An architecture for a school project. Lisp for intelligent agents
Date:
Message-ID: <fvg6en$2db$1@aioe.org>
Berlin Brown wrote:
> My Question:
> This is where I ask the lisp community for help; How do you think I
> should process the scripts off the queue. The scripts won't take too
> long to execute; for example, I might have a request:
Hello Berlin, interpreting each agent interaction would be costly in
memory and processor resources, better is the other way you are
thinking, besides if you have a running server you can load some
libraries that you might use continuosly.
The one in which you have a server backend, whose only job is to process
each queue . You should do some IPC (Inter Process Communication) via
Unix sockets o internet sockets in between your django code and your
backend server in lisp.
If the communication in between sockets is to difficult you might think
on having another lisp web server and just trying to invoke the results
of your lisp web server in between django's web response.
About the cron job to handle the queue should not be necessary if you
backend server starts to process the queue once the agent request comes.
The tricky part is that you are not going to answer back the results of
the queue inmediately. So you can start the processing of the queue and
return that it is "being processed", meanwhile the processing might be
executing in a separate thread so there is no blocking.
Hope it helps Good Luck.
Camilo Olarte
On May 2, 7:00 pm, colarte <·············@gmail.com> wrote:
> Berlin Brown wrote:
> > My Question:
> > This is where I ask the lisp community for help; How do you think I
> > should process the scripts off the queue. The scripts won't take too
> > long to execute; for example, I might have a request:
>
> Hello Berlin, interpreting each agent interaction would be costly in
> memory and processor resources, better is the other way you are
> thinking, besides if you have a running server you can load some
> libraries that you might use continuosly.
>
> The one in which you have a server backend, whose only job is to process
> each queue . You should do some IPC (Inter Process Communication) via
> Unix sockets o internet sockets in between your django code and your
> backend server in lisp.
>
> If the communication in between sockets is to difficult you might think
> on having another lisp web server and just trying to invoke the results
> of your lisp web server in between django's web response.
>
> About the cron job to handle the queue should not be necessary if you
> backend server starts to process the queue once the agent request comes.
>
> The tricky part is that you are not going to answer back the results of
> the queue inmediately. So you can start the processing of the queue and
> return that it is "being processed", meanwhile the processing might be
> executing in a separate thread so there is no blocking.
>
> Hope it helps Good Luck.
>
> Camilo Olarte
Yea, that is what I was thinking. I was trying to see if I didn't
sound too crazy.
Would you do think about araneida handling a lot of the requests from
the django web server to the lisp server. I could start threads from
araneida to do the processing.
Only problem there, I would have to send data from django to the lisp
web server. Ideally, a lighter weight tcp protocol would have been
easier as opposed to HTTP.
From: colarte
Subject: Re: An architecture for a school project. Lisp for intelligent agents
Date:
Message-ID: <fvitto$dvr$1@aioe.org>
>
> Only problem there, I would have to send data from django to the lisp
> web server. Ideally, a lighter weight tcp protocol would have been
> easier as opposed to HTTP.
Best integration i ever saw was with apache and mod-python, cause
python interpreter was loaded on apache process , so interpretation of a
python script was very fast .
Note that oposing apache-mod-lisp the interpreter on mod-lisp is an
external server which handles request across sockets.
So in an ideal world the lisp "interpeter" would be loaded in the apache
process thus making it very fast to interpret.
But in real world , I have not founded/done anything that works that
fast with lisp and apache. Most close thing was a cgi that "talked"
sockets to a clisp server. But it is sayed that mod-lisp with some
appropiate parameter tweaking would be very fast.
NOTE: that i've been talking about making apache talk to lisp fastly ,
not python talking to lisp fastly.
And also , sending data in between django and python should not be an
issue, cause you can modify some options on apache server to "proxy"
certain url so that those urls go directly to the lisp web server on the
background. see: http://www.accela.net/~dankna/guide.html
So: in your django code you would write html that includes urls that are
parsed directly by araneida/lisp...
Look at other web frameworks, specially those that have n-tiers....
zope, struts, app-fuse...