Hello,
I need to know whether it is possible to run some iterations
of a loop on a separate machine altogether.
Will it require some networking code ?
My model is Single Instruction Multiple Data.
Some thing like quick-sort : data can be split and it
is possible to do the 2 recursive calls independently,
so why not do it on separate machines, and collect the
result. In my case, the result is printed in a file.
TIA
Sandeep Koranne
--
"Lets Make Things Better"
ED&T Test, WAY 3.37
Philips Research Laboratories Botstraat, 7
5656 AA, Eindhoven 5654 NL, Eindhoven
The Netherlands The Netherlands
Phone: +31-(040)-27 45250 +31-(040)-2573492
Fax : +31-(040)-27 44626
E-mail: ···············@philips.com
Sandeep Koranne <·······@natlab.research.philips.com> writes:
> Hello,
> I need to know whether it is possible to run some iterations
> of a loop on a separate machine altogether.
Yes, it is possible.
> Will it require some networking code ?
Since reaching some other computer usually involves networks,
networking code will at some level usually be involved.
> My model is Single Instruction Multiple Data.
Which means that not only will you probably need to call some
function, but you will also need to move data from one address space
to another. This usually involves some form of printing and reading
back (or some optimized form thereof).
Depending on the data involved, and the implementations, this may be
quite easy to achieve, or a bit harder to do.
What kind of data will need to be transfered? What implementation(s)
will you want to use for this?
For easily printable/readable data (i.e. data for which printing and then
re-reading will be much faster than the processing you want to perform
on the data), and no security requirements it might as easy as
establishing a socket, listening for connections, and on connect to
read-eval(-print) the form that is passed in ASCII over the socket.
E.g. for CMUCL on x86 you could do (shamelessly snarfed and morphed
from MP code by Douglas T. Crosher in real-time, so any bugs are most
certainly mine):
(defun connection-handler (stream host)
(declare (ignore host))
(let ((*print-readably* t))
(print (ignore-errors (eval (read stream))) stream))
(finish-output stream)
(close stream))
(defun ip-address-string (address)
(format nil "~D.~D.~D.~D"
(ldb (byte 8 24) address)
(ldb (byte 8 16) address)
(ldb (byte 8 8) address)
(ldb (byte 8 0) address)))
(defun connection-listener (port handler)
(let ((fd (ext:create-inet-listener port)))
(unwind-protect
(progn
(setf (mp:process-name mp:*current-process*)
(format
nil "EVAL connection listener on port ~D with handler ~A"
port handler))
(loop
(mp:process-wait-until-fd-usable fd :input)
(multiple-value-bind (new-fd remote-host)
(ext:accept-tcp-connection fd)
(let ((host-entry (ext:lookup-host-entry remote-host))
(stream (sys:make-fd-stream new-fd :input t :output t)))
(mp:make-process
#'(lambda ()
(funcall handler stream host-entry))
:name (format nil "Connection from ~A"
(if host-entry
(ext:host-entry-name host-entry)
(ip-address-string remote-host))))))))
(when fd (unix:unix-close fd)))))
(defun establish-connection-listener (port)
(setf mp::*idle-process* mp::*initial-process*)
(mp:make-process
#'(lambda () (connection-listener port #'connection-handler))))
And on the client (master) side:
(defun remote-eval (form host port)
(let* ((fd (ext:connect-to-inet-socket host port))
(stream (sys:make-fd-stream fd :input t :output t)))
(let ((*print-readably* t))
(print form stream))
(terpri stream)
(finish-output stream)
(prog1
(read stream)
(close stream))))
The above works by blocking until the remote result is available, but
this doesn't need to be so: You could handle this either on the
server (slave) side, by implementing commands that do the processing
in the background (and signal the end of processing for examble by
out-of-band data on the socket), or on the server side, by
multiprocessing. Without MP support, things get a little bit more
complicated, but not unsolvably so.
On other implementations you can do very similar things. Note that
this depends on your data being "print-readably-p".
On CMUCL you might also want to take a look at the WIRE package.
Regs, Pierre.
--
Pierre Mai <····@acm.org> PGP and GPG keys at your nearest Keyserver
"One smaller motivation which, in part, stems from altruism is Microsoft-
bashing." [Microsoft memo, see http://www.opensource.org/halloween1.html]
····@acm.org (Pierre R. Mai) writes:
It just occurred to me that maybe I had not emphasized the following
point enough:
[ Sandeep's request and my response partially elided, the following is
me again: ]
> For easily printable/readable data (i.e. data for which printing and then
> re-reading will be much faster than the processing you want to perform
> on the data), and no security requirements it might as easy as
> establishing a socket, listening for connections, and on connect to
> read-eval(-print) the form that is passed in ASCII over the socket.
> E.g. for CMUCL on x86 you could do (shamelessly snarfed and morphed
> from MP code by Douglas T. Crosher in real-time, so any bugs are most
> certainly mine):
To repeat: The code that was following in my original posting offers
NO SECURITY PROTECTION AT ALL, SO ANYONE WHO CAN CONNECT TO THE CHOSEN
PORT ON YOUR COMPUTER WILL BE ABLE TO DO ANYTHING THAT THE USER WHO
STARTED THE LISP PROCESS WOULD HAVE BEEN ABLE TO DO, INCLUDING
READING, RETRIEVING AND DELETING FILES, POSSIBLY BECOMING ROOT, ETC.
DON'T USE THE CODE ON A COMPUTER/PORT THAT ANYONE YOU DON'T TRUST
TOTALLY CAN ACCESS!
In situations where there exist possibilities that untrusted users can
access the port in question, then other protocols have to be used to
authenticate the user, and it might be advisable not to expose the
whole R-E-P loop of the Lisp in question, but rather define a special
purpose (Lisp-based) protocol that only allows benign interactions.
Beware of *read-eval* if the implementation of the protocol uses read
for reading requests. Don't evaluate anything directly, and only read
from the stream when you have done sanity checks. Etc.
See postings by Erik Naggum and others on c.l.l on protocol design for
these sort of situations.
BTW: Using something like CORBA is another way of doing what you want,
and having the ORB's deal with many of the security issues in
question, if done right.
Regs, Pierre.
--
Pierre Mai <····@acm.org> PGP and GPG keys at your nearest Keyserver
"One smaller motivation which, in part, stems from altruism is Microsoft-
bashing." [Microsoft memo, see http://www.opensource.org/halloween1.html]