From: George Smith
Subject: Communicating between CMU CL and unix
Date: 
Message-ID: <36123B8C.A33135F4@hockett.phil1.uni-potsdam.de>
I've figured out how to access unix environment variables from within
CMU CL (18a).

	ext:*environment-list*

is an alist of variables and values, so something like:

	(cdr (assoc :HOME ext:*environment-list*))

is nice to be able to use.

I'm calling lisp from a shell script (which is itself called by
another program) and can communicate query strings from the calling
program via the shell script to lisp using unix environment variables.

Now I would like to communicate a result from lisp back to the shell
script which called it.

Environment variables don't seem to be good way to do this, since as
far as I understand, they aren't (can't be?) communicated from child
to parent. What else would be a good way to communicate from lisp to
the calling process? My initial thought is to write to a stream in
lisp which the calling process could then read from.

If I start lisp in the following manner from the shell script:

lisp ··@" -load test.lisp

Lisp prints that it is loading test.lisp. I can, for example, redirect
this to a file:

lisp ··@" -load test.lisp > foo

Without the redirection, the message stating that lisp is loading the
file is communicated by the shell script to the calling program. It
appears to me that if I could figure out what stream lisp is writing
to in this case, I could write to that same stream and my problem
would be solved. What stream could that be?

Output from a lisp function that writes to either *standard-output* or
system:*stdout* doesn't "show up" in the same way as does the above
message.

The form

	(do-all-symbols (symbol)
			(when (and (boundp symbol)
				   (streamp (symbol-value symbol))
				   (output-stream-p (symbol-value symbol)))
		      	(print symbol)))

yields

	SYSTEM:*STDOUT* 
	SYSTEM:*TTY* 
	SYSTEM:*STDERR* 
	*DEBUG-IO* 
	*QUERY-IO* 
	*TERMINAL-IO* 
	*ERROR-OUTPUT* 
	*TRACE-OUTPUT* 
	*STANDARD-INPUT* 
	*STANDARD-OUTPUT* 
	C::*COMPILER-ERROR-OUTPUT* 

I don't see any other obvious candidates here. Those streams that I
did try didn't "work".

I'm currently writing to a file in lisp and, in the shell script,
using cat to read the file and write it on  the standard output.

This works in testing. But in real use I would not only have the
presumably unnecessary file system activity, but also I would have to
deal with creating and deleting temporary files and making sure that
no two processes use temporary files with the same path.

Any suggestions?

Thanks,

George Smith
Institut fuer Germanistik
Universitaet Potsdam

From: Raymond Toy
Subject: Re: Communicating between CMU CL and unix
Date: 
Message-ID: <4nsoh9v9pk.fsf@rtp.ericsson.se>
>>>>> "George" == George Smith <············@hockett.phil1.uni-potsdam.de> writes:

    George> If I start lisp in the following manner from the shell
    George> script:

    George> lisp ··@" -load test.lisp

    George> Lisp prints that it is loading test.lisp. I can, for
    George> example, redirect this to a file:

    George> lisp ··@" -load test.lisp > foo

    George> Without the redirection, the message stating that lisp is
    George> loading the file is communicated by the shell script to
    George> the calling program. It appears to me that if I could
    George> figure out what stream lisp is writing to in this case, I
    George> could write to that same stream and my problem would be
    George> solved. What stream could that be?

Try using the -batch option.  I think then everything written to
*standard-output* goes where you expect it to go.

Ray
From: Martin Cracauer
Subject: Re: Communicating between CMU CL and unix
Date: 
Message-ID: <6vakg8$gds$1@counter.bik-gmbh.de>
Raymond Toy <···@rtp.ericsson.se> writes:

>>>>>> "George" == George Smith <············@hockett.phil1.uni-potsdam.de> writes:

>    George> If I start lisp in the following manner from the shell
>    George> script:

>    George> lisp ··@" -load test.lisp

>    George> Lisp prints that it is loading test.lisp. I can, for
>    George> example, redirect this to a file:

>    George> lisp ··@" -load test.lisp > foo

>    George> Without the redirection, the message stating that lisp is
>    George> loading the file is communicated by the shell script to
>    George> the calling program. It appears to me that if I could
>    George> figure out what stream lisp is writing to in this case, I
>    George> could write to that same stream and my problem would be
>    George> solved. What stream could that be?

>Try using the -batch option.  I think then everything written to
>*standard-output* goes where you expect it to go.

I once wrote this to to make CMUCL shut up unless it is asked
explicitly (that means: you write something yourself). It also causes
the debugger to just print a backtrace instead of trying to interact.

You have to save a CMUCL image with (setq *load-verbose* nil). Then,
pass the function you want to (silent <fun>) or the file to load to
(runcgi <filename>).

Martin

(setq *load-verbose* nil)

(defparameter *dev-null*
  #-lispm
  (make-two-way-stream (make-concatenated-stream) (make-broadcast-stream))
  ;; Since Lisp Machines have a built-in /dev/null which handles
  ;; additional, non-standard operations, we'll use that instead.
  #+lispm #'system:null-stream)

(defun silent-wrap (fun)
  (let ((res nil)
	(bso *standard-output*)
	(bse *error-output*)
	)
    (setq *standard-output* *dev-null*)
    (setq *error-output* *dev-null*)
    (format t "Darf nicht zu sehen sein")
    (setq res (funcall fun))
    (setq *error-output* bse)
    (setq *standard-output* bso)
    res))


(defun silenterr (condition)
  (format *error-output* "CMUCL-silent uncaught condition: ~a~%" condition)
  (debug:backtrace)
  (unix:unix-exit 5))

(defun silent (fun) 
;  (setf *silent-erroutput* *error-output*)
  (setf *debug-io* *error-output*) ; for example, to display backtraces
  (handler-bind (
		 (error #'(lambda (c) (silenterr c))))
		(funcall fun)))

(defun runcgi (filename)
  (silent #'(lambda () (load filename)))
  (quit))

(defun runcgi-compile (filename)
  (silent #'(lambda () (compile-file filename :load t)))
  (quit))
From: Martin Cracauer
Subject: Re: Communicating between CMU CL and unix
Date: 
Message-ID: <6vako8$gtc$1@counter.bik-gmbh.de>
Raymond Toy <···@rtp.ericsson.se> writes:

>>>>>> "George" == George Smith <············@hockett.phil1.uni-potsdam.de> writes:

>    George> If I start lisp in the following manner from the shell
>    George> script:

>    George> lisp ··@" -load test.lisp

>    George> Lisp prints that it is loading test.lisp. I can, for
>    George> example, redirect this to a file:

>    George> lisp ··@" -load test.lisp > foo

>    George> Without the redirection, the message stating that lisp is
>    George> loading the file is communicated by the shell script to
>    George> the calling program. It appears to me that if I could
>    George> figure out what stream lisp is writing to in this case, I
>    George> could write to that same stream and my problem would be
>    George> solved. What stream could that be?

>Try using the -batch option.  I think then everything written to
>*standard-output* goes where you expect it to go.

I once wrote this to to make CMUCL shut up unless it is asked
explicitly (that means: you write something yourself). It also causes
the debugger to just print a backtrace instead of trying to interact.

You have to save a CMUCL image with (setq *load-verbose* nil). Then,
pass the function you want to (silent <fun>) or the file to load to
(runcgi <filename>).

Martin

(setq *load-verbose* nil)

(defparameter *dev-null*
  #-lispm
  (make-two-way-stream (make-concatenated-stream) (make-broadcast-stream))
  ;; Since Lisp Machines have a built-in /dev/null which handles
  ;; additional, non-standard operations, we'll use that instead.
  #+lispm #'system:null-stream)

(defun silent-wrap (fun)
  (let ((res nil)
	(bso *standard-output*)
	(bse *error-output*)
	)
    (setq *standard-output* *dev-null*)
    (setq *error-output* *dev-null*)
    (format t "Darf nicht zu sehen sein")
    (setq res (funcall fun))
    (setq *error-output* bse)
    (setq *standard-output* bso)
    res))


(defun silenterr (condition)
  (format *error-output* "CMUCL-silent uncaught condition: ~a~%" condition)
  (debug:backtrace)
  (unix:unix-exit 5))

(defun silent (fun) 
;  (setf *silent-erroutput* *error-output*)
  (setf *debug-io* *error-output*) ; for example, to display backtraces
  (handler-bind (
		 (error #'(lambda (c) (silenterr c))))
		(funcall fun)))

(defun runcgi (filename)
  (silent #'(lambda () (load filename)))
  (quit))

(defun runcgi-compile (filename)
  (silent #'(lambda () (compile-file filename :load t)))
  (quit))
From: Martin Cracauer
Subject: Re: Communicating between CMU CL and unix
Date: 
Message-ID: <6val4f$hbm$1@counter.bik-gmbh.de>
Raymond Toy <···@rtp.ericsson.se> writes:

>>>>>> "George" == George Smith <············@hockett.phil1.uni-potsdam.de> writes:

>    George> If I start lisp in the following manner from the shell
>    George> script:

>    George> lisp ··@" -load test.lisp

>    George> Lisp prints that it is loading test.lisp. I can, for
>    George> example, redirect this to a file:

>    George> lisp ··@" -load test.lisp > foo

>    George> Without the redirection, the message stating that lisp is
>    George> loading the file is communicated by the shell script to
>    George> the calling program. It appears to me that if I could
>    George> figure out what stream lisp is writing to in this case, I
>    George> could write to that same stream and my problem would be
>    George> solved. What stream could that be?

>Try using the -batch option.  I think then everything written to
>*standard-output* goes where you expect it to go.

I once wrote this to to make CMUCL shut up unless it is asked
explicitly (that means: you write something yourself). It also causes
the debugger to just print a backtrace instead of trying to interact.

You have to save a CMUCL image with (setq *load-verbose* nil). Then,
pass the function you want to (silent <fun>) or the file to load to
(runcgi <filename>).

Martin

(setq *load-verbose* nil)

(defparameter *dev-null*
  #-lispm
  (make-two-way-stream (make-concatenated-stream) (make-broadcast-stream))
  ;; Since Lisp Machines have a built-in /dev/null which handles
  ;; additional, non-standard operations, we'll use that instead.
  #+lispm #'system:null-stream)

(defun silent-wrap (fun)
  (let ((res nil)
	(bso *standard-output*)
	(bse *error-output*)
	)
    (setq *standard-output* *dev-null*)
    (setq *error-output* *dev-null*)
    (format t "Darf nicht zu sehen sein")
    (setq res (funcall fun))
    (setq *error-output* bse)
    (setq *standard-output* bso)
    res))


(defun silenterr (condition)
  (format *error-output* "CMUCL-silent uncaught condition: ~a~%" condition)
  (debug:backtrace)
  (unix:unix-exit 5))

(defun silent (fun) 
;  (setf *silent-erroutput* *error-output*)
  (setf *debug-io* *error-output*) ; for example, to display backtraces
  (handler-bind (
		 (error #'(lambda (c) (silenterr c))))
		(funcall fun)))

(defun runcgi (filename)
  (silent #'(lambda () (load filename)))
  (quit))

(defun runcgi-compile (filename)
  (silent #'(lambda () (compile-file filename :load t)))
  (quit))
From: Martin Cracauer
Subject: Re: Communicating between CMU CL and unix
Date: 
Message-ID: <6vam2r$i2i$1@counter.bik-gmbh.de>
Raymond Toy <···@rtp.ericsson.se> writes:

>>>>>> "George" == George Smith <············@hockett.phil1.uni-potsdam.de> writes:

>    George> If I start lisp in the following manner from the shell
>    George> script:

>    George> lisp ··@" -load test.lisp

>    George> Lisp prints that it is loading test.lisp. I can, for
>    George> example, redirect this to a file:

>    George> lisp ··@" -load test.lisp > foo

>    George> Without the redirection, the message stating that lisp is
>    George> loading the file is communicated by the shell script to
>    George> the calling program. It appears to me that if I could
>    George> figure out what stream lisp is writing to in this case, I
>    George> could write to that same stream and my problem would be
>    George> solved. What stream could that be?

>Try using the -batch option.  I think then everything written to
>*standard-output* goes where you expect it to go.

I once wrote this to to make CMUCL shut up unless it is asked
explicitly (that means: you write something yourself). It also causes
the debugger to just print a backtrace instead of trying to interact.

You have to save a CMUCL image with (setq *load-verbose* nil). Then,
pass the function you want to (silent <fun>) or the file to load to
(runcgi <filename>).

Martin

(setq *load-verbose* nil)

(defparameter *dev-null*
  #-lispm
  (make-two-way-stream (make-concatenated-stream) (make-broadcast-stream))
  ;; Since Lisp Machines have a built-in /dev/null which handles
  ;; additional, non-standard operations, we'll use that instead.
  #+lispm #'system:null-stream)

(defun silent-wrap (fun)
  (let ((res nil)
	(bso *standard-output*)
	(bse *error-output*)
	)
    (setq *standard-output* *dev-null*)
    (setq *error-output* *dev-null*)
    (format t "Darf nicht zu sehen sein")
    (setq res (funcall fun))
    (setq *error-output* bse)
    (setq *standard-output* bso)
    res))


(defun silenterr (condition)
  (format *error-output* "CMUCL-silent uncaught condition: ~a~%" condition)
  (debug:backtrace)
  (unix:unix-exit 5))

(defun silent (fun) 
;  (setf *silent-erroutput* *error-output*)
  (setf *debug-io* *error-output*) ; for example, to display backtraces
  (handler-bind (
		 (error #'(lambda (c) (silenterr c))))
		(funcall fun)))

(defun runcgi (filename)
  (silent #'(lambda () (load filename)))
  (quit))

(defun runcgi-compile (filename)
  (silent #'(lambda () (compile-file filename :load t)))
  (quit))
From: Martti Halminen
Subject: Re: Communicating between CMU CL and unix
Date: 
Message-ID: <361259CF.7C3C@rm-spam-trap.dpe.fi>
George Smith wrote:

> I'm calling lisp from a shell script (which is itself called by
> another program) and can communicate query strings from the calling
> program via the shell script to lisp using unix environment variables.
> 
> Now I would like to communicate a result from lisp back to the shell
> script which called it.
> 
> Environment variables don't seem to be good way to do this, since as
> far as I understand, they aren't (can't be?) communicated from child
> to parent. What else would be a good way to communicate from lisp to
> the calling process? My initial thought is to write to a stream in
> lisp which the calling process could then read from.


If I understood your problem correctly this seems to be exactly what
Expect was written for. You could just write your results to
*standard-output* and have expect catch it, filtering out any
non-interesting stuff. Worked for me with CMU_CL 17f a few years back,
at least.


See http://expect.nist.gov/ for more info on Expect.



-- 
________________________________________________________________
    ^.          Martti Halminen
   / \`.        Design Power Europe Oy
  /   \ `.      Tekniikantie 12, FIN-02150 Espoo, Finland
 /\`.  \ |      Tel:+358 9 4354 2306, Fax:+358 9 455 8575
/__\|___\|      ······················@dpe.fi   http://www.dpe.fi
From: Steve Gonedes
Subject: Re: Communicating between CMU CL and unix
Date: 
Message-ID: <m2lnn0desl.fsf@KludgeUnix.com>
George Smith <············@hockett.phil1.uni-potsdam.de> writes:


< I'm calling lisp from a shell script (which is itself called by
< another program) and can communicate query strings from the calling
< program via the shell script to lisp using unix environment variables.
<
< Now I would like to communicate a result from lisp back to the shell
< script which called it.
<
< Environment variables don't seem to be good way to do this, since as
< far as I understand, they aren't (can't be?) communicated from child
< to parent. What else would be a good way to communicate from lisp to
< the calling process? My initial thought is to write to a stream in
< lisp which the calling process could then read from.


I think it may be possible to modify an environmental variable, but I
believe it isn't safe or portable (I could also be remembering wrong).
Have you thought about using FIFOs? They work well in certain
situations. You could use the process environment to initiate the
communication.

< I'm currently writing to a file in lisp and, in the shell script,
< using cat to read the file and write it on  the standard output.
<
< This works in testing. But in real use I would not only have the
< presumably unnecessary file system activity, but also I would have to
< deal with creating and deleting temporary files and making sure that
< no two processes use temporary files with the same path.

You can use `$$' for the temp file, and in bash you can trap on EXIT.
Just set the proper traps and reuse a FIFO.

< Any suggestions?

I like the idea of using the process environment, you could pass the
lisp the name of a FIFO to write to. Here is something I just tossed
together which has the lisp read a file and write it to the fifo while
the shell spawns a process to read the fifo and spit the info to
stdout. It works just like the unix `cat' command (sorta).

The lisp will expect to have a list with two keywords bound to the
environment variable `LISP_FILES'. The keywords are :input and :fifo.
This makes the protocol simple as you don't have to remeber the order
of the files and such. Also I'm not sure how CMUCL stores the value of
a environment variable, I imagine it would be a string which makes it
available for read-from-string.


;;; BOF

(in-package :user)

(defun cat-to-fifo (infile outfifo)
  (flet ((fifo-exists-p (file)
           (or (probe-file file)
               ;; let the shell catch up
               (progn (sleep 1) (probe-file file)))))

    (if (not (fifo-exists-p outfifo))
        (format *error-output* "~2&Cannot locate fifo: ~a" outfifo)
      (with-open-file
          (output outfifo :direction :output
           :if-exists :supersede)
        (with-open-file
            (input infile :direction :input)
          (loop repeat (file-length input) doing
             (write-char (read-char input) output))
           ;; write the #\null so the shell knows we're done
          (write-char #\Null output))))))

(defun get-env (var)
   (cdr (assoc var ext:*environment-list*)))

(defun fetch-files-from-environment ()
  ;; values: input file name, output fifo name
  (destructuring-bind (&key input fifo)
      (read-from-string (get-env :LISP_FILES))
    (values input fifo)))

(defun lisp-catenation ()
  (multiple-value-bind (in-file out-fifo)
      (fetch-files-from-environment)
    (cat-to-fifo in-file out-fifo)))

;;;; EOF

*** Beginning of Limbo

The shell script follows. I'm not sure how you would have the lisp
eval a form from the command-line, acl uses the '-e' so you could just
replace those lines with whatever is appropriate.

You could also just put `(lisp-catenation) (quit)' at the bottom of
the lisp file - it's kinda hard to test it that way though.

*** end of Limbo

### BOF
#! /bin/bash

if [ -z "$*" ]
then
   echo "Usage: `basename $0` <file>" 1>&2
   exit 1
fi

runlisp () {
   local file=$1
   LISP_FILES="(:input \"$file\" :fifo \"$fifo\")"
   bash -c "cat < $fifo" &
   lisp -e '(load "fifo.fasl")' \
        -e '(lisp-catenation)' \
        -e '(exit 0 :quiet t)'
   echo
}

fifo=$$
trap "rm -f $fifo" SIGHUP SIGINT EXIT
mkfifo "$fifo" || (echo "Cannot make fifo: $fifo" 1>&2 ; exit 1)

for file in $*
do
   if [ ! -f $file ]
   then
      echo "No such file: $file" 1>&2
      exit 1
   else
      runlisp $file
   fi
done
exit 0
### EOF


Maybe this will give you some ideas. You could have two FIFOs one for
input and one for output - but I think that this would get too
complicated. You could use unix domain sockets or TCP sockets. There
is a program that lets you use sockets from a shell script (I don't
remeber the name - sorry).

Anyway, hope you enjoy playing with your problem as much as I did :)
Maybe you could post an update on what you decided to do - it sounds
like a neat problem.
From: George Smith
Subject: Re: Communicating between CMU CL and unix
Date: 
Message-ID: <36137ABE.773776B8@hockett.phil1.uni-potsdam.de>
Steve Gonedes wrote:
> 
> George Smith <············@hockett.phil1.uni-potsdam.de> writes:

[...]

> < Now I would like to communicate a result from lisp back to the shell
> < script which called it.

[...]

> I like the idea of using the process environment, you could pass the
> lisp the name of a FIFO to write to. Here is something I just tossed
> together which has the lisp read a file and write it to the fifo while
> the shell spawns a process to read the fifo and spit the info to
> stdout. It works just like the unix `cat' command (sorta).
> 
> The lisp will expect to have a list with two keywords bound to the
> environment variable `LISP_FILES'. The keywords are :input and :fifo.
> This makes the protocol simple as you don't have to remeber the order
> of the files and such. Also I'm not sure how CMUCL stores the value of
> a environment variable, I imagine it would be a string which makes it
> available for read-from-string.

Yes, it is a string.

[...]

> Maybe this will give you some ideas. You could have two FIFOs one for
> input and one for output - but I think that this would get too
> complicated. You could use unix domain sockets or TCP sockets. There
> is a program that lets you use sockets from a shell script (I don't
> remeber the name - sorry).
> 
> Anyway, hope you enjoy playing with your problem as much as I did :)
> Maybe you could post an update on what you decided to do - it sounds
> like a neat problem.

I certainly enjoyed playing with your solution!

About the same time I finally understood everything, something in your
solution or the man pages I consulted while working through it made me
wonder whether maybe *standard-output* was being buffered and not
flushed before lisp quit. It turns out that that was the case. If I
call #'finish-output on *standard-output* before lisp quits, the shell
script which calls lisp receives everything and can redirect it where
I want it to go.

Thanks to everyone for the suggestions,

George Smith
Institut fuer Germanistik
Universitaet Potsdam