At regular intervals on comp.lang.lisp someone asks how to create
'standalone executables' Common Lisp. Rather than enter into a
technical discussion of what is meant by a 'standalone executables',
let me instead rephrase the question:
How do I deliver a Lisp application with the minimum amount of
burden on the recipient?
We wish to deliver to the user a package that he can easily unpack,
install, and run without having to know the details of compiling,
loading, and running Lisp. A secondary goal is to make sure the
delivered package is not unduly large. First, let us examine what goes
in to a Lisp program.
A Lisp system consists of one or more of these parts:
- A virtual machine
- An initial memory image for the virtual machine
- Runtime support libraries
- A number of compiled lisp files
- A number of lisp source files
These are much like you might expect in a Java system.
There are several options for delivery, depending on the needs and
sophistication of the recipient. A Lisp expert would probably prefer
the sources, or at most the compiled lisp files. A non-technical user
would likely not have the virtual machine installed. For the purposes
of this discussion, let us assume that our intended recipient has no
sort of Lisp on his machine at all.
The best option in this case may be to 'dump a world'. This involves
loading the compiled lisp files into the development system and
creating a snapshot of the memory contents. To run the application,
the Lisp virtual machine is started with the dumped world as its
initial image. The package which you deliver to the user will consist
of the virtual machine, the dynamic libraries that the virtual machine
needs, and the dumped world. You may also need to deliver a simple
script or batch file to initialize the virtual machine with the dumped
image.
Obviously, delivering an entire virtual machine and runtime support
along with the dumped image will take more room than the dumped image
alone. If the intended recipient already has the Lisp virtual machine
installed, then these may be omitted from the deliverable package. It
may be worthwhile to provide a package consisting only of the dumped
world.
Different Lisp systems have different virtual machines. This means
that if you developed your application under CLISP, you cannot expect
a dumped world to run under Franz Allegro. In addition, different Lisp
systems have different mechanisms by which you generate a dumped
world. Commercial systems such as Franz Allegro or Lispworks have
sophisticated mechanisms to create a monolithic deliverable that the
recipient can simply execute. These commercial systems may have 'tree
shakers' to prune unnecessary code and data from the delivered image.
Comments? Questions?
--
~jrm
Joe Marshall wrote:
let me instead rephrase the question:
How do I deliver a Lisp application with the minimum
amount of burden on the recipient?
We wish to deliver to the user a package that he can
easily unpack, install, and run without having to know
the details of compiling, loading, and running Lisp.
I have difficulties understanding what is being asked here.
Suppose I attempt to deploy my application, super-frob, as a
lisp source file. The instructions on my web page are
carefully worded and the substance of them is:
from the shell prompt, make a directory to keep super-frob
in
bash$ mkdir sf
Go to that directory
bash$ cd sf
Use your browser or download tool to fetch super-frob.lisp
For example
bash$ netscape http://www.super-frob.org/downloads.html
Quit from your download tool and check that super-frob has
arrived by using the ls command from the shell prompt
bash$ ls
Your computer should respond with a single line
super-frob.lisp
From the shell prompt, fire up your Lisp system
bash$ lisp
From the lisp prompt, compile the file
lisp* (compile-file "super-frob")
Your computer should response with a message such as
Converted FROB.
Compiling DEFUN FROB:
Byte Compiling Top-Level Form:
#p"/home/alan/tempdir/super-frob.x86f"
NIL
NIL
You can now run super-frob from the Lisp prompt
lisp* (load "super-frob")
The compilation step is only needed the first time you run
super-frob. Now that you have compiled super-frob you can
run it, either from Lisp, or from the shell prompt
bash$ lisp -eval '(load "super-frob")'
Does this meet the requirement of the the user not having
to know the /details/ of compiling, loading, and running Lisp?
I can argue both yes and no
NO, the user is having to run Lisp and use both
compile-file, and load
YES, the user is not required to know anything, let alone
details. He does not even have to type accurately. The
instructions on the webpage are literal instructions. He
can cut and paste them from the webpage, without filling in
any variables by putting finger to keyboard.
Is there a point to my quibbling? I hope so. In particular,
my experience is that the major burden I encounter as a
recipient of software is when the instructions don't work.
By "work" I mean that the instructions are keystroke
perfect. If they don't work it is because I have not
followed them accurately, and I can make them work by
starting again, and being careful and literal minded, doing
exactly what is asked, not a keystroke more or less.
So instructions that "work" require neither common sense nor
system administration skills. No error messages, requiring
the obvious corrective answer. No wizards asking me
questions I did not expect, and do not feel competent to
answer.
From this perspective, it doesn't matter very much whether
the instructions are simple or complicated. Complicated that
"works" beats simple that doesn't "work". Complicated/works
beats simple/broken every time, and by a large margin.
Joe's question contrasts unpacking and installing with
compiling, loading, and running. This seems to be orthogonal
to my concerns. I just want the instructions to work. If the
installation script that is supposed to make it easy for me
by hiding the ordeal of compiling, loading, and running from
me happens to break, that is a much bigger burden than
cutting and pasting a few opaque commands from a webpage.
So I see the question
How do I deliver a Lisp application with the minimum
amount of burden on the recipient?
and that takes me off in the direction wanting the most
reliable means of deployment. What decisions will allow me
to write instructions that will work for everybody, every
time? Are there problems with distributing source code that
might catch me by surprise?
Alan Crowe
Edinburgh
Scotland
"Alan Crowe" <····@cawtech.freeserve.co.uk> wrote in message
···················@cawtech.freeserve.co.uk...
....snip...
> From the shell prompt, fire up your Lisp system
>
> bash$ lisp
...snip...
> Are there problems with distributing source code that
> might catch me by surprise?
Besides the user not having a compatible Lisp system or some other minor
dependency (like defsystem, perhaps)?
Nope.
Besides the fact that there are those who may not care to share their source
code?
Well, I guess that's not really a surprise, is it?
The motivation of bundling an Application is that it then becomes even more
language agnostic in that the language chosen for development has lesser
impact on the generic user.
This was most exemplified on early Macintosh when it was pretty much a
requirement for a development environment to be able to distribute "double
clickable" applications.
Sun has, at least for internal deployment, addressed their application
distribution issues with WebStart, which is system that will not only
download the code for the application, but the appropriate JVM version if
necessary to run it.
It will certainly work over the internet (there are several WebStart enabled
demos and such out there), but if you get stuck with a JVM download, it's
going to hurt.
Back when OSX first came out, I was surprised and delighted that Apple chose
to include WebStart functionality out of the box.
WebStart comes with its own launcher that will even keep the app "up to
date", and provides an icon for the user to start the application.
Microsoft is running into similar issues with the fact that not everyone is
loading .NET, AND they're already running into versioning issues to where
people can not rely on a) whether a user has .NET at all or b) that the one
they have will even work.
It's already an issue in Java where different vendors ship a version of the
JVM with their app, and then run off that. Done haphazardly, you end up have
several different copies of the JVM on your system.
Of course, with Microsoft and .NET, I don't know how well the system likes
having multiple versions of the .NET runtime floating around. With Java,
it's bloaty, but at least they're easily isolated and don't conflict.
Once all of the dependencies, whatever they are, are on the users system,
distribution of applications becomes much easier. But getting those
dependencies is always the trick.
Regards,
Will Hartung
(·····@msoft.com)
"Will Hartung" <·····@msoft.com> writes:
> Sun has, at least for internal deployment, addressed their application
> distribution issues with WebStart, which is system that will not only
> download the code for the application, but the appropriate JVM version if
> necessary to run it.
The webstart thingie becomes way less impressive if you happen to be
outside of Sun's definition of "anywhere", which pretty much
translates to "on Win32/x86, Solaris/Sparc, or Linux/x86". I remember
WebStart being not very helpfull on FreeBSD, for example.
> Of course, with Microsoft and .NET, I don't know how well the system likes
> having multiple versions of the .NET runtime floating around.
Scince we discussed the sanity of Joel Spolsky recently anyway, here
is one of his recent rants that seems to be on topic:
<http://www.joelonsoftware.com/articles/PleaseLinker.html>
* Henrik Motakef wrote:
> The webstart thingie becomes way less impressive if you happen to be
> outside of Sun's definition of "anywhere", which pretty much
> translates to "on Win32/x86, Solaris/Sparc, or Linux/x86".
That's a 99% accurate definition of `anywhere', of course.
--tim
Alan Crowe <····@cawtech.freeserve.co.uk> writes:
> Joe Marshall wrote:
>
> let me instead rephrase the question:
>
> How do I deliver a Lisp application with the minimum
> amount of burden on the recipient?
>
> We wish to deliver to the user a package that he can
> easily unpack, install, and run without having to know
> the details of compiling, loading, and running Lisp.
What about:
tar zxvf myapp.tar.gz
myapp/myapp
?
Here is an extract of the Makefile I use to build such application
package (I've got similar rules to build with sbcl or cmucl (not yet
for cmucl :-( )):
CLISP=/usr/local/bin/clisp
CLISP_FLAGS_DEFAULT=-q -ansi -norc # no -K full since we want plain COMMON-LISP
CLISP_FLAGS=$(CLISP_FLAGS_DEFAULT)
CLISP_INIT=(LOAD \"init.lisp\")
CLISP_DEBUG=-interactive-debug
CLISP_NODEBUG=
CLISP_LINKINGSET=base
# IMG : lisp image
# BIN : lisp binary (executable)
# EXE : trampoline: small shell script used to launch the binary passing it
# the command-line arguments (in the ARGUMENT environment variable),
# and the lisp image and other options.
CLISP_IMG=$(PGMNAME)-clisp/$(PGMNAME).mem
CLISP_BIN=$(PGMNAME)-clisp/$(PGMNAME).bin
CLISP_EXE=$(PGMNAME)-clisp/$(PGMNAME)
CLISP_LSD=$(PGMNAME)-clisp/$(CLISP_LINKINGSET)
$(CLISP_EXE) : $(CLISP_BIN) $(CLISP_IMG).gz
@ echo "Generating $(CLISP_EXE)"
-$(TRACE) mkdir -m 755 -p $(CLISP_LSD) >/dev/null 2>&1 || true
$(TRACE) echo '#!/bin/bash' > $(CLISP_EXE)
$(TRACE) echo 'export ARGUMENTS="$$(echo -n \(\ ;for arg in "$$0" ···@";do b="$${arg//\\\\/\\\\}";c="$${b//\"/\\\"}";echo -n \""$${c}"\"\ ;done;echo \))"' >>$(CLISP_EXE)
$(TRACE) echo 'exec "$$(dirname $$0)/$(PGMNAME).bin" $(CLISP_FLAGS) -M "$$(dirname $$0)/$(PGMNAME).mem.gz"' >> $(CLISP_EXE)
$(TRACE) chmod 755 $(CLISP_BIN) $(CLISP_EXE)
$(TRACE) chmod 644 $(CLISP_IMG).gz
$(CLISP_IMG) : $($(PGMNAME)_OBJECTS) $($(PGMNAME)_CLISP_OBJECTS)
@ echo "Generating $(CLISP_IMG)"
-$(TRACE) mkdir -m 755 -p $(CLISP_LSD) >/dev/null 2>&1 || true
$(TRACE) $(CLISP) $(CLISP_FLAGS) $(CLISP_DEBUG) -x "$(CLISP_INIT) (LOAD \"$($(PGMNAME)_ROOT)\") (DEFUN MAIN () ($($(PGMNAME)_PKGNAME):MAIN (READ-FROM-STRING (EXT:GETENV \"ARGUMENTS\"))) (EXT:QUIT)) (EXT:SAVEINITMEM \"$(CLISP_IMG)\" :QUIET T :INIT-FUNCTION (FUNCTION MAIN)) (EXT:QUIT)"
$(CLISP_IMG).gz : $(CLISP_IMG)
$(TRACE) gzip -f $(CLISP_IMG)
$(CLISP_BIN) :
@ echo "Generating $(CLISP_BIN)"
-$(TRACE) mkdir -m 755 -p $(CLISP_LSD) >/dev/null 2>&1 || true
$(TRACE) cp `which $(CLISP)` $(CLISP_BIN)
······@
#or TRACE='' to see commands
--
__Pascal_Bourguignon__ http://www.informatimago.com/
There is no worse tyranny than to force a man to pay for what he doesn't
want merely because you think it would be good for him.--Robert Heinlein
http://www.theadvocates.org/
Alan Crowe <····@cawtech.freeserve.co.uk> writes:
> Joe Marshall wrote:
>
> let me instead rephrase the question:
>
> How do I deliver a Lisp application with the minimum
> amount of burden on the recipient?
>
> We wish to deliver to the user a package that he can
> easily unpack, install, and run without having to know
> the details of compiling, loading, and running Lisp.
>
> I have difficulties understanding what is being asked here.
> Suppose I attempt to deploy my application, super-frob, as a
> lisp source file.
I've given your example some thought.
> The instructions on my web page are
> carefully worded and the substance of them is:
[snipped detailed description of downloading]
> From the shell prompt, fire up your Lisp system
>
> bash$ lisp
>
> From the lisp prompt, compile the file
>
> lisp* (compile-file "super-frob")
>
> Your computer should response with a message such as
>
> Converted FROB.
> Compiling DEFUN FROB:
> Byte Compiling Top-Level Form:
>
> #p"/home/alan/tempdir/super-frob.x86f"
> NIL
> NIL
>
> You can now run super-frob from the Lisp prompt
>
> lisp* (load "super-frob")
>
> The compilation step is only needed the first time you run
> super-frob. Now that you have compiled super-frob you can
> run it, either from Lisp, or from the shell prompt
>
> bash$ lisp -eval '(load "super-frob")'
>
> Does this meet the requirement of the the user not having
> to know the /details/ of compiling, loading, and running Lisp?
At this point, I have to say no, this isn't in the `spirit' of
delivering an application.
> Is there a point to my quibbling? I hope so. In particular,
> my experience is that the major burden I encounter as a
> recipient of software is when the instructions don't work.
For instance, if you are not using bash, if your lisp is not named
`lisp', if your lisp does not include a file compiler, if your lisp
does not accept -eval as a command prompt, etc.
> Joe's question contrasts unpacking and installing with
> compiling, loading, and running. This seems to be orthogonal
> to my concerns. I just want the instructions to work.
I don't think it is orthogonal, though it might appear that way. You
just want the instructions to work. The most foolproof way to achieve
this is to not have instructions.
> If the installation script that is supposed to make it easy for me
> by hiding the ordeal of compiling, loading, and running from me
> happens to break, that is a much bigger burden than cutting and
> pasting a few opaque commands from a webpage.
I agree. Much attention has to be placed on `auto-install' of some
sort.
> So I see the question
>
> How do I deliver a Lisp application with the minimum
> amount of burden on the recipient?
>
> and that takes me off in the direction wanting the most
> reliable means of deployment. What decisions will allow me
> to write instructions that will work for everybody, every
> time? Are there problems with distributing source code that
> might catch me by surprise?
Good questions.
--
~jrm
* Alan Crowe wrote:
> Does this meet the requirement of the the user not having
> to know the /details/ of compiling, loading, and running Lisp?
> I can argue both yes and no
No, it fails, `Not knowing the details' means: click on the link in
your web browser. The program will run, displaying pretty pictured,
installing suitable viruses on your computer and participating in a
DDoS attack on SCO.
--tim
Joe Marshall wrote:
> At regular intervals on comp.lang.lisp someone asks how to create
> 'standalone executables' Common Lisp. Rather than enter into a
> technical discussion of what is meant by a 'standalone executables',
> let me instead rephrase the question:
Yes, entering into technical discussions has little affect. IMO, a direct
reference to a tutorial example should be given. This of course depends on
the CL implementation, but each tutorial should be direct, step by step and
without any value statements in the directions. Just state that that it is
the way it is. Just the facts. End of story.
Even talking about CL as a virtual machine is too theoretical.
for example:
CMUCL startup with a #! executable "hello World" example.
Wade
Wade Humeniuk <····································@telus.net> wrote:
+---------------
| Joe Marshall wrote:
| > At regular intervals on comp.lang.lisp someone asks how to create
| > 'standalone executables' Common Lisp. Rather than enter into a
| > technical discussion of what is meant by a 'standalone executables'...
|
| Yes, entering into technical discussions has little affect.
| IMO, a direct reference to a tutorial example should be given.
...
| for example:
| CMUCL startup with a #! executable "hello World" example.
+---------------
Yes, some tutorial material on that should be helpful, since as distributed
CMUCL does not come with a built-in reader macro for "#!" [even though most
of us using CMUCL for any time quickly build a core image containing one!].
Fortunately, most of the Unix/Linux operating systems assume a /bin/sh
script if the script does not start with "#!" (or "#", for "csh" users),
so with the minimal cost of a fork/exec of /bin/sh, the following idiom[1]
suffices for most scripting needs:
% cat foo
":" ; exec cmucl -quiet -noinit -load "`which $0`" ······@"}
(format t "Hello, world!~%")
(quit)
% time foo
Hello, world!
0.009u 0.012s 0:00.03 33.3% 592+3636k 0+0io 0pf+0w
%
19 milliseconds -- not too shabby! [Well, on a 1.8 GHz Athlon... ;-} ]
Add a few "convenience features", and you get this:
% cat foo2
":" ; exec cmucl -quiet -noinit -load "`which $0`" ······@"}
;;; Provide script name and command-line args [if any] to script:
(defvar *script-name* (nth 4 *command-line-strings*)) ; See [2], below.
(defvar *script-args* (nthcdr 5 *command-line-strings*))
(defvar *script-interactive-repl* nil)
(format t "Hello, world!~%")
(format t "My name = '~a'~%" *script-name*)
(loop for i from 1
and a in *script-args*
do (format t "arg[~a] = '~a'~%" i a))
(unless *script-interactive-repl* ; set when debugging.
(quit))
% foo2 bar baz 'many words here'
Hello, world!
My name = 'foo2'
arg[1] = 'bar'
arg[2] = 'baz'
arg[3] = 'many words here'
%
And as long as I'm on the topic, it seems I made a mistake in my posting
the other day when I showed a script "header" for FASL files. Turns out it
was written for an older version of CMUCL in which READ-LINE didn't complain
if given a binary stream. But CMUCL-18e's READ-LINE *does* bitch and moan
(arguably rightfully), so here is an updated version (tested) that works
nicely with CMUCL-18e:
% cat bin/cmucl-fasl-script-header
":" ; exec cmucl -quiet -noinit -load "`which $0`" ······@"}
;;;; Hack to allow CMUCL FASL files to run as shell scripts.
;;;;
;;;; USAGE: cat {this_file} foo1.x86f ... fooN.x86f > foo ; chmod +x foo
;;;; The last "fooN.x86f" must end with a top-level form which starts the
;;;; application [and then should fall-through, *not* exit! -- see below].
;;; Provide script name and command-line args [if any] to FASL file(s):
(defvar *script-name* (nth 4 *command-line-strings*))
(defvar *script-args* (nthcdr 5 *command-line-strings*))
(defvar *script-interactive-repl* nil)
(setf ext::*complain-about-illegal-switches* nil)
(with-open-file (s *script-name* :element-type '(unsigned-byte 8))
(loop for c = (read-byte s) ; Eat header text from stream through
until (= c 90) ; final "magic character" [uppercase "z",
finally (read-byte s)) ; see below] and the newline which
(load s :contents :binary) ; follows, then pass the rest to FASLOAD.
(if *script-interactive-repl* ; If set later, start up a Lisp top-level.
(continue) ; Keep initial LOAD of script from falling into FASL files.
(quit))) ; So non-REPL case exits cleanly.
;;;; NOTA BENE: The end of this line is magic: Z
%
which is used this way:
% cat foo3.lisp
(format t "Hello, world!~%")
(format t "My name = '~a'~%" *script-name*)
(loop for i from 1
and a in *script-args*
do (format t "arg[~a] = '~a'~%" i a))
% cmucl-compile foo3.lisp
...compiler chatter...
; foo3.x86f written.
; Compilation finished in 0:00:00.
% cat cmucl-fasl-script-header foo3.x86f >foo3
% chmod +x foo3
% ls -l foo3
-rwxr-xr-x 1 rpw3 rpw3 2714 Feb 7 05:00 foo3
% foo3 what a "really fun" hack.
Hello, world!
My name = 'foo3'
arg[1] = 'what'
arg[2] = 'a'
arg[3] = 'really fun'
arg[4] = 'hack.'
%
-Rob
[1] Which I first learned from the PLT scheme, crowd. Thanks guys!
[2] CMUCL experts may wonder why I didn't use the "command-line-switch"
routines here, e.g., (car (get-command-line-switch "load")) instead
of (nth 4 *command-line-strings*)) for *script-name*, etc., and the
(setf ext::*complain-about-illegal-switches* nil). The reason is that
the above can easily be emulated in CLISP and other CLs, and is more
similar to what the user expects coming from a C or Perl environment.
And the SETF makes it easy to use application-specific options, e.g.:
% foo3 -v -o bar bar.in
Hello, world!
My name = 'foo3'
arg[1] = '-v'
arg[2] = '-o'
arg[3] = 'bar'
arg[4] = 'bar.in'
%
-----
Rob Warnock <····@rpw3.org>
627 26th Avenue <URL:http://rpw3.org/>
San Mateo, CA 94403 (650)572-2607
On Sat, 07 Feb 2004 07:20:13 -0600, Rob Warnock wrote:
> Fortunately, most of the Unix/Linux operating systems assume a /bin/sh
> script if the script does not start with "#!" (or "#", for "csh" users),
> so with the minimal cost of a fork/exec of /bin/sh, the following idiom[1]
> suffices for most scripting needs:
It is the shell which assumes that. A script not beginning with #! is not
recognizable by the kernel and will not work when run from something other
than directly from a shell. A proper standalone executable script must
begin with #!.
--
__("< Marcin Kowalczyk
\__/ ······@knm.org.pl
^^ http://qrnik.knm.org.pl/~qrczak/
On Sat, Feb 07, 2004 at 04:28:17PM +0100, Marcin 'Qrczak' Kowalczyk wrote:
> On Sat, 07 Feb 2004 07:20:13 -0600, Rob Warnock wrote:
>
> > Fortunately, most of the Unix/Linux operating systems assume a /bin/sh
> > script if the script does not start with "#!" (or "#", for "csh" users),
> > so with the minimal cost of a fork/exec of /bin/sh, the following idiom[1]
> > suffices for most scripting needs:
>
> It is the shell which assumes that. A script not beginning with #! is not
> recognizable by the kernel and will not work when run from something other
> than directly from a shell. A proper standalone executable script must
> begin with #!.
Try:
#!/bin/sh
":" ; exec lisp -quiet -noinit -nositeinit -eval "(set-dispatch-macro-character #\# #\! #'(lambda (s c n) c n (peek-char #\newline s nil) nil))" -load "`which $0`" ······@"}
(format t "Hello, world!~%")
(quit)
--
; Matthew Danish <·······@andrew.cmu.edu>
; OpenPGP public key: C24B6010 on keyring.debian.org
; Signed or encrypted mail welcome.
; "There is no dark side of the moon really; matter of fact, it's all dark."
Joe Marshall <·············@comcast.net> writes:
> At regular intervals on comp.lang.lisp someone asks how to create
> 'standalone executables' Common Lisp. Rather than enter into a
> technical discussion of what is meant by a 'standalone executables',
> let me instead rephrase the question:
...
> Comments? Questions?
Seems jolly good for giving a newbie an overview. If anything, you
might also mention how patching can be done. (Well, there's no
universal way, I guess, but at least there's more support for it than
just the standard "download and install the whole new version".)
Best,
Thomas
--
Thomas Lindgren
"It's becoming popular? It must be in decline." -- Isaiah Berlin