From: verec
Subject: fasl and lisp image
Date: 
Message-ID: <472d3552$0$508$5a6aecb4@news.aaisp.net.uk>
Assuming a compiler-only CL implementation (like SBCL) what is the
fundamental difference between the lisp image and the set of .fasl
files that cen be somehow combined to "re-create" it?

In other words, and in this "compiler-only" context, is it possible
to simplify and say that:

- at boot time, some "kernel" loads a predefined set of fasl files
- any new definition (at the REPL) is just stored into a
  <package>:repl.fasl of some sort, with only that name
  eg: cl-user:repl.fasl being written down as "scratch" file to be
  remembered accross REPL sessions?

Stated yet another way, does a "compiler-only" CL implementation
need anything more than a remembered set of fasl to load (both
predefined list and per-session saved) and thus get entirely
rid of both the concept and the implementation and support of
the "lisp-image" ?

Many thanks for unveiling yet another corner of the guts below CL :-)
--
JFB

From: Ari Johnson
Subject: Re: fasl and lisp image
Date: 
Message-ID: <m2mytud3a5.fsf@nibbler.theari.com>
verec <·····@mac.com> writes:

> Assuming a compiler-only CL implementation (like SBCL) what is the
> fundamental difference between the lisp image and the set of .fasl
> files that cen be somehow combined to "re-create" it?

The fundamental difference is that a lisp image is generally mapped
into memory, as by mmap(2), while fasl files each consist of a set of
some number of objects that are loaded into memory just as if they
were being created in memory at the time they are loaded.

> In other words, and in this "compiler-only" context, is it possible
> to simplify and say that:

Why limit yourself to a compiler-only context?  Interpreted functions
as equally susceptible to being stored in fasls as are their compiled
kin.

> - at boot time, some "kernel" loads a predefined set of fasl files

There are probably bootstrapping issues with only having fasls and not
having a traditional lisp image mapped into memory.  For instance,
without knowing the address of NIL, you can't fast-load any lists.

> Stated yet another way, does a "compiler-only" CL implementation
> need anything more than a remembered set of fasl to load (both
> predefined list and per-session saved) and thus get entirely
> rid of both the concept and the implementation and support of
> the "lisp-image" ?

Anything is possible.
From: verec
Subject: Re: fasl and lisp image
Date: 
Message-ID: <472d5827$0$510$5a6aecb4@news.aaisp.net.uk>
On 2007-11-04 04:48:34 +0000, Ari Johnson <·········@gmail.com> said:

>> In other words, and in this "compiler-only" context, is it possible
>> to simplify and say that:
> 
> Why limit yourself to a compiler-only context?  Interpreted functions
> as equally susceptible to being stored in fasls as are their compiled
> kin.

What is the difference between a .lisp file and an "interpreted function
as equally susceptible to being stored in fasls" ?

I thought that I understood that one of Lisp strength was that
its source code *was* its parse tree, in which case I don't understand
why a .fasl would contain explicitely that parse tree for later
interpretation, when it could just as well reference its original
source? What am I missing?

Many thanks
--
JFB
From: Rainer Joswig
Subject: Re: fasl and lisp image
Date: 
Message-ID: <joswig-902ED8.10274104112007@news-europe.giganews.com>
In article <·······················@news.aaisp.net.uk>,
 verec <·····@mac.com> wrote:

> Assuming a compiler-only CL implementation (like SBCL) what is the
> fundamental difference between the lisp image and the set of .fasl
> files that cen be somehow combined to "re-create" it?
> 
> In other words, and in this "compiler-only" context, is it possible
> to simplify and say that:
> 
> - at boot time, some "kernel" loads a predefined set of fasl files
> - any new definition (at the REPL) is just stored into a
>   <package>:repl.fasl of some sort, with only that name
>   eg: cl-user:repl.fasl being written down as "scratch" file to be
>   remembered accross REPL sessions?
> 
> Stated yet another way, does a "compiler-only" CL implementation
> need anything more than a remembered set of fasl to load (both
> predefined list and per-session saved) and thus get entirely
> rid of both the concept and the implementation and support of
> the "lisp-image" ?

What's the difference between code and the result of its execution?

It also is the difference between building a house
every time or just walking into the house that is there.

The result depends on:

* the value of RANDOM
* the contents on the disk
* data read from networks
* system events
* data read via streams

and so on.

The image captures the state of the application
(minus some problems when you need recreate
possibly external objects like windows).
Reaching the state of the application can be time
consuming.

The image also is generated by a function that can
do a few things like reorganizing the
bits in memory for optimized layout or compressing
the result.

Historically the image could be booted in minutes,
but the (even fasl) code needed possibly hours to
(compile and) load. Even loading of fasl code
could take a long time. But even today, you
can either start an image in a second or
wait possibly a minute or more every time to get your
code loaded and state recreated.

When I work with a Lisp machine, much care is taken
to define a file that loads all the code and
state I need. Usually I make the environment
usable (setting up a few things, and loading basic
patches. I set up all the environment data,
definition of networks, servers, file-systems, ...)
and save a base image.
Using that base image I load all the code.

I load all the documentation
and make it writeable. I load all the data
I need. Load database definitions. Setup restart
functions. Then I save an optimized incremental
image. When I later restart the Lisp,
the base image plus the incremental image
gets loaded. On an old system this would take
three minutes on a new it takes a few seconds.
Then I just load the patches and all is there
and ready to use.

Say, you have an in-core database with large amounts
of data - interning this data possibly takes
time (hours, days, weeks, ...). Saving an image and
restarting that is fast. Trying to recreate
that from scratch takes time.

Eventually you need to test out if loading code into
a fresh Lisp works, but during development
using an image works just fine and saves a lot of time.

You can also transport the image easily to another machine,
since it is most of the time just one file.

I guess that with more Lisp usage over time and libraries
like McCLIM, images are getting more useful.

> 
> Many thanks for unveiling yet another corner of the guts below CL :-)
> --
> JFB
From: verec
Subject: Re: fasl and lisp image
Date: 
Message-ID: <472e2804$0$513$5a6aecb4@news.aaisp.net.uk>
On 2007-11-04 09:27:42 +0000, Rainer Joswig <······@lisp.de> said:

> In article <·······················@news.aaisp.net.uk>,
>  verec <·····@mac.com> wrote:
> 
> What's the difference between code and the result of its execution?
> 
> It also is the difference between building a house
> every time or just walking into the house that is there.
> 
> The result depends on:
> 
> * the value of RANDOM
> * the contents on the disk
> * data read from networks
> * system events
> * data read via streams
> 
> and so on.

I see. But that's more an issue of convenience than strict
theoretical difference, no? As for "events" I am not sure
what you mean. State? Yes definitely. But AFAIK, state is
"preserved" in .fasl too, no?

I mean that if you have a (defparameter *xyz* ....) inside
a .lisp file at some point compiled to a .fasl, and if your
(exit) function does a "for-each-package, save-to-<package>.fasl)
then the state is preserved the same way it would if you were
to (save-image) no?

> You can also transport the image easily to another machine,
> since it is most of the time just one file.

You could probably transport .fasl in the same way, no?

So far, it seems that all the arguments I have seen in favor
of a separate Lisp image are about loading speed that we
all know that the (in)fanous "sufficiently smart compiler" could
sort out ... :-)

Any *fundamental* reason that I am missing?

Many thanks
--
JFB
From: Rainer Joswig
Subject: Re: fasl and lisp image
Date: 
Message-ID: <joswig-B25CB0.22081104112007@news-europe.giganews.com>
In article <·······················@news.aaisp.net.uk>,
 verec <·····@mac.com> wrote:

> On 2007-11-04 09:27:42 +0000, Rainer Joswig <······@lisp.de> said:
> 
> > In article <·······················@news.aaisp.net.uk>,
> >  verec <·····@mac.com> wrote:
> > 
> > What's the difference between code and the result of its execution?
> > 
> > It also is the difference between building a house
> > every time or just walking into the house that is there.
> > 
> > The result depends on:
> > 
> > * the value of RANDOM
> > * the contents on the disk
> > * data read from networks
> > * system events
> > * data read via streams
> > 
> > and so on.
> 
> I see. But that's more an issue of convenience than strict
> theoretical difference, no? As for "events" I am not sure
> what you mean.

Mouse clicks, messages from other programs, window events,
signals, ...

> State? Yes definitely. But AFAIK, state is
> "preserved" in .fasl too, no?

No.

> 
> I mean that if you have a (defparameter *xyz* ....) inside
> a .lisp file

No, the compiler compiles it to instructions it does
not dump the contents of the variables.

You can do (defparameter *xyz* #.foo) and have the
compiler dump the contents of foo.

> at some point compiled to a .fasl, and if your
> (exit) function does a "for-each-package, save-to-<package>.fasl)
> then the state is preserved the same way it would if you were
> to (save-image) no?

No, the compiler does not know how to dump arbitrary data
to fasls. Read about what the compiler can dump and
what you can do about it in the CLHS.

http://www.lisp.org/HyperSpec/Body/sec_3-2-4.html

Example with closures:

The file only contains:
(defparameter *foo-closure* #.*foo*)



* (defparameter *foo* (let ((f 10)) (lambda () f)))

*FOO*
* (compile-file "/tmp/test1.lisp")

; compiling file "/tmp/test1.lisp" (written 04 NOV 2007 10:00:08 PM):
; compiling (DEFPARAMETER *FOO-CLOSURE* ...)
; file: /tmp/test1.lisp
; in: DEFPARAMETER *FOO-CLOSURE*
;     (DEFPARAMETER *FOO-CLOSURE* #<FUNCTION # {11632B2D}>)
; --> PROGN EVAL-WHEN SB-IMPL::%DEFPARAMETER SB-IMPL::%DEFPARAMETER 
; ==>
;   #<FUNCTION (LAMBDA #) {11632B2D}>
; 
; caught ERROR:
;   Objects of type FUNCTION can't be dumped into fasl files.
; 



> > You can also transport the image easily to another machine,
> > since it is most of the time just one file.
> 
> You could probably transport .fasl in the same way, no?

Sure. With the necessary information in which order to load them.
> 
> So far, it seems that all the arguments I have seen in favor
> of a separate Lisp image are about loading speed that we
> all know that the (in)fanous "sufficiently smart compiler" could
> sort out ... :-)

It is not about speed. It is about having all data available
and just doing a memory dump. Loading the memory dump
brings you back to the point where you were before.

> Any *fundamental* reason that I am missing?

That's fundamental.

> 
> Many thanks
> --
> JFB
From: verec
Subject: Re: fasl and lisp image
Date: 
Message-ID: <472e4496$0$516$5a6aecb4@news.aaisp.net.uk>
On 2007-11-04 21:08:11 +0000, Rainer Joswig <······@lisp.de> said:

>> I see. But that's more an issue of convenience than strict
>> theoretical difference, no? As for "events" I am not sure
>> what you mean.
> 
> Mouse clicks, messages from other programs, window events,
> signals, ...

I still don't understand how an event, which is by nature
transient, can be preserved. That a state resulting from an
event could, yes. But preserving an event itself? This
escapes me.

>> at some point compiled to a .fasl, and if your
>> (exit) function does a "for-each-package, save-to-<package>.fasl)
>> then the state is preserved the same way it would if you were
>> to (save-image) no?
> 
> No, the compiler does not know how to dump arbitrary data
> to fasls. Read about what the compiler can dump and
> what you can do about it in the CLHS.
> 
> http://www.lisp.org/HyperSpec/Body/sec_3-2-4.html
> 
> Example with closures:
> 
> The file only contains:
> (defparameter *foo-closure* #.*foo*)
> 
> * (defparameter *foo* (let ((f 10)) (lambda () f)))
> 
> *FOO*
> * (compile-file "/tmp/test1.lisp")
> 
> ; compiling file "/tmp/test1.lisp" (written 04 NOV 2007 10:00:08 PM):
> ; compiling (DEFPARAMETER *FOO-CLOSURE* ...)
> ; file: /tmp/test1.lisp
> ; in: DEFPARAMETER *FOO-CLOSURE*
> ;     (DEFPARAMETER *FOO-CLOSURE* #<FUNCTION # {11632B2D}>)
> ; --> PROGN EVAL-WHEN SB-IMPL::%DEFPARAMETER SB-IMPL::%DEFPARAMETER
> ; ==>
> ;   #<FUNCTION (LAMBDA #) {11632B2D}>
> ;
> ; caught ERROR:
> ;   Objects of type FUNCTION can't be dumped into fasl files.
> ;

Ah. Thanks .,,

I've read this 3.2.4.1 and this section

http://www.lispworks.com/documentation/HyperSpec/Body/26_glo_e.htm#externalizable_object

of 

which a function is not an example of, and also all the
associated friends (MAKE-LOAD-FORM, and associated clean-up issues)
but I still fail to see how this is anything other than a concession
to existing implementations at the time the standard was devised,
as opposed to a fundamental issue tied to the deep semantics of
the language.

In other words, if I were to instrument SBCL or other compiler
I have access to, such that a compiled function could be saved
in the same way as it currently is within the image, then that
difference would disappear, no?

Or are you saying that there is a fundamental reason, tied to
the very nature of CL that will prevent me from ever being able
to instrument a compiler such that a compiled function could
be saved to and restored from a fasl?

Many thanks
--
JFB
From: Rainer Joswig
Subject: Re: fasl and lisp image
Date: 
Message-ID: <joswig-C728DF.23370704112007@news-europe.giganews.com>
In article <·······················@news.aaisp.net.uk>,
 verec <·····@mac.com> wrote:

> On 2007-11-04 21:08:11 +0000, Rainer Joswig <······@lisp.de> said:
> 
> >> I see. But that's more an issue of convenience than strict
> >> theoretical difference, no? As for "events" I am not sure
> >> what you mean.
> > 
> > Mouse clicks, messages from other programs, window events,
> > signals, ...
> 
> I still don't understand how an event, which is by nature
> transient, can be preserved. That a state resulting from an
> event could, yes. But preserving an event itself? This
> escapes me.

If I dump my Lisp IDE it contains lots of stuff, editor buffers,
input histories, CLOS caches, application state, ...

How do you want recreate that from a FASL?

Sure you can take a Lisp code, compile that Lisp code,
even combine the compiled code to one file (some implementations
have functionality for that) and reload that on startup.
The you still don't have the data. You have to load it
somehow. The compiler can't dump it (unless you tell him
how to do it) in the general case.

Still, dumping the image and restarting it is much simpler.

...

> > http://www.lisp.org/HyperSpec/Body/sec_3-2-4.html

...

> http://www.lispworks.com/documentation/HyperSpec/Body/26_glo_e.htm#externalizable_object
> 
> of 
> 
> which a function is not an example of, and also all the
> associated friends (MAKE-LOAD-FORM, and associated clean-up issues)
> but I still fail to see how this is anything other than a concession
> to existing implementations at the time the standard was devised,
> as opposed to a fundamental issue tied to the deep semantics of
> the language.
> 
> In other words, if I were to instrument SBCL or other compiler
> I have access to, such that a compiled function could be saved
> in the same way as it currently is within the image, then that
> difference would disappear, no?

In Common Lisp no implementation is required to be able to dump closures.
 
> Or are you saying that there is a fundamental reason, tied to
> the very nature of CL that will prevent me from ever being able
> to instrument a compiler such that a compiled function could
> be saved to and restored from a fasl?

No, there are compilers which can do that.

But that does not solve the general problem of dumping arbitrary
data to fasl files or the need to reload data.

> 
> Many thanks
> --
> JFB