Greetings!
In the course of working with GCL and several complex applications
built atop it and shipped with Debian over the last few years, I've
increasingly realized the weaknesses of what I'm calling the 'On Lisp'
development model (after the book of the same name), and would like to
engineer some improvements.
Developers like lisp in part because everything, including the
compiler, is so well integrated. Well, the downside of this is that
when one needs to fix something in any one layer, GCL and all its
dependent programs must be recompiled to incorporate the change. What
looks great from a development point of view looks pretty bad from a
'shipping and maintaining working applications' point of view.
Somehow, it appears that what is needed is a lisp heap in an external
shared library, with copy-on-write pages handling the modifications,
and instructing #'save-system to write only dirty pages to disk and
properly overlay the shared lib on re-execution. This would appear
really intractable in a copying garbage collector system -- thank
goodness this is not the case for the bulk of the GCL heap.
I'm just soliciting ideas for improvement here -- not trying to start
any sort of flame war or the like.
Take care,
--
Camm Maguire ····@enhanced.com
==========================================================================
"The earth is but one country, and mankind its citizens." -- Baha'u'llah
From: Will Hartung
Subject: Re: Problems with the 'On Lisp' development model ...
Date:
Message-ID: <2uqqcmF2eccarU1@uni-berlin.de>
"Camm Maguire" <····@enhanced.com> wrote in message
···················@intech19.enhanced.com...
> Somehow, it appears that what is needed is a lisp heap in an external
> shared library, with copy-on-write pages handling the modifications,
> and instructing #'save-system to write only dirty pages to disk and
> properly overlay the shared lib on re-execution. This would appear
> really intractable in a copying garbage collector system -- thank
> goodness this is not the case for the bulk of the GCL heap.
I don't know GCL, but most of the other Lisps are basically "heap" based
using compiled FASL files. When you wish to send out a delta change, you
basically send a delta FASL that gets loaded when the system starts up as a
patch.
I also think that systems like Allegro will allow you to load FASL files as,
essentially, "read only" so that you can share them across multiple Lisp
processes, but save space much like a shared library.
Now, all I've heard about GCL is basically "it creates C with is compiled by
GCC", so I don't know what the resulting files are like for a GCL
application, and I don't know if the kind of behaviors I've described are
possible/practical with a GCL application.
Regards,
Will Hartung
(·····@msoft.com)
Greetings, and thanks for your reply!
"Will Hartung" <·····@msoft.com> writes:
> "Camm Maguire" <····@enhanced.com> wrote in message
> ···················@intech19.enhanced.com...
> > Somehow, it appears that what is needed is a lisp heap in an external
> > shared library, with copy-on-write pages handling the modifications,
> > and instructing #'save-system to write only dirty pages to disk and
> > properly overlay the shared lib on re-execution. This would appear
> > really intractable in a copying garbage collector system -- thank
> > goodness this is not the case for the bulk of the GCL heap.
>
> I don't know GCL, but most of the other Lisps are basically "heap" based
> using compiled FASL files. When you wish to send out a delta change, you
> basically send a delta FASL that gets loaded when the system starts up as a
> patch.
>
> I also think that systems like Allegro will allow you to load FASL files as,
> essentially, "read only" so that you can share them across multiple Lisp
> processes, but save space much like a shared library.
>
> Now, all I've heard about GCL is basically "it creates C with is compiled by
> GCC", so I don't know what the resulting files are like for a GCL
> application, and I don't know if the kind of behaviors I've described are
> possible/practical with a GCL application.
>
All of this is applicable to GCL and programs compiled on top of it --
thanks!
I'm still wondering how this would be done in practice in a
distribution setting like Debian, though. The mechanism you describe
above is an overlay of previously distributed code with the incorrect
behavior. Should these compiled FASL files (standard ELF .o files in
GCL's case with appended lisp initialization code) be packaged
separately, and installed into some known directory, with gcl, maxima,
acl2, and axiom searching this on startup and loading in the overlay?.
Or should there be a one time process which runs on patch installation
which would load the module into any installed maxima, acl2, or axiom
images present and resave the heap in place. This could be
problematic keeping track of future package installations, etc.
With shared libraries, everything is so simple. Upgrade libc, and
every dependent program has the fix on next execution, no overlay, no
bookkeeping.
Am I missing something?
Am also interested in Allegro's read-only policy. Can users not
change the functions therein, or are the old definitions preserved
while new ones are added elsewhere and fbound to the proper symbols,
the former being exempted from garbage collection and effectively
serving as dead allocated memory?
Take care,
> Regards,
>
> Will Hartung
> (·····@msoft.com)
>
>
>
--
Camm Maguire ····@enhanced.com
==========================================================================
"The earth is but one country, and mankind its citizens." -- Baha'u'llah
> I'm still wondering how this would be done in practice in a
> distribution setting like Debian, though. The mechanism you
> describe above is an overlay of previously distributed code with the
> incorrect behavior. Should these compiled FASL files (standard ELF
> .o files in GCL's case with appended lisp initialization code) be
> packaged separately, and installed into some known directory, with
> gcl, maxima, acl2, and axiom searching this on startup and loading
> in the overlay?. Or should there be a one time process which runs
> on patch installation which would load the module into any installed
> maxima, acl2, or axiom images present and resave the heap in place.
> This could be problematic keeping track of future package
> installations, etc.
CMU Lisp lets you use either approach. Building CMU Lisp involves
saving an image. You can do this at any time. You can also load fasl
files if you want to.
So there are two models that you could use:
1) Load your patched code into a buggy image. Save the image as a new
version and replace the old image with this new non-buggy image.
2) Have one or more patch directories that get scanned on startup to
load patched code.
You could actually do both --- periodically save an image with
well-tested code and load patches that aren't as well-tested on
startup.
--
Fred Gilham ······@csl.sri.com
Do remember you're there to fuddle him. From the way some of you
young fiends talk, anyone would suppose it was our job to teach!
-- The Screwtape Letters, C. S. Lewis
From: Will Hartung
Subject: Re: Problems with the 'On Lisp' development model ...
Date:
Message-ID: <2vcibvF2kh3moU1@uni-berlin.de>
"Camm Maguire" <····@enhanced.com> wrote in message
···················@intech19.enhanced.com...
> All of this is applicable to GCL and programs compiled on top of it --
> thanks!
>
> I'm still wondering how this would be done in practice in a
> distribution setting like Debian, though. The mechanism you describe
> above is an overlay of previously distributed code with the incorrect
> behavior. Should these compiled FASL files (standard ELF .o files in
> GCL's case with appended lisp initialization code) be packaged
> separately, and installed into some known directory, with gcl, maxima,
> acl2, and axiom searching this on startup and loading in the overlay?.
> Or should there be a one time process which runs on patch installation
> which would load the module into any installed maxima, acl2, or axiom
> images present and resave the heap in place. This could be
> problematic keeping track of future package installations, etc.
You could do it both ways, of course.
You could treat maxima as a single application, and load it independently
into your running Lisp image (much like you load an .exe into your system
when you want to run a program).
Or you could load maxima once, and then save the entire image as a whole.
Another thing you could do is put some startup code in your maxima
application that looks in a specified place for updates.
So, perhaps, you have Maxima ver. 3.0 (I don't know anything about maxima,
I'm making this up out of whole cloth...).
You can have maxima look in ~/.maxima/fasl for files named
"patch_3.0_*.fasl", and load them automatically at start up.
When you roll out a new version, it can look for updates in
"patch_3.1_*.fasl", or whatever.
This means that there is a start up cost to load the patches each time. You
could also, of course, save an image with the current patches at anytime,
and have the patch "patch" the loader to change where it starts to look for
future patches (i.e. what version number, say "patch_3.0_27")
This gives you the flexibility of automatic updates for which the user must
do nothing to get them, (save place them in the proper directory) yet if
they want that extra bleem of performance and forgo the patching process on
startup, they can take the extra step of creating a new image. IME, fasl
patches tend to be small and load quickly, however.
And, of course, you can always provide "pre-patched" images to any version
level you'd like.
So, what you get, "for free", is the ability to incrementally patch your
applications easily, yet also provide the latest and greatest. This also
make it easy for folks to make custom changes to their local version, yet
still load the incremental patches that are released from "Official"
versions. Assuming they don't conflict, that means you get to keep you local
changes and still be up to date. Obviously there can be issues with this,
but they are probably less of a problem than you might think.
> With shared libraries, everything is so simple. Upgrade libc, and
> every dependent program has the fix on next execution, no overlay, no
> bookkeeping.
>
> Am I missing something?
Certainly at a monolithic level, the shared library has its benefits (and
its curse vis-a-vi "DLL Hell"). But Lisp is a bit more organic that way, and
interdepdencies can get very ugly very quickly. When you have an image of
the system at your beck and call, you tend to leverage its capability,
giving you higher levels of integration. But it can certainly make
rebuilding difficult...
Also, with the mini-patches, you don't have to worry as much (IMHO) about
having the wrong version. Typically, you'd need "all" of the versions, and
they're all named properly. Whereas with a typical .so, they all share the
same name, "libxyz.so", regardless of the version (unless you do something
clever like use a link).
When you look in your "patch" directory, it's fairly obvious what the latest
version is (since they're all named that way). Also makes it easy to add
"optional" patches as well, something that is difficult with a shared
library.
Finally, its very easy to undo changes by removing the incremental bits. So,
in case the "new patch" "breaks something", it's easy to undo.
> Am also interested in Allegro's read-only policy. Can users not
> change the functions therein, or are the old definitions preserved
> while new ones are added elsewhere and fbound to the proper symbols,
> the former being exempted from garbage collection and effectively
> serving as dead allocated memory?
My understanding is that the read-only sections are just big blocks of
compiled Lisp that the image mounts and make available. You can redefine
anything you want that specified within the read-only image, because the new
binding will take place in your local read-write memory. Kind of like the
way the old Mac OS would patch the ROMs. The look up vectors were in RAM,
even though the routines were in ROM. You could change the RAM vector of a
routine to point to a new RAM version.
The idea was that if you had big chunks of Lisp that was fairly static, you
could save overal system memory across processes by sharing this code ala a
Shared Library, but since the symbols are in local RAM, you could redefine
them all you wanted. Caveat, I've never actually used this system, it's just
my understanding of how it works.
> Take care,
Regards,
Will Hartung
(·····@msoft.com)
From: Frode Vatvedt Fjeld
Subject: Re: Problems with the 'On Lisp' development model ...
Date:
Message-ID: <2hsm7svyb3.fsf@vserver.cs.uit.no>
Camm Maguire <····@enhanced.com> writes:
> Developers like lisp in part because everything, including the
> compiler, is so well integrated. Well, the downside of this is that
> when one needs to fix something in any one layer, GCL and all its
> dependent programs must be recompiled to incorporate the change.
> What looks great from a development point of view looks pretty bad
> from a 'shipping and maintaining working applications' point of
> view.
Could you explain more clearly what it is that causes you to need to
recompile GCL and all its dependents?
--
Frode Vatvedt Fjeld
Greetings!
Frode Vatvedt Fjeld <······@cs.uit.no> writes:
> Camm Maguire <····@enhanced.com> writes:
>
> > Developers like lisp in part because everything, including the
> > compiler, is so well integrated. Well, the downside of this is that
> > when one needs to fix something in any one layer, GCL and all its
> > dependent programs must be recompiled to incorporate the change.
> > What looks great from a development point of view looks pretty bad
> > from a 'shipping and maintaining working applications' point of
> > view.
>
> Could you explain more clearly what it is that causes you to need to
> recompile GCL and all its dependents?
>
Well, GCL has a 'heap' of memory with all its functions, symbols,
packages, compiler, etc. maxima, acl2, and axiom essentially compile
and load new packages and functions into this heap, and dump the
result to an executable file. In fact, in axiom's case, there are
several intermediary layers in this compile/load/dump cycle.
Take care,
> --
> Frode Vatvedt Fjeld
--
Camm Maguire ····@enhanced.com
==========================================================================
"The earth is but one country, and mankind its citizens." -- Baha'u'llah
From: lin8080
Subject: Re: Problems with the 'On Lisp' development model ...
Date:
Message-ID: <41883C20.DA1E0416@freenet.de>
Camm Maguire schrieb:
> Somehow, it appears that what is needed is a lisp heap in an external
> shared library, with copy-on-write pages handling the modifications,
> and instructing #'save-system to write only dirty pages to disk and
> properly overlay the shared lib on re-execution. ...
And who big becomes this libery in say 2 years of working?
stefan
loading ....