Here's my situation: I'm using CL to conduct experiments that involve
controlling machinery and monitoring test equipment. It's the kind of
thing LabView is used for extensively, I believe.
I have a growing number of small 'libraries', each consisting of from
10 to a few hundred lines of lisp. I build an experiment by writing a
top-level program in lisp, and it includes a bunch of the libraries.
Each experiment includes a different bunch of libraries. Some of the
libraries depend on each other, but mostly they dont.
The functionality I want is something similar to what's provided in C
via #include. Or in Python via import. I'd like to have a single
location where I could put all my libraries, and a means to easily
load them as required.
To run an experiment, I want to start up lisp and in the REPL just
(load "experiment"). That will load experiment.lisp or experiment.fasl
from the current directory, and then either run the experiment
automatically, or wait for me to invoke a function from the REPL.
As far as I can tell, the canonical way to do this is to make each of
my libraries into a package, and to define a system for each package.
So, for each library I would create xxx.lisp and xxx.asd files. I
would put those files in the asdf central registry. Then each of my
experiments would begin with a sequence of calls to (asdf:oos...) to
load the appropriate libraries.
I understand how to do that, but it seems like a lot of work, and
doubles the number of files I have to deal with. Have I got it
basically right, or is there some simplification that I'm missing? It
looks as though the CL functions require and provide are almost what I
want.. except not. I notice that the hyperspec says they are
deprecated, because non-portable. Do people use them?
In my fantasy, I wish for a function like (load-lib "libname"). It
would look for libname in a list of loaded libraries. If libname was
not in the list, it would search for "libname.fasl" or "libname.lisp"
in a set of standard directories (that I would initialize in my lisp
startup file). If if found the library, it would load it, as if I had
typed (load "libname"). Then, to create a new library, I just drop the
lisp file into the standard location.
(Note: if I was doing this in C, I would be using a makefile. Perhaps
this indicates that asdf would be the correct solution for CL after
all).
Neil Baylis <···········@gmail.com> writes:
> Here's my situation: I'm using CL to conduct experiments that involve
> controlling machinery and monitoring test equipment. It's the kind of
> thing LabView is used for extensively, I believe.
>
> I have a growing number of small 'libraries', each consisting of from
> 10 to a few hundred lines of lisp. I build an experiment by writing a
> top-level program in lisp, and it includes a bunch of the libraries.
> Each experiment includes a different bunch of libraries. Some of the
> libraries depend on each other, but mostly they dont.
>
> The functionality I want is something similar to what's provided in C
> via #include. Or in Python via import. I'd like to have a single
> location where I could put all my libraries, and a means to easily
> load them as required.
>
> To run an experiment, I want to start up lisp and in the REPL just
> (load "experiment"). That will load experiment.lisp or experiment.fasl
> from the current directory, and then either run the experiment
> automatically, or wait for me to invoke a function from the REPL.
>
> As far as I can tell, the canonical way to do this is to make each of
> my libraries into a package, and to define a system for each package.
> So, for each library I would create xxx.lisp and xxx.asd files. I
> would put those files in the asdf central registry. Then each of my
> experiments would begin with a sequence of calls to (asdf:oos...) to
> load the appropriate libraries.
>
> I understand how to do that, but it seems like a lot of work, and
> doubles the number of files I have to deal with. Have I got it
> basically right, or is there some simplification that I'm missing? It
> looks as though the CL functions require and provide are almost what I
> want.. except not. I notice that the hyperspec says they are
> deprecated, because non-portable. Do people use them?
>
> In my fantasy, I wish for a function like (load-lib "libname"). It
> would look for libname in a list of loaded libraries. If libname was
> not in the list, it would search for "libname.fasl" or "libname.lisp"
> in a set of standard directories (that I would initialize in my lisp
> startup file). If if found the library, it would load it, as if I had
> typed (load "libname"). Then, to create a new library, I just drop the
> lisp file into the standard location.
>
> (Note: if I was doing this in C, I would be using a makefile. Perhaps
> this indicates that asdf would be the correct solution for CL after
> all).
Yes, but it's overkill.
Just do it. What you fancy I mean. Write a loader.lisp file (or call
it libname.lisp if you like), that contains:
(dolist (src '(lib1 lib2 lib3 lib4 ...))
(load (compile-file (make-pathname :name (string src)
:type "LISP" :case :common))))
Just put the libs in the right order, that is, if lib1 defines a
_macro_ or a _function_ used by a _macro_ in lib2, then lib1 must be
loaded before lib2. 'used' at macro expansion time.
This can be done really easily with:
(dolist (src '(packages macros lib1 lib2 lib3 lib4 ...))
(load (compile-file (make-pathname :name (string src)
:type "LISP" :case :common))))
having all the DEFPACKAGEs in the file packages.lisp, all the
DEFMACROs in the file macros.lisp, and then the libs in random order.
But then, you would just load this loader.lisp file once. Per
generation of a lisp image. That is, you boot a lisp, load the
loader, which loads all the libraries in the image, and then save the
lisp image, possibly even as an executable.
#+clisp ext:saveinitmem
#+sbcl sb-ext:save-lisp-and-dir
So when you want to run your experiments, you just launch the lisp
image with all the libraries already loaded, and you don't have to
bother with dependencies: you will have all you need at your
fingertip.
--
__Pascal Bourguignon__ http://www.informatimago.com/
"Indentation! -- I will show you how to indent when I indent your skull!"
From: Richard M Kreuter
Subject: Re: Managing many small libraries
Date:
Message-ID: <87eja2zvqv.fsf@progn.net>
···@sevak.isi.edu (Thomas A. Russ) writes:
> It is actually best not to include a file type, so that load
> will then load compiled binaries if they exist.
Not necessarily. From the CLHS entry for LOAD:
| If the file specification is not complete and both a source file and
| a compiled file exist which might match, then which of those files
| ‘load’ selects is implementation-dependent.
--
RmK
From: Kent M Pitman
Subject: Re: Managing many small libraries
Date:
Message-ID: <uy78a90o8.fsf@nhplace.com>
Richard M Kreuter <·······@progn.net> writes:
> ···@sevak.isi.edu (Thomas A. Russ) writes:
>
> > It is actually best not to include a file type, so that load
> > will then load compiled binaries if they exist.
>
> Not necessarily. From the CLHS entry for LOAD:
>
> | If the file specification is not complete and both a source file and
> | a compiled file exist which might match, then which of those files
> | ‘load’ selects is implementation-dependent.
It's true it's not strictly well-defined, but I tend to agree with
tar's advice anyway... In practice I think it works often enough that
it's the right thing to do. (Kind of like still relying on Newton
rather than Einstein when working out simple physics problems.)
As I recall, the reason for this was lack of agreement among the
committee about whether the filename "foo" should parse as
:NAME "foo" :TYPE :UNSPECIFIC
or as
:NAME "foo" :TYPE NIL
That, in turn, was because the committee subjective assessment, was
due to the fact that the committee didn't want to confront the
difference between "parse this name using lisp rules" and "parse this
name using file system rules".
The concern was that if someone did
(load "foo")
that on some file systems there might reall _be_ a file with that name
that needed to be loaded, while on others that would be an incomplete
filename. And even then, whether "foo" should be preferred over
"foo.lisp" or "foo.fasl" was not well-agreed upon.
The LispM was up against this issue "for real" because it had multiple
file systems of its own just to start with, and it tended to interact
so commonly with multiple operating systems around it that it just
couldn't ever assume there was a single canonical file system, and it
consequently had a natural separation of these two operations. But
the people on the committee who did not have LispMs seemed (in my
personal assessment) to want to indulge the illusion that there was no
conflict of interest between Lisp's needs and the OS's needs, and they
tended to want to take only parts of what the LispM had worked out
over numerous years, and so they insisted that parse-namestring should
just do the right thing... which it just can't, since there are
multiple right things to choose among. They wanted to believe that the
problems the LispM had were peculiar to the hardware or some other aspect
of the LispM, rather than, as turned out to be the case, peculiar to the
modern heterogeneous networked environment and the need for portability.
... or such was, and is, my belief... Obviously, everyone has a bias
based on their own perspective. I'm sure "stock hardware" (that's
what the LispM crowed called non-LispMs) vendors are represented here
on comp.lang.lisp and can chime in with alternate points of view.
You could, of course, do
(load (make-pathname :name "FOO" :type nil :case :common))
or
(load (make-pathname :name "FOO" :type :unspecific :case :common))
but there is not a portable way (other than maybe to try it inside an
IGNORE-ERRORS form) to know whether the type is permitted to be a
:UNSPECIFIC. Which returns to my original point.
OK, well, lots of good advice here, thanks.
Some suggested that I simply load all the libraries all the time. Of
course this is eminently practical & pragmatic, yet it's a bit
distasteful to me. I have plenty of memory, and the startup time is
not a problem, but it somehow just doesn't seem right.
I had mentioned that (require ..) does almost what I want, and Mr
Tilton favored coloring outside the lines when it comes to
'deprecated', so I investigated that option further.
I'm using openmcl for most things, and sbcl on rare occasions, so I
wanted a solution that works for both of those. I discovered that
openmcl declares a list "*module-search-path*. If I added the path to
my libraries to that list, then openmcl would allow me to (require...)
them exactly the way I wanted. But sbcl didn't have this list.
In addition, I noticed that sbcl allows me to (require...) anything
that I had installed using asdf-install, yet this did not work for
openmcl. This turned out to be because sbcl defines a module provider
function for asdf, but openmcl does not. So, I cloned that function
from sbcl, and added it to my openmcl init file. Now both lisps allow
me to (require...) anything installed by asdf-install.
I noticed further that sbcl provides a module provider function that
provides modules from the sbcl contrib directoriy. So, I cloned that
function (calling it module-provide-user) and modified it to search my
lisp library path. Now sbcl successfully finds my libraries.
Finally, I thought this was a more portable solution than the openmcl
*module-search-path* list, so I got rid of that and added module-
provide-user to my openmcl init file. Now both lisps use module-
provide-user to find my libraries, and both of them allow me to
(require...) things that were installed by asdf-install.
So, I think that works pretty well. If any of my libraries grow too
complex, I'll turn them into .asd systems and let asdf take care of
them, yet still load them with (require 'foo).
Fortunately, sbcl and openmcl name their fasl files differently, so I
can have compiled versions for both lisps in my library path, and each
lisp will choose the correct one if present. If I ever use another
lisp that conflicts with either of these... well, I'll fall off that
bridge when I come to it.
Thanks everyone, for the advice.
Neil
Just kidding.
http://smuglispweeny.blogspot.com/2008/03/tlop-worst-thing-you-can-say-about.html
kenny
--
http://smuglispweeny.blogspot.com/
http://www.theoryyalgebra.com/
"In the morning, hear the Way;
in the evening, die content!"
-- Confucius
From: vanekl
Subject: Re: Managing many small libraries
Date:
Message-ID: <fs4410$o9$1@aioe.org>
Neil Baylis wrote:
> OK, well, lots of good advice here, thanks.
>
> Some suggested that I simply load all the libraries all the time. Of
> course this is eminently practical & pragmatic, yet it's a bit
> distasteful to me. I have plenty of memory, and the startup time is
> not a problem, but it somehow just doesn't seem right.
>
> I had mentioned that (require ..) does almost what I want, and Mr
> Tilton favored coloring outside the lines when it comes to
> 'deprecated', so I investigated that option further.
>
> I'm using openmcl for most things, and sbcl on rare occasions, so I
> wanted a solution that works for both of those. I discovered that
> openmcl declares a list "*module-search-path*. If I added the path to
> my libraries to that list, then openmcl would allow me to (require...)
> them exactly the way I wanted. But sbcl didn't have this list.
>
> In addition, I noticed that sbcl allows me to (require...) anything
> that I had installed using asdf-install, yet this did not work for
> openmcl. This turned out to be because sbcl defines a module provider
> function for asdf, but openmcl does not. So, I cloned that function
> from sbcl, and added it to my openmcl init file. Now both lisps allow
> me to (require...) anything installed by asdf-install.
>
> I noticed further that sbcl provides a module provider function that
> provides modules from the sbcl contrib directoriy. So, I cloned that
> function (calling it module-provide-user) and modified it to search my
> lisp library path. Now sbcl successfully finds my libraries.
>
> Finally, I thought this was a more portable solution than the openmcl
> *module-search-path* list, so I got rid of that and added module-
> provide-user to my openmcl init file. Now both lisps use module-
> provide-user to find my libraries, and both of them allow me to
> (require...) things that were installed by asdf-install.
>
> So, I think that works pretty well. If any of my libraries grow too
> complex, I'll turn them into .asd systems and let asdf take care of
> them, yet still load them with (require 'foo).
>
> Fortunately, sbcl and openmcl name their fasl files differently, so I
> can have compiled versions for both lisps in my library path, and each
> lisp will choose the correct one if present. If I ever use another
> lisp that conflicts with either of these... well, I'll fall off that
> bridge when I come to it.
>
> Thanks everyone, for the advice.
>
> Neil
Before you fall off any bridges, could you post your sbcl
module-provide-user function? I'm curious. TIA
--
it's not unusual that certain numbers (pi, e) are infinite, because most numbers are infinite.
what would be interesting is to know under what number system those two numbers are not infinite.
>
> Before you fall off any bridges, could you post your sbcl
> module-provide-user function? I'm curious. TIA
> --
OK here it is. Where it says #P"/Users/neil/devel/lisplib" is where
you would put the path to your library files.
(defun module-provide-user (name)
(let* ((filesys-name (string-downcase (string name)))
(unadorned-path
(merge-pathnames
(make-pathname
:name filesys-name)
(truename #P"/Users/neil/devel/lisplib")))
(fasl-path (merge-pathnames
(make-pathname :type "fasl")
unadorned-path))
(lisp-path (merge-pathnames (make-pathname :type "lisp")
unadorned-path)))
(when (or (probe-file unadorned-path)
(probe-file fasl-path)
(probe-file lisp-path))
(load unadorned-path)
t)))
And then you need to:
(pushnew 'module-provide-user *module-provider-functions*)
to add it to the list.
Neil
From: Richard M Kreuter
Subject: Re: Managing many small libraries
Date:
Message-ID: <87y789xljf.fsf@progn.net>
Kent M Pitman <······@nhplace.com> writes:
> Richard M Kreuter <·······@progn.net> writes:
>> ···@sevak.isi.edu (Thomas A. Russ) writes:
>>
>> > It is actually best not to include a file type, so that load
>> > will then load compiled binaries if they exist.
>>
>> Not necessarily. From the CLHS entry for LOAD:
>>
>> | If the file specification is not complete and both a source file and
>> | a compiled file exist which might match, then which of those files
>> | ‘load’ selects is implementation-dependent.
>
> It's true it's not strictly well-defined, but I tend to agree with
> tar's advice anyway... In practice I think it works often enough that
> it's the right thing to do. (Kind of like still relying on Newton
> rather than Einstein when working out simple physics problems.)
SBCL at least has a peculiar mechanism for resolving this, which will
signal an error if there's both a compiled file and a source file and
the source is newer than the compiled one, on the assumption that the
fasl file is out of date. So it can sometimes happen that a stray
edit or file system manipulation can mess things up.
--
RmK
From: Kent M Pitman
Subject: Re: Managing many small libraries
Date:
Message-ID: <utzix9cpz.fsf@nhplace.com>
Richard M Kreuter <·······@progn.net> writes:
> Kent M Pitman <······@nhplace.com> writes:
> > Richard M Kreuter <·······@progn.net> writes:
> >> ···@sevak.isi.edu (Thomas A. Russ) writes:
> >>
> >> > It is actually best not to include a file type, so that load
> >> > will then load compiled binaries if they exist.
> >>
> >> Not necessarily. From the CLHS entry for LOAD:
> >>
> >> | If the file specification is not complete and both a source file and
> >> | a compiled file exist which might match, then which of those files
> >> | ‘load’ selects is implementation-dependent.
> >
> > It's true it's not strictly well-defined, but I tend to agree with
> > tar's advice anyway... In practice I think it works often enough that
> > it's the right thing to do. (Kind of like still relying on Newton
> > rather than Einstein when working out simple physics problems.)
>
> SBCL at least has a peculiar mechanism for resolving this, which will
> signal an error if there's both a compiled file and a source file and
> the source is newer than the compiled one, on the assumption that the
> fasl file is out of date. So it can sometimes happen that a stray
> edit or file system manipulation can mess things up.
Interesting. I wouldn't have thought of that way of doing it, but it
makes a certain rational kind of sense and does sound conforming. I'd
argue there really was an opportunity for confusion if there were both
kinds of files there.
It's reasonable to suppose that this should be a "serious condition"
and not an "error" though, since it's vaguely in the same meta-grey
area that things like resource limitations are in, involving things
that aren't semantically wrong but can't be executed due to
possibly-transient circumstances related to the configuration.
Kent M Pitman <······@nhplace.com> writes:
> Richard M Kreuter <·······@progn.net> writes:
> [...]
>> SBCL at least has a peculiar mechanism for resolving this, which will
>> signal an error if there's both a compiled file and a source file and
>> the source is newer than the compiled one, on the assumption that the
>> fasl file is out of date. So it can sometimes happen that a stray
>> edit or file system manipulation can mess things up.
>
> Interesting. I wouldn't have thought of that way of doing it, but it
> makes a certain rational kind of sense and does sound conforming. I'd
> argue there really was an opportunity for confusion if there were both
> kinds of files there.
>
> It's reasonable to suppose that this should be a "serious condition"
> and not an "error" though, since it's vaguely in the same meta-grey
> area that things like resource limitations are in, involving things
> that aren't semantically wrong but can't be executed due to
> possibly-transient circumstances related to the configuration.
And clisp has an option to automatically recompile the .fas if it is
not up-to-date.
See :OBSOLETE-ACTION on
http://clisp.cons.org/impnotes/system-dict.html#loadfile
--
__Pascal Bourguignon__ http://www.informatimago.com/
The world will now reboot. don't bother saving your artefacts.
Richard M Kreuter <·······@progn.net> wrote:
+---------------
| SBCL at least has a peculiar mechanism for resolving this, which will
| signal an error if there's both a compiled file and a source file and
| the source is newer than the compiled one, on the assumption that the
| fasl file is out of date. So it can sometimes happen that a stray
| edit or file system manipulation can mess things up.
+---------------
I like the ":IF-SOURCE-NEWER :COMPILE" extension to LOAD that's
in CMUCL. My own LOAD*[1] wraps a loop around this that compares
the modify times of both the source and compiled versions and
LOADs only if at least one is more recent than a saved timestamp
of the last time LOAD* was run. So you edit one or more of the
current set of LOAD* files, then say "(LOAD*)", and just the
"newer" ones get recompiled/reloaded.[2]
-Rob
[1] Convenience function for use only in exploratory programming
in the REPL. I use ASDF for any productin code.
[2] It was a personal design decision that LOAD* would recompile
source files only if *some* compiled version already existed.
That way LOAD* can be used on a mix of interpreted and compiled
sources.
-----
Rob Warnock <····@rpw3.org>
627 26th Avenue <URL:http://rpw3.org/>
San Mateo, CA 94403 (650)572-2607
On Sun, 23 Mar 2008 21:11:47 -0500, ····@rpw3.org (Rob Warnock) wrote:
> Richard M Kreuter <·······@progn.net> wrote:
> +---------------
> | SBCL at least has a peculiar mechanism for resolving this, which
> | will signal an error if there's both a compiled file and a source
> | file and the source is newer than the compiled one, on the
> | assumption that the fasl file is out of date. So it can sometimes
> | happen that a stray edit or file system manipulation can mess
> | things up.
> +---------------
>
> I like the ":IF-SOURCE-NEWER :COMPILE" extension to LOAD that's in
> CMUCL.
For the sake of completeness, here's how LispWorks does it:
http://www.lispworks.com/documentation/lw50/LWRM/html/lwref-211.htm
Edi.
--
European Common Lisp Meeting, Amsterdam, April 19/20, 2008
http://weitz.de/eclm2008/
Real email: (replace (subseq ·········@agharta.de" 5) "edi")
Neil Baylis wrote:
> Here's my situation: I'm using CL to conduct experiments that involve
> controlling machinery and monitoring test equipment. It's the kind of
> thing LabView is used for extensively, I believe.
>
> I have a growing number of small 'libraries', each consisting of from
> 10 to a few hundred lines of lisp. I build an experiment by writing a
> top-level program in lisp, and it includes a bunch of the libraries.
> Each experiment includes a different bunch of libraries. Some of the
> libraries depend on each other, but mostly they dont.
>
> The functionality I want is something similar to what's provided in C
> via #include.
Er, no.
> Or in Python via import. I'd like to have a single
> location where I could put all my libraries, and a means to easily
> load them as required.
How about one big "utils" library? Or are you concerned with overtaxing
that TRS-80?
>
> To run an experiment, I want to start up lisp and in the REPL just
> (load "experiment"). That will load experiment.lisp or experiment.fasl
> from the current directory, and then either run the experiment
> automatically, or wait for me to invoke a function from the REPL.
>
> As far as I can tell, the canonical way to do this is to make each of
> my libraries into a package, and to define a system for each package.
> So, for each library I would create xxx.lisp and xxx.asd files. I
> would put those files in the asdf central registry. Then each of my
> experiments would begin with a sequence of calls to (asdf:oos...) to
> load the appropriate libraries.
>
> I understand how to do that, but it seems like a lot of work,..
Well then let's go shopping!
>... and
> doubles the number of files I have to deal with.
Hard drives are cheap. Speaking of which, I better get one fast, my RAID
seems to be down to one.
> Have I got it
> basically right, or is there some simplification that I'm missing? It
> looks as though the CL functions require and provide are almost what I
> want.. except not. I notice that the hyperspec says they are
> deprecated, because non-portable. Do people use them?
The great thing about us hard-heads refusing to "move Lisp forward" is
that the only deprecation that matters is the next asteroid, which will
get here before the CL standard gets revisited.
>
> In my fantasy, I wish for a function like (load-lib "libname"). It
> would look for libname in a list of loaded libraries. If libname was
> not in the list, it would search for "libname.fasl" or "libname.lisp"
> in a set of standard directories (that I would initialize in my lisp
> startup file). If if found the library, it would load it, as if I had
> typed (load "libname"). Then, to create a new library, I just drop the
> lisp file into the standard location.
>
> (Note: if I was doing this in C, I would be using a makefile. Perhaps
> this indicates that asdf would be the correct solution for CL after
> all).
Funny how writing a post can help think things through.
My other thought is that from the sound of it you can just load things
at will, they sound kinda small.
ASDF is a pain, I'd use require in your shoes.
kenny
--
http://smuglispweeny.blogspot.com/
http://www.theoryyalgebra.com/
"In the morning, hear the Way;
in the evening, die content!"
-- Confucius
On Thu, 20 Mar 2008 12:18:59 -0700 (PDT), Neil Baylis
<···········@gmail.com> wrote:
> Here's my situation: I'm using CL to conduct experiments that involve
> controlling machinery and monitoring test equipment. It's the kind of
> thing LabView is used for extensively, I believe.
>
> I have a growing number of small 'libraries', each consisting of from
> 10 to a few hundred lines of lisp. I build an experiment by writing a
> top-level program in lisp, and it includes a bunch of the libraries.
> Each experiment includes a different bunch of libraries. Some of the
> libraries depend on each other, but mostly they dont.
>
> The functionality I want is something similar to what's provided in C
> via #include. Or in Python via import. I'd like to have a single
> location where I could put all my libraries, and a means to easily
> load them as required.
>
> To run an experiment, I want to start up lisp and in the REPL just
> (load "experiment"). That will load experiment.lisp or experiment.fasl
> from the current directory, and then either run the experiment
> automatically, or wait for me to invoke a function from the REPL.
>
> As far as I can tell, the canonical way to do this is to make each of
> my libraries into a package, and to define a system for each package.
> So, for each library I would create xxx.lisp and xxx.asd files. I
> would put those files in the asdf central registry. Then each of my
> experiments would begin with a sequence of calls to (asdf:oos...) to
> load the appropriate libraries.
>
> I understand how to do that, but it seems like a lot of work, and
> doubles the number of files I have to deal with. Have I got it
> basically right, or is there some simplification that I'm missing? It
> looks as though the CL functions require and provide are almost what I
> want.. except not. I notice that the hyperspec says they are
> deprecated, because non-portable. Do people use them?
>
> In my fantasy, I wish for a function like (load-lib "libname"). It
> would look for libname in a list of loaded libraries. If libname was
> not in the list, it would search for "libname.fasl" or "libname.lisp"
> in a set of standard directories (that I would initialize in my lisp
> startup file). If if found the library, it would load it, as if I had
> typed (load "libname"). Then, to create a new library, I just drop the
> lisp file into the standard location.
>
> (Note: if I was doing this in C, I would be using a makefile. Perhaps
> this indicates that asdf would be the correct solution for CL after
> all).
Have a look at require and provide forms in your lisp.
They seem what you have asked for.