From: Sam Steingold
Subject: fasl-file-valid-p
Date: 
Message-ID: <umzx04k8m.fsf@gnu.org>
Build tools, like defsystem or asdf, have to determine whether a file
needs to be recompiled.  Obviously, when the .fasl is older than the
.lisp, recompilation is in order.

Alas, there are other situations when this might be necessary, e.g.,
when the implementation changes the FASL format or when two
implementations use the same name for their FASL files
(are there such cases?)


Proposal:

function (FASL-FILE-VALID-P file-name) => valid-p
return T if the file is a valid FASL file for this implementation and
NIL otherwise.
E.g.,
(fasl-file-valid-p "foo.lisp") ==> NIL
(fasl-file-valid-p (compile-file "foo.lisp")) ==> T

the trivial implementation:

(defun fasl-file-valid-p (file-name)
  (not (nth-value 1 (ignore-errors (load file-name)))))

is wrong because,

1. LOAD may fail even though the file is valid:
   even when "foo.lisp" contains "(error)",
      (fasl-file-valid-p (compile-file "foo.lisp"))
   should still return T

2. this is not side-effect-free, i.e., this may define new functions and
   macros (or, worse yet, redefine some existing functions and macros or
   execute some malicious code).

Cost to implementors:  something like this:
#+clisp
(defun fasl-file-valid-p (file-name)
  (with-open-file (in file-name :direction :input :if-does-not-exist nil)
    (and in
         (let ((line (read-line in nil nil)))
           (and line
                (string= line "(SYSTEM::VERSION " :start1 0
                         :end1 #.(length "(SYSTEM::VERSION "))
                (null (nth-value 1 (ignore-errors
                                     (eval (read-from-string line))))))))))))


Comments?

-- 
Sam Steingold (http://www.podval.org/~sds) running w2k
<http://www.camera.org> <http://www.iris.org.il> <http://www.memri.org/>
<http://www.mideasttruth.com/> <http://www.honestreporting.com>
Binaries die but source code lives forever.

From: Daniel Barlow
Subject: Re: fasl-file-valid-p
Date: 
Message-ID: <87zn10ldej.fsf@noetbook.telent.net>
Sam Steingold <···@gnu.org> writes:

> Alas, there are other situations when this might be necessary, e.g.,
> when the implementation changes the FASL format or when two
> implementations use the same name for their FASL files
> (are there such cases?)

".fasl", surprisingly enough.  Used by both ACL and SBCL.

> function (FASL-FILE-VALID-P file-name) => valid-p
> return T if the file is a valid FASL file for this implementation and
> NIL otherwise.
> E.g.,
> (fasl-file-valid-p "foo.lisp") ==> NIL
> (fasl-file-valid-p (compile-file "foo.lisp")) ==> T

> Comments?

I think it's a good idea.  

1) I'd like to see its behaviour in exceptional situations (e.g. the
file is unreadable/does not exist/etc) defined.

2) The word "fasl" doesn't appear in the ANSI glossary, but the term
"compiled file" does, so I would suggest that COMPILED-FILE-P or
something similar (i.e. using standard terminology) is a better name.


-dan

-- 
"please make sure that the person is your friend before you confirm"
From: Sam Steingold
Subject: Re: fasl-file-valid-p
Date: 
Message-ID: <uact04ho8.fsf@gnu.org>
> * Daniel Barlow <···@gryrag.arg> [2004-11-29 18:26:44 +0000]:
>
> Sam Steingold <···@gnu.org> writes:
>
>> Alas, there are other situations when this might be necessary, e.g.,
>> when the implementation changes the FASL format or when two
>> implementations use the same name for their FASL files
>> (are there such cases?)
>
> ".fasl", surprisingly enough.  Used by both ACL and SBCL.

yuk.

>> function (FASL-FILE-VALID-P file-name) => valid-p
>> return T if the file is a valid FASL file for this implementation and
>> NIL otherwise.
>> E.g.,
>> (fasl-file-valid-p "foo.lisp") ==> NIL
>> (fasl-file-valid-p (compile-file "foo.lisp")) ==> T
>
>> Comments?
>
> I think it's a good idea.  
>
> 1) I'd like to see its behaviour in exceptional situations (e.g. the
> file is unreadable/does not exist/etc) defined.

expected use is

(defun file-newer (f0 f1)
  "Return T if the first arg is newer than the second.
Non-existent files are assumed to be VERY old."
  (flet ((fwd (ff) (or (ignore-errors (file-write-date ff)) 0)))
    (> (fwd f0) (fwd f1))))

      (let ((cf (compile-file-pathname file)))
        (when (or (file-newer file cf)
                  (not (compiled-file-p cf)))
          (compile-file file)))

so I'd say that COMPILED-FILE-P should return T only if the file (and
the path to it) exists, is readable, and the contents is good, and NIL
in all other cases.
the only exceptional situation is when the argument is not a pathname
designator, in which case a type-error is signaled.
(and one could make a case that since a non-pathname designator is not
a compiled file, then COMPILED-FILE-P should just return NIL and not
signal)

> 2) The word "fasl" doesn't appear in the ANSI glossary, but the term
> "compiled file" does, so I would suggest that COMPILED-FILE-P or
> something similar (i.e. using standard terminology) is a better name.

OK.

-- 
Sam Steingold (http://www.podval.org/~sds) running w2k
<http://www.camera.org> <http://www.iris.org.il> <http://www.memri.org/>
<http://www.mideasttruth.com/> <http://www.honestreporting.com>
Lisp is a way of life.  C is a way of death.
From: Svein Ove Aas
Subject: Re: fasl-file-valid-p
Date: 
Message-ID: <cofsmd$6j4$1@services.kq.no>
Sam Steingold wrote:

> Build tools, like defsystem or asdf, have to determine whether a file
> needs to be recompiled.  Obviously, when the .fasl is older than the
> .lisp, recompilation is in order.
> 
> Alas, there are other situations when this might be necessary, e.g.,
> when the implementation changes the FASL format or when two
> implementations use the same name for their FASL files
> (are there such cases?)
> 

I've run into one situation where this happens that's particularily
annoying. Sometimes an asdf-driven compilation will fail; when this
happens, it tends to leave behind invalid .fasl files. If the reason for
its failing was in another file, it doesn't even try to recompile it after
it's fixed.

Just food for thought.
From: Sam Steingold
Subject: Re: fasl-file-valid-p
Date: 
Message-ID: <u653oiftl.fsf@gnu.org>
> * Svein Ove Aas <·········@nnf.ab> [2004-11-29 20:18:34 +0100]:
>
> Sam Steingold wrote:
>
>> Build tools, like defsystem or asdf, have to determine whether a file
>> needs to be recompiled.  Obviously, when the .fasl is older than the
>> .lisp, recompilation is in order.
>> 
>> Alas, there are other situations when this might be necessary, e.g.,
>> when the implementation changes the FASL format or when two
>> implementations use the same name for their FASL files
>> (are there such cases?)
>> 
>
> I've run into one situation where this happens that's particularily
> annoying. Sometimes an asdf-driven compilation will fail; when this
> happens, it tends to leave behind invalid .fasl files. If the reason
> for its failing was in another file, it doesn't even try to recompile
> it after it's fixed.

I have seen this happen only when the CL implementation crashed.


-- 
Sam Steingold (http://www.podval.org/~sds) running w2k
<http://www.camera.org> <http://www.iris.org.il> <http://www.memri.org/>
<http://www.mideasttruth.com/> <http://www.honestreporting.com>
I don't want to be young again, I just don't want to get any older.
From: Nikodemus Siivola
Subject: Re: fasl-file-valid-p
Date: 
Message-ID: <cokde0$i6anj$3@midnight.cs.hut.fi>
Sam Steingold <···@gnu.org> wrote:

> Proposal:

> function (FASL-FILE-VALID-P file-name) => valid-p
> return T if the file is a valid FASL file for this implementation and
> NIL otherwise.
> E.g.,
> (fasl-file-valid-p "foo.lisp") ==> NIL
> (fasl-file-valid-p (compile-file "foo.lisp")) ==> T

1. I agree that compiled-file-p is a better name.

2. I'd make the description a bit more accurate -- and permissive:

 "Return T if the file appears to be a compiled file for the
 implementation. Implementations are required to inspect the contents
 (checking eg. pathname type is not sufficient), but the accuracy
 of the inspection is not specified, but is expected to be a conservetive
 approximation with regards to eg. file format changes between versions."

3. Additional specification for error-signalling from LOAD would be a good
thing: if the operation fails due to an error in the format of the
compiled file, an error of eg. type COMPILED-FILE-FORMAT-ERROR should be
signalled.

Cheers,

 -- Nikodemus
From: Martin Ginkel
Subject: Re: fasl-file-valid-p
Date: 
Message-ID: <coka41$28aj$1@gwdu112.gwdg.de>
Sam Steingold wrote:
> Build tools, like defsystem or asdf, have to determine whether a file
> needs to be recompiled.  Obviously, when the .fasl is older than the
> .lisp, recompilation is in order.
> 
> Alas, there are other situations when this might be necessary, e.g.,
> when the implementation changes the FASL format or when two
> implementations use the same name for their FASL files
> (are there such cases?)
> 
Your proposal is not bad. But in your example I see a different problem:

Asdf should IMHO support a binary tree different from the source path.

My current practice with defsystem
(and a reason still not switching to asdf) is to create logical paths
"xxx:bin;" automatically
based on (lisp-implementation-type) (lisp-implementation-version) 
(software-type) and (machine-version), always filtered for invalid chars
and putting fasls and images there. This path is used as the
:binary-pathname.
This mechanism is better than the often seen:
#+(and :allegro (version>= 6 0)) "bin-acl6"
#+:clisp "bin-clisp"
because it automatically works with new versions. The only drawback is
a loooong directory name.

If I compile on a different lisp, the logical pathname translation 
change and (my?) defsystem creates new binary directories, and will
find no fasls, leading to a new compile for this version.
This system works for release changes of Lisp as well, so I
currently test some source trees with 4 different lisps, keeping all
binaries and have no more recompiles as necessary.

	CU
	Martin
From: Daniel Barlow
Subject: Re: fasl-file-valid-p
Date: 
Message-ID: <87brdejhnr.fsf@noetbook.telent.net>
Martin Ginkel <·······@mpi-magdeburg.mpg.de> writes:

> Asdf should IMHO support a binary tree different from the source path.

The choice of where to put binaries is a matter for the code user, not
for the code author (in the general case that the two are not the same
person), and the code user is not expected to have to edit the .asd
file - that will cause merge conflicts if/when he upgrades to a new
version from the author.  So, adding a defsystem keyword option for
this would not really be a great solution.
http://ww.telent.net/diary/2004/11/#8.82988 explains this in more
detail.

Note that as a code user you may add a asdf:output-files :around
method (e.g. in your Lisp startup files) to achieve the effect of
separate directories for fasls.  The documentation on this is not all
that great (at least, if _I_ can't find where I wrote that it's
permitted, I suspect that others may also have trouble) but I assure
you that it is part of the "supported" interface.

(http://constantly.at/lisp/asdf/Creating-new-operations.html has words
in the right order that hint that this is usual practice, through the
formatting appears to be slightly screwed.  It should say something
more like

  An operation must provide methods for the following generic
  functions when invoked with an object of type source-file:

    * output-files
    * perform 
        The perform method must call output-files to find out where to put 
        its files, because the user is allowed to override output-files for 
        local policy 
    * explain
    * operation-done-p, if you don't like the default one 

Other than that, recourse is to the source)


-dan

-- 
"please make sure that the person is your friend before you confirm"
From: Peter Seibel
Subject: Re: fasl-file-valid-p
Date: 
Message-ID: <m34qj6hr8w.fsf@javamonkey.com>
Daniel Barlow <···@telent.net> writes:

> Martin Ginkel <·······@mpi-magdeburg.mpg.de> writes:
>
>> Asdf should IMHO support a binary tree different from the source path.
>
> The choice of where to put binaries is a matter for the code user,
> not for the code author (in the general case that the two are not
> the same person), and the code user is not expected to have to edit
> the .asd file - that will cause merge conflicts if/when he upgrades
> to a new version from the author. So, adding a defsystem keyword
> option for this would not really be a great solution.
> http://ww.telent.net/diary/2004/11/#8.82988 explains this in more
> detail.

Yes. That's all rightous stuff.

> Note that as a code user you may add a asdf:output-files :around
> method (e.g. in your Lisp startup files) to achieve the effect of
> separate directories for fasls.

Is there some in-principle reason that ASDF can't make this easier out
of the box for the code user? Or is it just a matter of no one's
submitted a patch? I haven't looked at it closely but maybe there's
some way to provide a variable or two for the code user to set that
says, put my fasls over here, rather than requiring the code user to
have to write new methods. Would you be ammenable to such a patch were
someone to provide it. Or does that go against some design principle
of ASDF, such as that all code-user-level customization of it's
behavior should be done by writing new methods?

-Peter

-- 
Peter Seibel                                      ·····@javamonkey.com

         Lisp is the red pill. -- John Fraser, comp.lang.lisp
From: Martin Ginkel
Subject: Re: fasl-file-valid-p
Date: 
Message-ID: <col5tq$8rn$1@news.boerde.de>
Peter Seibel wrote:
> 
> Is there some in-principle reason that ASDF can't make this easier out
> of the box for the code user? Or is it just a matter of no one's
> submitted a patch? I haven't looked at it closely but maybe there's
> some way to provide a variable or two for the code user to set that
> says, put my fasls over here, rather than requiring the code user to
> have to write new methods. Would you be ammenable to such a patch were
> someone to provide it. Or does that go against some design principle
> of ASDF, such as that all code-user-level customization of it's
> behavior should be done by writing new methods?

I habe seen a user-side fix in message <···············@my.nada.kth.se>
but i still see that in many cases that package authors
want or have to configure the build directories for his package.
Without messing with the system-wide methods of asdf, or requiring a 
global map of all source and binary directories.
CL has logical pathnames for that.

	CU
	Martin
From: Daniel Barlow
Subject: Re: fasl-file-valid-p
Date: 
Message-ID: <87vfbliwnr.fsf@noetbook.telent.net>
Peter Seibel <·····@javamonkey.com> writes:

> Is there some in-principle reason that ASDF can't make this easier out
> of the box for the code user? Or is it just a matter of no one's
> submitted a patch? I haven't looked at it closely but maybe there's

More the latter.  I certainly have no objection to making it easier,
if it means I spend less time answering this question ;-)

It would be nice if the patch were moderately general and thought
about issues beyond "fasls for cl source files" : intermediate files
geenrated by tools like ffigen, or foreign glue files generated by 
C compilers.  I'm not saying it has to cater for all cases, but I
wouldn't like to close the door on them either.


-dan

-- 
"please make sure that the person is your friend before you confirm"
From: Christophe Rhodes
Subject: Re: fasl-file-valid-p
Date: 
Message-ID: <sqacsxvd3n.fsf@cam.ac.uk>
Peter Seibel <·····@javamonkey.com> writes:

> Daniel Barlow <···@telent.net> writes:
>
>> Note that as a code user you may add a asdf:output-files :around
>> method (e.g. in your Lisp startup files) to achieve the effect of
>> separate directories for fasls.
>
> Is there some in-principle reason that ASDF can't make this easier out
> of the box for the code user? Or is it just a matter of no one's
> submitted a patch?

One reason that it's hard -- but that it ought to be provided by asdf
more completely than just define a method -- is that there are really
three parties: code author, user, and sysadmin.  (Bear with me.)

On at least Debian GNU/Linux, a number of Lisp packages are available
for site-wide installation through asdf; the lisp management software
(common-lisp-controller) arranges for lisp sources to be kept in
/usr/share/common-lisp/source/, while fasls end up in per-user,
per-implementation directories under /var/.

To achieve this, well, it defines an :around method on output-files,
checking to see if the source file is under /usr/share, and arranging
the appropriate return if so.  So far, so good -- except that now the
user can no longer define an :around method on output-files himself.

It's possible that output-files needs a custom method combination, or
some other mechanism, to allow both user and sysadmin a chance at
customization.  No-one's really bitten the bullet on this one yet, to
my knowledge.

Christophe
From: Rahul Jain
Subject: Re: fasl-file-valid-p
Date: 
Message-ID: <873by2h9ma.fsf@nyct.net>
Christophe Rhodes <·····@cam.ac.uk> writes:

> To achieve this, well, it defines an :around method on output-files,
> checking to see if the source file is under /usr/share, and arranging
> the appropriate return if so.  So far, so good -- except that now the
> user can no longer define an :around method on output-files himself.

Looks like no one's responded to this yet...

Debian's clc does this just fine with clc-register-user-package, and it
needs to be handled by clc's output-files :around method because it
needs to create directories for the output files for the various
implementations that are installed.

-- 
Rahul Jain
·····@nyct.net
Professional Software Developer, Amateur Quantum Mechanicist
From: Martin Ginkel
Subject: Re: fasl-file-valid-p
Date: 
Message-ID: <col4lc$8qo$1@news.boerde.de>
Daniel Barlow wrote:
> The choice of where to put binaries is a matter for the code user, not
> for the code author (in the general case that the two are not the same
> person), and the code user is not expected to have to edit the .asd
> file - that will cause merge conflicts if/when he upgrades to a new
> version from the author.  So, adding a defsystem keyword option for
> this would not really be a great solution.
> http://ww.telent.net/diary/2004/11/#8.82988 explains this in more
> detail.

I disagree with this argument. It is unnecessary to change the asd, if
the package author builds his stuff with configurable (logical) 
pathnames. In my systems there is a config-file, setting up logical 
pathnames. This is generated by a configure-script, where the user has
all choices. But with the current asdf, it is impossible for the package
author to create better defaults than given by asdf.

One problem with system-definition stuff is, it is rather difficult to
customize because it is the first thing to load.

> Note that as a code user you may add a asdf:output-files :around
> method (e.g. in your Lisp startup files) to achieve the effect of
> separate directories for fasls.  

Bad choice for the author. And he still gets the bug-reports about
fasl-version mismatches.
On the other hand: How many *users* override methods in the guts of
asdf to change the binary dir?
Also: this method then applies to all asdf packages on the
computer-system in the same,
and may break the assumptions of package authors about the directory 
layout of their stuff. (If they expect the stuff in the default places 
for copy/install operations).


> (http://constantly.at/lisp/asdf/Creating-new-operations.html has words
> in the right order that hint that this is usual practice, through the
> formatting appears to be slightly screwed.  It should say something
> more like
> 
>   An operation must provide methods for the following generic
>   functions when invoked with an object of type source-file:
> 
>     * output-files
>     * perform 
>         The perform method must call output-files to find out where to put 
>         its files, because the user is allowed to override output-files for 
>         local policy 
>     * explain
>     * operation-done-p, if you don't like the default one 

Hmm: different operations then compile and load files are rarely used.
Asdf is used for maintaining Lisp-Packages, I don't expect it to act as 
a general replacement to make.
I rather started writing new file types (for Corba-idl and others) .


	CU
	Martin Ginkel
From: Daniel Barlow
Subject: Re: fasl-file-valid-p
Date: 
Message-ID: <87pt1tiwcv.fsf@noetbook.telent.net>
Martin Ginkel <·············@epost.de> writes:

> Daniel Barlow wrote:
>> http://ww.telent.net/diary/2004/11/#8.82988 explains this in more
>> detail.
>
> I disagree with this argument. It is unnecessary to change the asd, if
> the package author builds his stuff with configurable (logical)
> pathnames. In my systems there is a config-file, setting up logical
> pathnames. This is generated by a configure-script, where the user has
> all choices. But with the current asdf, it is impossible for the package
> author to create better defaults than given by asdf.

Nothing stops you from using logical pathnames with asdf.  I must
admit that I personally am much happier that I no longer /have/ to set
up a LP host for every third-party package I want to use these days,
but the option is there if you want it.  You can provide an absolute
(logical) pathname at the system level with the :pathname option,
and two translations for it so that source and binary files are mapped
to different places.


-dan

-- 
"please make sure that the person is your friend before you confirm"
From: Pascal Bourguignon
Subject: Re: fasl-file-valid-p
Date: 
Message-ID: <87pt1tc5b7.fsf@thalassa.informatimago.com>
Daniel Barlow <···@telent.net> writes:

> Martin Ginkel <·············@epost.de> writes:
> 
> > Daniel Barlow wrote:
> >> http://ww.telent.net/diary/2004/11/#8.82988 explains this in more
> >> detail.
> >
> > I disagree with this argument. It is unnecessary to change the asd, if
> > the package author builds his stuff with configurable (logical)
> > pathnames. In my systems there is a config-file, setting up logical
> > pathnames. This is generated by a configure-script, where the user has
> > all choices. But with the current asdf, it is impossible for the package
> > author to create better defaults than given by asdf.
> 
> Nothing stops you from using logical pathnames with asdf.  I must
> admit that I personally am much happier that I no longer /have/ to set
> up a LP host for every third-party package I want to use these days,
> but the option is there if you want it.  You can provide an absolute
> (logical) pathname at the system level with the :pathname option,
> and two translations for it so that source and binary files are mapped
> to different places.

I set up a LP at the root of where I install third-party packages.
So I can load: "PACKAGES:COM;INFORMATIMAGO;COMMON-LISP;INVOICE"
or:            "PACKAGES:NET;SOURCEFORGE;CLOCC;CLOCC;CLOCC"
or:            "PACKAGES:NET;SOURCEFORGE;CL-PDB;CL-PDB;SRC;PDB.ASD"
or:            "PACKAGES:COM;HP;ZEBU;ZEBU-INIT.LISP"
etc...

Of course, it would be easier if all followed the same directory
structure and the same conventions, but at least I can reach them
uniformily.  In any case, since they each need LP or some other
specific incantation, I have in addition a file for each in "LOADER:"
LP so I can load theme easily:
    (load "LOADERS:CLOCC")
    (load "LOADERS:ZEBU")
etc..


-- 
__Pascal Bourguignon__                     http://www.informatimago.com/
The world will now reboot; don't bother saving your artefacts.