I have defined two systems, system-a and system-b where system-b
depends on system-a. See sample files at the bottom.
The first time I load-op system-b, system-a are compiled and loaded
and then system-b are compiled and loaded. Then I make a change in
system-a's source file(s) and issue another load-op on system-b. Now
only system-a's source is compiled and loaded. This is not what I
would expect. The changes in system-a might be macros that system-b is
using.
What I would expect is that a load-op on system-b would recompile that
entire system if any of the systems it depends on needs some
recompilation (or loading).
I have tested this with the latest version (1.109) of asdf in SBCL 1.0
as well as older versions in CMUCL and CLISP.
Example files:
system-a.asd:
(defsystem "system-a"
:components ((:file "system-a")))
system-a.lisp:
(eval-when (:compile-toplevel)
(write-line "compiling a"))
(eval-when (:load-toplevel)
(write-line "loading system a"))
(write-line "execute a")
system-b.asd:
(defsystem "system-b"
:depends-on ("system-a")
:components ((:file "system-b")))
system-b.lisp:
(eval-when (:compile-toplevel)
(write-line "compiling b"))
(eval-when (:load-toplevel)
(write-line "loading system b"))
(write-line "execute b")
Consider two asdf systems, "lib" and "app". "app" depends on "lib":
(defsystem "lib"
:components ( ... ))
(defsystem "app"
:depends-on ("lib")
:components ( ... ))
I make a change (in a macro) in "lib", "app" is left untouched. When I then
enter (asdf:oos 'asdf:compile-op "app") asdf will recompile the relevant
files in "lib", but no files in "app". Shouldn't asdf recompile the
entire "app" in this case? Or am I doing something wrong in my system
definitions?
I've done some more investigation and realized that the problem is related
to modules, however since a system is subclass of module it also applies to
systems. Consider this example system:
(defsystem "testdep"
:components ((:module "module-d"
:depends-on ("file-c")
:components ((:file "file-d")))
(:file "file-c" :depends-on ("file-b"))
(:file "file-b" :depends-on ("file-a"))
(:file "file-a")))
On the first compilation of this system the components are compiled in order
file-a, file-b, file-c, file-d, module-d and testdep, just as expected.
Then I make a change in file-a and do another compile-op on testdep. Now
asdf compiles file-a, file-b, file-c, module-d and testdep. Note that
file-d is missing.
Apperently asdf does understands that module-d must be recompiled when
file-c is recompiled, but this of no use as the perform method for
compile-op on a module does nothing. For me it looks like there is a bug in
the method traverse. It should have recursed on the dependant components of
with a forced operation. Unfortunately I have not been able to make this
change, as I still do not fully understand the traverse method.
From: Richard M Kreuter
Subject: Re: asdf depends-on problem
Date:
Message-ID: <87abr8cfaz.fsf@progn.net>
Thomas Bakketun <···········@bakketun.net> writes:
> Consider two asdf systems, "lib" and "app". "app" depends on "lib":
>
> (defsystem "lib"
> :components ( ... ))
>
> (defsystem "app"
> :depends-on ("lib")
> :components ( ... ))
>
> I make a change (in a macro) in "lib", "app" is left untouched. When
> I then enter (asdf:oos 'asdf:compile-op "app") asdf will recompile
> the relevant files in "lib", but no files in "app". Shouldn't asdf
> recompile the entire "app" in this case? Or am I doing something
> wrong in my system definitions?
Probably not. asdf's de facto notion of intersystem dependencies
doesn't do what you want. I say more about this below.
> I've done some more investigation and realized that the problem is
> related to modules, however since a system is subclass of module it
> also applies to systems. Consider this example system:
>
> (defsystem "testdep"
> :components ((:module "module-d"
> :depends-on ("file-c")
> :components ((:file "file-d")))
> (:file "file-c" :depends-on ("file-b"))
> (:file "file-b" :depends-on ("file-a"))
> (:file "file-a")))
>
> On the first compilation of this system the components are compiled
> in order file-a, file-b, file-c, file-d, module-d and testdep, just
> as expected.
>
> Then I make a change in file-a and do another compile-op on
> testdep. Now asdf compiles file-a, file-b, file-c, module-d and
> testdep. Note that file-d is missing.
>
> Apperently asdf does understands that module-d must be recompiled
> when file-c is recompiled, but this of no use as the perform method
> for compile-op on a module does nothing. For me it looks like there
> is a bug in the method traverse. It should have recursed on the
> dependant components of with a forced operation. Unfortunately I
> have not been able to make this change, as I still do not fully
> understand the traverse method.
IIRC, implementation-wise, the only conditions that induce an
operation on a component are (a) if the operation's outputs for that
component are not found, (b) if the operation/component pair is a
dependent of some other pending operation/component pair.
For your latter example, understand that "file-d" does not depend on
"file-c"; it's "module-d" that depends on "file-c". If you want
"file-d" to depend on one of those other files, don't put it in a
separate module, and declare its dependencies.
For your former example, it happens that a system-level :DEPENDS-ON
argument to DEFSYSTEM does not associate dependencies between the
components of the dependent system ("app", in your case) and the
dependee ("lib"). Consequently, when asdf constructs its execution
plan, even though that plan will include recompilation of some files
in the dependee system, the information isn't there to induce
compilation of the files in the dependent system.
Now, while this situation is not what you want at present and is
arguably the wrong thing, note that intersystem dependencies introduce
a bunch of problems with no obviously right solutions. Even your
two-system model raises some of these problems: while you want
(asdf:operate 'asdf:compile-op "app")
to recompile "lib" and "app" in case any of "lib"'s fasls are out of
date, what should the consequences of the following be:
(asdf:operate 'asdf:compile-op "app")
;; Some modifications to "lib"'s files
(asdf:operate 'asdf:compile-op "lib")
In general, arbitrary changes to files in "lib" can cause "app" not to
work properly; for example, maybe you've altered or removed some
functions from "lib"'s public interface. Should recompilation of
"lib" induce recompilation of "app"? How should the backward
dependency information be represented and maintained? What about Lisp
systems in the file system but not registered in the current Lisp
image that depend on "lib"? Should asdf aggressively try to find such
systems and recompile them immediately? What if there are two
mutually incompatible Lisp systems (i.e., they can't both be processed
in the same image, say, because they manipulate a package in
incompatible ways) on disk that would have to be recompiled as a
result?
One approach, developed by Andreas Fuchs in the system called
asdf-dependency-grovel, has been to automatically merge several asdf
systems into one and to maintain very fine-grained dependencies among
components in these merged systems in order to get compilation
approximately at the times you say you want. I'm not quite certain of
the precise semantics of asdf-dependency-grovel, so I don't know
whether and how it answers all questions about intersystem
dependencies.
Finally, if you'll permit a bit of editorializing, I'd suggest not
expecting that asdf will match your intuitions about anything in
particular. IMO, asdf is one of those programs for which many of the
details outside of its core ideas are defined more by their
implementation than by a complete or explicable design.
--
RmK
Richard M Kreuter wrote:
> Now, while this situation is not what you want at present and is
> arguably the wrong thing, note that intersystem dependencies introduce
> a bunch of problems with no obviously right solutions. Even your
> two-system model raises some of these problems: while you want
>
> (asdf:operate 'asdf:compile-op "app")
>
> to recompile "lib" and "app" in case any of "lib"'s fasls are out of
> date, what should the consequences of the following be:
>
> (asdf:operate 'asdf:compile-op "app")
> ;; Some modifications to "lib"'s files
> (asdf:operate 'asdf:compile-op "lib")
> Should recompilation of "lib" induce recompilation of "app"?
I would expect not.
> How should the backward dependency information be represented and
> maintained?
There is no need to maintain this information as it can be discovered when
needed.
> What about Lisp systems in the file system but not registered in the
> current Lisp image that depend on "lib"? Should asdf aggressively try to
> find such systems and recompile them immediately?
Probably not a good idea.
> What if there are two mutually incompatible Lisp systems
Also this could trigger massive compilation of systems which, where some
might be known to be broken.
However I did not suggest this kind behaviour. Changes in "lib" should only
trigger recompilation of "app" when a recompilation of is "app" requested.
In my previous post I concluded that this could be solved by detecting that
something in "lib" is recompiled and then force recompilation of "app", but
this will fail when doing:
(asdf:operate 'asdf:compile-op "app")
;; update lib
(asdf:operate 'asdf:compile-op "lib")
(asdf:operate 'asdf:compile-op "app")
Since "lib" is up-to-date, compiling app will not trigger any recompilation
of "lib". compile-op on "app" is considered done:
(asdf:operation-done-p (make-instance 'asdf:compile-op)
(asdf:find-system "app")))
So operation-done-p should be made recurisve on dependencies, but this
doesn't not seem to fit with asdf's design. Dependency checking is allready
done elsewere. If operation-done-p returned when the operation was done
(write time of latest modifed output-file), then the dependency checking
code would be able to dectect when an component is out of date.
> One approach, developed by Andreas Fuchs in the system called
> asdf-dependency-grovel, has been to automatically merge several asdf
> systems into one and to maintain very fine-grained dependencies among
> components in these merged systems in order to get compilation
> approximately at the times you say you want.
Sounds like this could solve the problem, all thought more of a work around
than a real solution.