From: Vladimir Zolotykh
Subject: Modules vs CONCATENATE-SYSTEM
Date: 
Message-ID: <20060414125410.63f1a459.gsmith@eurocom.od.ua>
Let me consider the following simple session (Using ACL80)

  $ cat a.cl
  (in-package :cl-user)
  (defpackage :a-package
    (:use #:cl #:excl)
    (:export #:a-print))
  (in-package :a-package)
  (eval-when (:load-toplevel :execute)
    (push :a-package *features*))
  (defun a-print ()
    (print "a-print"))
  # Compile a.cl
  $ mlisp -C a.cl -kill
  $ cat b.cl
  (in-package :cl-user)
  (defpackage :b-package
    (:use #:cl #:excl))
  (in-package :b-package)
  (defun b-print ()
    (a-package:a-print))
  # Load previously compiled a.fasl then compile b.cl 
  $ mlisp -L a.fasl -C b.cl -kill
  $ mlisp
  cl-user(1): :ld b
  ; Fast loading /home/vlz/log/2006/04/14/b.fasl
  Error: No package exists of name a-package.
    [condition type: package-error]

It is the above error that I'd like to ask about.

It seems that b.fasl doesn't contain package A-PACKAGE. Is this
conventional? If so, would you mind telling me what I should read to
familiarise myself a bit more with the subject? I've skimmed through
the "3.2 Compilation" in HyperSpec, unfortunately this didn't improve
my undertanding of the subject much.

Another question. Considering the situation in the above example, I
needed the package A-PACKAGE. How I should establish its presence?
The approaches which would come into my mind at present:

  1) cat a.fasl b.fasl > bb.fasl

     or in a case with more complicated dependencies to use ASDF
     assuming that it (ASDF that is) can do something like
     CONCATENATE-SYSTEM

  2) make a module from a.cl ,e.g. adding (provide :a-package) to
     a.cl, then compiling the file and copying the result somewhere,
     so that (require :a-package) can find it.  And adding

     (eval-when (:compile-toplevel :load-toplevel :execute)
       (require :a-package))

     to b.cl

Which would be better? Or maybe there are better solutions?


-- 
Vladimir Zolotykh

From: Peter Seibel
Subject: Re: Modules vs CONCATENATE-SYSTEM
Date: 
Message-ID: <m2acao1dyf.fsf@gigamonkeys.com>
Vladimir Zolotykh <······@eurocom.od.ua> writes:

>   1) cat a.fasl b.fasl > bb.fasl
>
>      or in a case with more complicated dependencies to use ASDF
>      assuming that it (ASDF that is) can do something like
>      CONCATENATE-SYSTEM

FWIW, here's some code I wrote the other day to do just that thing:

  (defclass concat-fasls (operation)
    ((concatenator :initarg :concatenator :initform #'(lambda (f) (format t "~&~a~%" f)))))

  (defun build-one-fasl (system &optional (output (output-fasl system)))
    (with-open-file (out output :direction :output :if-exists :supersede :element-type '(unsigned-byte 8))
      (flet ((add-file (file)
               (format t "~&;; Appending ~a~%" (truename file))
               (with-open-file (in file :element-type '(unsigned-byte 8))
                 (loop with buffer = (make-array 4096 :element-type '(unsigned-byte 8))
                    for read = (read-sequence buffer in)
                    while (plusp read) do
                      (write-sequence buffer out :end read)))))
        (oos 'concat-fasls :triple-store :concatenator #'add-file))))

  (defun output-fasl (system-name)
    (make-pathname :name (string-downcase system-name) :type (fasl-type)))

  (defun fasl-type ()
    (pathname-type (compile-file-pathname (make-pathname :type "lisp"))))

  (defmethod perform ((o concat-fasls) (c cl-source-file))
    (with-slots (concatenator) o
      (map nil concatenator (input-files o c))))

  (defmethod perform ((operation concat-fasls) (c static-file)) nil)

  (defmethod operation-done-p ((operation concat-fasls) (c cl-source-file)) nil)
  (defmethod operation-done-p ((operation concat-fasls) (c static-file)) t)

  (defmethod output-files ((o operation) (c component)) nil)

  (defmethod component-depends-on ((operation concat-fasls) (c cl-source-file))
    (cons (list 'compile-op (component-name c))
          (call-next-method)))

  ;; Nicked from load-source-op more or less.
  (defmethod component-depends-on ((o concat-fasls) (c component))
    (let ((what-would-load-op-do (cdr (assoc 'load-op
                                             (slot-value c 'asdf::in-order-to)))))
      (mapcar #'(lambda (dep)
                  (if (eql (car dep) 'load-op)
                      (cons 'concat-fasls (cdr dep))
                      dep))
              what-would-load-op-do)))


-Peter

-- 
Peter Seibel           * ·····@gigamonkeys.com
Gigamonkeys Consulting * http://www.gigamonkeys.com/
Practical Common Lisp  * http://www.gigamonkeys.com/book/
From: Vladimir Zolotykh
Subject: Re: Modules vs CONCATENATE-SYSTEM
Date: 
Message-ID: <20060415181207.6d632778.gsmith@eurocom.od.ua>
Thanks for code. The BUILD-ONE-FASL function is exactly what I've been
looking for. I did the kind of CONCATENATE-OP myself, however it would
open target file many times (in :append mode). I didn't know how to
avoid this because I was locked in the cell of ASDF:OPERATION concept
which is called as many times as are files in the system, the idea to
wrap (oos 'concatenate-op ...)  in a function and thus establishing an
open file for the whole operation didn't come into my mind however
simple it may not seem for me now.

Would you mind explaining why are the following methods needed in your
code? I mean

>   (defmethod perform ((operation concat-fasls) (c static-file)) nil)
[snip]
>   (defmethod operation-done-p ((operation concat-fasls) (c static-file)) t)
> 
>   (defmethod output-files ((o operation) (c component)) nil)


-- 
Vladimir Zolotykh
From: Peter Seibel
Subject: Re: Modules vs CONCATENATE-SYSTEM
Date: 
Message-ID: <m2zmim5pq9.fsf@gigamonkeys.com>
Vladimir Zolotykh <······@eurocom.od.ua> writes:

> Thanks for code. The BUILD-ONE-FASL function is exactly what I've been
> looking for. I did the kind of CONCATENATE-OP myself, however it would
> open target file many times (in :append mode). I didn't know how to
> avoid this because I was locked in the cell of ASDF:OPERATION concept
> which is called as many times as are files in the system, the idea to
> wrap (oos 'concatenate-op ...)  in a function and thus establishing an
> open file for the whole operation didn't come into my mind however
> simple it may not seem for me now.
>
> Would you mind explaining why are the following methods needed in your
> code? I mean
>
>>   (defmethod perform ((operation concat-fasls) (c static-file)) nil)

Basically, when building a monolithic .fasl we don't need to do
anything about static files--they're not going to be put into the
.fasl. Of course that means that if we have code that does something
like:

  (with-open-file (in (merge-pathnames "some-static-file.dat" *load-truename*)) ...)

the .fasl will need to be put in a directory with "some-static-file.dat".

>>   (defmethod operation-done-p ((operation concat-fasls) (c static-file)) t)

Likewise, nothing to do.

>>   (defmethod output-files ((o operation) (c component)) nil)

Uh, I'm not sure what this is for--must have been cut-n-pasted since
it's not specialized on my new class.

-Peter

-- 
Peter Seibel           * ·····@gigamonkeys.com
Gigamonkeys Consulting * http://www.gigamonkeys.com/
Practical Common Lisp  * http://www.gigamonkeys.com/book/
From: Peter Seibel
Subject: Re: Modules vs CONCATENATE-SYSTEM
Date: 
Message-ID: <m2vet95yic.fsf@gigamonkeys.com>
Vladimir Zolotykh <······@eurocom.od.ua> writes:

> Peter,
>
> I'm humbly begging your patience's pardon and hoping it is not gonna
> regret answering my first question about ASDF 'cos of being swamped by
> subsequent questions ;^)
>
> Here is the quote from http://constantly.at/lisp/asdf/
>
>   "asdf's pre-defined operations are in no way "privileged", but it is
>    requested that developers never use the asdf package for operations
>    they develop themselves. The rationale for this rule is that we
>    don't want to establish a "global asdf operation name registry",
>    but also want to avoid name clashes."
>
> (Section Creating New Operations). How would you explain it? If we
> were to create a new operation like CONCAT-FASLS we had to use symbols
> from ASDF package like PERFORM, OPERATION-DONE-P etc. Had we not? Or
> the quotation above means something else?

It means something else. They're just saying that the symbol
CONCAT-FASLS should not be interned in the ASDF package. I don't
recall whether I included the IN-PACKAGE expression in the code I
posted, but in fact that code is defined in a file that contains

  (in-packgae :com.gigamonkeys.asdf-extensions)

where :com.gigamonkeys.asdf-extensions uses :cl and :asdf and thus
inherits the symbols PERFORM, OPERATION-DONE, etc. It's perfectly okay
to use those symbols for the purposes intended, namely defining
methods on the generic functions they name.

-Peter

P.S. The method:

  (defmethod output-files ((o operation) (c component)) nil)

that you asked about before came into my code via cut-n-paste of the
methods used to implement the built in ASDF:LOAD-OP operation. It
seems to me that that, based on the source file organization, that
that method should have been defined as:

  (defmethod output-files ((o load-op) (c component)) nil)

in asdf.lisp. (Or the more generic method should be moved up in the
file since it, in fact, applies to all operations that don't define a
more specific method.) Given that the non-specific method already
exists, I can, and should, just take it out of my code. Or maybe be
explicit and define it like this:

  (defmethod output-files ((o concat-fasls) (c component)) nil)


-- 
Peter Seibel           * ·····@gigamonkeys.com
Gigamonkeys Consulting * http://www.gigamonkeys.com/
Practical Common Lisp  * http://www.gigamonkeys.com/book/