From: javuchi
Subject: How to save lists containing functions to disk
Date: 
Message-ID: <1132718791.478944.200120@g47g2000cwa.googlegroups.com>
Doing this in CMUCL:

 (defun pp (x) (* x 2))
 (compile 'pp)
 (list 'a 'b (symbol-function 'pp))

I obtain this:

(A B #<Function PP {58A94879}>)

Is there any way to save the list to a file, including the compiled
function, and then, in another session, reload and use that function?
Thanks.

From: Pisin Bootvong
Subject: Re: How to save lists containing functions to disk
Date: 
Message-ID: <1132738251.753397.317470@g43g2000cwa.googlegroups.com>
CL-STORE allow you to serialize the function name. So, while you can
not actually serialize the code of the function, you can still
serialize the function name.
Later in another session you can load all the code. And then use
CL-STORE to load the list containing function to call.
From: Pascal Bourguignon
Subject: Re: How to save lists containing functions to disk
Date: 
Message-ID: <87veyj67mv.fsf@thalassa.informatimago.com>
"javuchi" <·······@gmail.com> writes:

> Doing this in CMUCL:
>
>  (defun pp (x) (* x 2))
>  (compile 'pp)
>  (list 'a 'b (symbol-function 'pp))
>
> I obtain this:
>
> (A B #<Function PP {58A94879}>)
>
> Is there any way to save the list to a file, including the compiled
> function, and then, in another session, reload and use that function?
> Thanks.

There's no standard API to save compiled functions.  
Perhaps your implementation has something for it?

You can try to save the source of the function.  

Note that for functions enclosed in a closures, it may be hard to
retrieve a meaningful source, but let's ignore this issue for now.

You could try to use the standard function FUNCTION-LAMBDA-EXPRESSION.
Unfortunately, implementations are allowed to return NIL, rendering 
FUNCTION-LAMBDA-EXPRESSION useless when you need it (Murphy's Law).

So the only portable way to do it, is to save yourself the function
definition.  For functions defined with DEFUN, you could shadow DEFUN,
and write your own macro that would save the source of the function on
the property list of the function name symbol.  But a smarter way
needs to be implemented anyways to handle (SETF name) and anonymous
functions built with LAMBDA.

Something like:


(defparameter *function-sources* (make-hash-table :test (function equal)))

(shadow '(defun defmacro lambda))

(cl:defmacro defmacro (name args &body body)
   `(progn 
        (compile (cl:defmacro ,name ,args ,@body))
        (setf (gethash (fdefinition ',name) *function-sources*)
              '(defmacro ,name ,args ,@body))
        ',name))

(cl:defmacro defun (name args &body body)
   `(progn 
        (compile (cl:defun ,name ,args ,@body))
        (setf (gethash (fdefinition ',name) *function-sources*)
              '(defun ,name ,args ,@body))
        ',name))

(cl:defmacro lambda (args &body body)
  (let ((fun (gensym)))
   `(let ((,fun (compile nil (cl:lambda ,args ,@body))))
        (setf (gethash ,fun *function-sources*) '(lambda ,args ,@body))
        ,fun)))


(defun print-function (fun &optional (stream t))
  (print (or (gethash fun *function-sources*) fun) stream))


(Note that I compile the functions immediately, which may not be the
normal behavior when an interpreter is available.)

So now you can write:

(defun f (x) (1+ x))
(defmacro g (x) `(print (1+ ,x)))
(setf c (lambda (z) (+ (sqrt -1) z)))

(dolist (x (list (function f) (symbol-function 'g) c))
   (print-function x)) 

-->
(DEFUN F (X) (1+ X)) 
(DEFMACRO G (X) `(PRINT (1+ ,X))) 
(LAMBDA (Z) (+ (SQRT -1) Z)) 


Now, for the closures.  In the case of a single function in a closure,
we could extend the above macros to get hold of the environment, and
try to reproduce it, printing (LET (...) (DEFUN ...)).

But environments are opaque in Common Lisp, so you cannot do anything
portably.  If you have the good taste of using an implementation that
gives an API to access the environment, then you could generate
additionnal functions in the closure to let you retrieve the values in
the closure at run-time, when you print the function; something like:


(cl:defmacro defun (&environment env    name args &body body)
   `(progn 
        (compile (cl:defun ,name ,args ,@body))
        (setf (gethash (fdefinition ',name) *function-sources*)
              (lambda ()
                 (list 'let 
                       ,(cons 'list
                              (%map-env ; IMPLEMENTATION SPECIFIC! 
                                 (lambda (var) 
                                    (list 'list (list 'quote var) var))
                                 env))
                       '(defun ,name ,args ,@body))))
        ',name))

;; And do the same for defmacro and lambda...

which would generate:

[93]> (defmacro ee1 (&environment env form) 
        `(print (macroexpand-1 ',form ',env)))
EE1
[94]> (let ((x 1) (y 2)) (ee1 (defun f (x) (1+ x))))

(PROGN (COMPILE (COMMON-LISP:DEFUN F (X) (1+ X)))
 (SETF (GETHASH (FDEFINITION 'F) *FUNCTION-SOURCES*)
  (LAMBDA NIL (LIST 'LET (LIST (LIST 'Y Y) (LIST 'X X)) '(DEFUN F (X) (1+ X)))))
 'F) 
(PROGN (COMPILE (COMMON-LISP:DEFUN F (X) (1+ X)))
 (SETF (GETHASH (FDEFINITION 'F) *FUNCTION-SOURCES*)
  (LAMBDA NIL (LIST 'LET (LIST (LIST 'Y Y) (LIST 'X X)) '(DEFUN F (X) (1+ X)))))
 'F)
[95]> 

So instead of keeping directly the source form to print for each
function, we keep a closure that will generate this source form with
the current state of the closure when it'll be called:


(defun print-function (fun &optional (stream t))
  (let ((clo  (gethash fun *function-sources*)))
    (print (if clo (funcall clo) fun) stream)))

Not portable, but this can be manageable for a couple of implementations.

Now what will be harder, is to be able to print something meaningful
for these functions:

(let ((x 1))
  (defun incx (i) (incf x i))
  (defun resetx () (setf x 0))
  (defparameter getx (lambda (password) 
                      (if (= password 42)
                          x
                          (error "Go to jail without taking the $20,000")))))


(print-function incx)    --> ?
(print-function resetx)  --> ?
(print-function getx)    --> ?


What we'd want is to print:

(let ((x 57)) ; the current value when we print it!
  (defun incx (i) (incf x i))
  (defun resetx () (setf x 0))
  (defparameter getx (lambda (password) 
                      (if (= password 42)
                          x
                          (error "Go to jail without taking the $20,000")))))

once for the three print-function calls.  
What would you print if only one is called?
What would you print for:

(print-function incx)
(incx 34)
(print-function getx)

?


Have fun!


-- 
__Pascal Bourguignon__                     http://www.informatimago.com/

There is no worse tyranny than to force a man to pay for what he does not
want merely because you think it would be good for him. -- Robert Heinlein
From: Pascal Bourguignon
Subject: Re: How to save lists containing functions to disk
Date: 
Message-ID: <87iruj65nt.fsf@thalassa.informatimago.com>
Pascal Bourguignon <····@mouse-potato.com> writes:
> But environments are opaque in Common Lisp, so you cannot do anything
> portably.  If you have the good taste of using an implementation that
> gives an API to access the environment, then you could generate
> additionnal functions in the closure to let you retrieve the values in
> the closure at run-time, when you print the function; something like:
>
>
> (cl:defmacro defun (&environment env    name args &body body)
>    `(progn 
>         (compile (cl:defun ,name ,args ,@body))
>         (setf (gethash (fdefinition ',name) *function-sources*)
>               (lambda ()
>                  (list 'let 
>                        ,(cons 'list
>                               (%map-env ; IMPLEMENTATION SPECIFIC! 
>                                  (lambda (var) 
>                                     (list 'list (list 'quote var) var))
>                                  env))
>                        '(defun ,name ,args ,@body))))
>         ',name))

I forgot to mention, for more fun, the environment includes macrolets,
lexical functions (flet & labels), and symbol-macrolets, if you want
to be complete...


-- 
__Pascal Bourguignon__                     http://www.informatimago.com/
You never feed me.
Perhaps I'll sleep on your face.
That will sure show you.
From: javuchi
Subject: Re: How to save lists containing functions to disk
Date: 
Message-ID: <1132792350.711649.238450@g44g2000cwa.googlegroups.com>
Pascal, your code is pretty smart. I must say that I'm almost a newbe
(still reading OnLisp), but I have understood it.
This solution seems to be reasonable. I must write source code, be
aware of closures (hope CMUCL supports accesing the env, am I lucky?),
and load and compile it when reading the file.

(Note: Although it is a good solution, i have a big problem... my
program is intended to manage a big amount of functions of this type,
99% of them generated at runtime. I hope CMUCL to be fast enough
compiling and doing GC...(this might be another problem, i don't think
GC might work for deleting unused functions as it does with data).
Perhaps a program which works in this manner is not correctly designed
and I should reconsiderate another paradigm.)

Really thanks.

Just to be curius... how much time did you get to think and write it?

Bye.
From: Pascal Bourguignon
Subject: Re: How to save lists containing functions to disk
Date: 
Message-ID: <87lkze4zbv.fsf@thalassa.informatimago.com>
"javuchi" <·······@gmail.com> writes:

> Pascal, your code is pretty smart. I must say that I'm almost a newbe
> (still reading OnLisp), but I have understood it.
> This solution seems to be reasonable. I must write source code, be
> aware of closures (hope CMUCL supports accesing the env, am I lucky?),
> and load and compile it when reading the file.
>
> (Note: Although it is a good solution, i have a big problem... my
> program is intended to manage a big amount of functions of this type,
> 99% of them generated at runtime. I hope CMUCL to be fast enough
> compiling and doing GC...(this might be another problem, i don't think
> GC might work for deleting unused functions as it does with data).

Perhaps you may consider the use of weak references (weak pointers in
CMUCL) for the hash table: when a function can be garbage collected,
there's no use keeping it or its source in the hash table.  

> Perhaps a program which works in this manner is not correctly designed
> and I should reconsiderate another paradigm.)

Depending on how you generate the functions.  Most of the time, all
the functions are very similar (either they're closures sharing the
same code, or generated from the same pattern s-expression.  If you
have to save a lot of these similar functions, it could be better to
write instead the generating parameters, the "genes" of these functions.


If you have compilation time constraints, you could instead generate
these functions in a source file, and then use: 
(COMPILE-FILE "generated-functions.lisp")
and then load them at once with:
(LOAD (COMPILE-FILE-PATHNAME  "generated-functions.lisp"))


You've not explained why you need to save the functions either.
Perhaps you could just save an image, the whole memory with all the
data and functions. Later you can restart lisp with the saved core image.
http://common-lisp.net/project/cmucl/doc/cmu-user/extensions.html#toc48


> Just to be curius... how much time did you get to think and write it?

Just the time to write it.  This question has already been asked here,
so the principle is known.

-- 
__Pascal Bourguignon__                     http://www.informatimago.com/
You never feed me.
Perhaps I'll sleep on your face.
That will sure show you.