From: budden
Subject: file level metaprogramming
Date: 
Message-ID: <49c8460f-b59d-4e07-aa5e-c10e42b518bc@m24g2000vbp.googlegroups.com>
Hi list,
  I found it very inconvinient to debug generated CL code. The problem
is that there is no "source location". So GUI's won't help. This can
be fixed with code like this (it requires cl-fix, browse
recent "portable hierarchical packages" thread to obtain it).
).

(in-package :budden-tools)
(in-readtable :buddens-readtable)

(defparameter *defun-to-file-directory*
  #+win32 "l:/sw/defun-to-file"
  #-win32 "/home/denis/sw/defun-to-file")

(defun maybe-add-slash (string)
  (if (string= (sequence-last string) "/")
    string
    (str+ string "/")))

(defmacro defun-to-file (name &rest more)
  (assert (every #L(not (find !1 "\\/.?* ")) (string name)))
  (let1 filename (str+ (maybe-add-slash *defun-to-file-directory*)
name)
    (with-open-file (out (str+ filename ".lisp")
                         :direction :output
                         :if-does-not-exist :create
                         :if-exists :supersede)
      (pprint `(in-package ,(keywordize (package-name *package*)))
out)
      (pprint `(defun ,name ,@more) out))
    (if (compile-file (str+ filename ".lisp"))
      (load filename))))

Sorry for posting code which depends on my extensions,
got no time to fix it. Note that function would be defined in a zero
lexical
environment.

Feedback/ideas/critics is welcome.

From: Alessio Stalla
Subject: Re: file level metaprogramming
Date: 
Message-ID: <b134724b-7230-4bc3-9c7b-80147691b6c1@21g2000vbk.googlegroups.com>
On May 8, 1:52 pm, budden <···········@mail.ru> wrote:
> Hi list,
>   I found it very inconvinient to debug generated CL code. The problem
> is that there is no "source location". So GUI's won't help.

Actually, I think implementations have the freedom to provide what
you're asking for, and in fact I think at least commercial
implementations offer it.

> This can
> be fixed with code like this (it requires cl-fix, browse
> recent "portable hierarchical packages" thread to obtain it).
> ).
>
> (in-package :budden-tools)
> (in-readtable :buddens-readtable)
>
> (defparameter *defun-to-file-directory*
>   #+win32 "l:/sw/defun-to-file"
>   #-win32 "/home/denis/sw/defun-to-file")
>
> (defun maybe-add-slash (string)
>   (if (string= (sequence-last string) "/")
>     string
>     (str+ string "/")))
>
> (defmacro defun-to-file (name &rest more)
>   (assert (every #L(not (find !1 "\\/.?* ")) (string name)))
>   (let1 filename (str+ (maybe-add-slash *defun-to-file-directory*)
> name)
>     (with-open-file (out (str+ filename ".lisp")
>                          :direction :output
>                          :if-does-not-exist :create
>                          :if-exists :supersede)
>       (pprint `(in-package ,(keywordize (package-name *package*)))
> out)
>       (pprint `(defun ,name ,@more) out))
>     (if (compile-file (str+ filename ".lisp"))
>       (load filename))))

I'm not an expert about CL compilation, but I believe N invocations to
your defun-to-file have not the same semantics of compiling and
loading a file with N functions definitions. AFAIK when you compile a
file it gets wrapped in a with-compilation-unit i.e. compiled as a
single entity. More important, on a second thought, is that if you
compile a file with a defun-to-file in it you will get strange
effects: the compiled function will be loaded at compilation time, and
if you'll load the compiled file in a fresh image, you won't have the
function defined! At least, this is my understanding of it.

On a side note about your utilities, I happen to have defined a str+
utility too, with the same name :) I also "keywordize" quite often,
although I still haven't bothered defining a function for it.

hth,
Alessio

> Sorry for posting code which depends on my extensions,
> got no time to fix it. Note that function would be defined in a zero
> lexical
> environment.
>
> Feedback/ideas/critics is welcome.
From: budden
Subject: Re: file level metaprogramming
Date: 
Message-ID: <6feda0bf-0243-4ffd-96ef-573fe217135f@n8g2000vbb.googlegroups.com>
>More important, on a second thought, is that if you
>compile a file with a defun-to-file in it you will get strange
>effects: the compiled function will be loaded at compilation time, and
>if you'll load the compiled file in a fresh image, you won't have the
>function defined!
Yes, you're right. This is an issue I didn't think of. Are there
suggestions about how to fix it? Would this help:

...
 (assert (compile-file (str+ filename ".lisp"))) ; compile at
mexpansion time
 `(load ,filename)))) ; load at evaluation time

I tested it and it looks like it helps.

> I also "keywordize" quite often, although I still haven't bothered
> defining a function for it.
iterate::keywordize
#L syntax is also from iterate. Also there is a dsetq, which
allows destructuring in setf.
From: Alessio Stalla
Subject: Re: file level metaprogramming
Date: 
Message-ID: <b99395f5-0a16-4b4f-9b65-d0e306b54fa8@21g2000vbk.googlegroups.com>
On May 8, 3:15 pm, budden <···········@mail.ru> wrote:
> >More important, on a second thought, is that if you
> >compile a file with a defun-to-file in it you will get strange
> >effects: the compiled function will be loaded at compilation time, and
> >if you'll load the compiled file in a fresh image, you won't have the
> >function defined!
>
> Yes, you're right. This is an issue I didn't think of. Are there
> suggestions about how to fix it? Would this help:
>
> ...
>  (assert (compile-file (str+ filename ".lisp"))) ; compile at
> mexpansion time
>  `(load ,filename)))) ; load at evaluation time
>
> I tested it and it looks like it helps.

Well, for what I can see I think this is ok. But as I said I don't
know all the details behind loading and compilation. Is loading a file
with some code the same than loading a file that loads another file
with the same code? Probably yes, but I'm not 100% sure.

> > I also "keywordize" quite often, although I still haven't bothered
> > defining a function for it.
>
> iterate::keywordize
> #L syntax is also from iterate. Also there is a dsetq, which
> allows destructuring in setf.

Cool, I've never really used iterate though I've always wanted to try
it. Those are just more reasons to :)
From: budden
Subject: Re: file level metaprogramming
Date: 
Message-ID: <d98f8c0b-f51a-48c5-a589-12d17b33e6e9@q14g2000vbn.googlegroups.com>
> Is loading a file with some code the same than loading a file that
> loads another file with the same code? Probably yes, but I'm not 100% sure.
I'm unsure too. I think there are some subtleties. But, in practice, I
was unable
to debug a error in my generated function until I wrote defun-to-file.
Function was
about 50 lines long, but its body was built with heave use of
quasiquoting.
So it was really hard to find out where "undefined variable" resides.
With defun-to-file, I solved this problem in a minute. I don't think
it is reasonable to replace defun with defun-to-file, but it is
reasonable to replace (eval `(defun ...)) with (eval `(defun-to-
file ...))
And it is (sometimes) reasonable when defun is a result of
macroexpansion
or a part of macroexpander code.

I also have to note that generated filename should also include
package designator
and this is not the case with my code.

> Cool, I've never really used iterate though I've always wanted to try
> it. Those are just more reasons to :)
Iterate is much more useful than loop. First of all, it allows for
ordinary CL control structures such as cond, if, when, progn. Complex
control structures are super ugly with LOOP. Second, it is extensible
-
you can define your own clauses with ease. Definitely, I think that
iterate should have been included in a standard instead of LOOP.

I recommend using my fork (in fact with only a small patch),
http://sourceforge.net/projects/iteratekeywords/

With it, you can use keywords to designate clauses. So the only symbol
you
need to import is iter:iter itself. Otherwise, you would need to
import
(or always qualify) symbols such as 'iter:for and 'iter:collecting,
which would cause
clashes with libraries as cl-utilities and/or alexandria. There is a
workaround
with synonyms, but it does not cover all the cases. This is why I have
made my patch.
From: Marco Antoniotti
Subject: Re: file level metaprogramming
Date: 
Message-ID: <441abad4-c853-4730-ae52-1c88d3ba691e@o14g2000vbo.googlegroups.com>
On May 9, 12:40 am, Alessio Stalla <·············@gmail.com> wrote:
> On May 8, 3:15 pm, budden <···········@mail.ru> wrote:
>
> > >More important, on a second thought, is that if you
> > >compile a file with a defun-to-file in it you will get strange
> > >effects: the compiled function will be loaded at compilation time, and
> > >if you'll load the compiled file in a fresh image, you won't have the
> > >function defined!
>
> > Yes, you're right. This is an issue I didn't think of. Are there
> > suggestions about how to fix it? Would this help:
>
> > ...
> >  (assert (compile-file (str+ filename ".lisp"))) ; compile at
> > mexpansion time
> >  `(load ,filename)))) ; load at evaluation time
>
> > I tested it and it looks like it helps.
>
> Well, for what I can see I think this is ok. But as I said I don't
> know all the details behind loading and compilation. Is loading a file
> with some code the same than loading a file that loads another file
> with the same code? Probably yes, but I'm not 100% sure.
>
> > > I also "keywordize" quite often, although I still haven't bothered
> > > defining a function for it.
>
> > iterate::keywordize
> > #L syntax is also from iterate. Also there is a dsetq, which
> > allows destructuring in setf.
>
> Cool, I've never really used iterate though I've always wanted to try
> it. Those are just more reasons to :)

IMHO.  The only reason to use ITERATE it is if (1) you have special
data structure, (2) you actually wrote an ITERATE extension to deal
with it, (3) your code heavily depends on multiple traversal of said
data structure and ITERATE extension.

Again, IMHO, mere replacement of LOOP by ITERATE is just an annoyance
adding needless dependenciees on code.

Since we are at it... for the record.  I like Scheme Lisp-1 style.  I
also like ITERATE.  I know when I need them and when not.

Cheers
--
Marco
ELS 2009 www.european-lisp-symposium.org
From: Thomas A. Russ
Subject: Re: file level metaprogramming
Date: 
Message-ID: <ymiskjfmmaf.fsf@blackcat.isi.edu>
budden <···········@mail.ru> writes:

> (defparameter *defun-to-file-directory*
>   #+win32 "l:/sw/defun-to-file"
>   #-win32 "/home/denis/sw/defun-to-file")

Well, this sort of thing is EXACTLY what logical pathnames were invented
to solve.  You should use the following parameter definition:

(defparameter *defun-to-file-directory*
              "budden:sw;defun-to-file;")

> (defun maybe-add-slash (string)
>   (if (string= (sequence-last string) "/")
>     string
>     (str+ string "/")))

And then you can dispense with the maybe adding slash stuff entirely.
Of course, if you really want to do it properly you should manipulate
pathnames in your code rather than strings denoting pathnames.  That way
you stay portable across operating systems.

All of the file-system specific stuff then gets handled in a
translations file that you load at the start of a session, (probably as
part of process of loading your code or in your lisp init file.)

(setf (logical-pathname-translation "budden")
      '(("budden:sw;**;*.*"  ...)))

where "..." gets the file-system specific translation.  If you want to
conditionalize this on the implementation then you could do that here in
order to have a single translation file.  But this also makes it easy
for someone else to use a completely different directory structure to
store the location for your code.

-- 
Thomas A. Russ,  USC/Information Sciences Institute
From: budden
Subject: Re: file level metaprogramming
Date: 
Message-ID: <d0e958b0-da42-46d4-83ad-a088636bf6ab@m24g2000vbp.googlegroups.com>
> budden <···········@mail.ru> writes:
> > (defparameter *defun-to-file-directory*
> >   #+win32 "l:/sw/defun-to-file"
> >   #-win32 "/home/denis/sw/defun-to-file")
>
> Well, this sort of thing is EXACTLY what logical pathnames were invented
> to solve.  You should use the following parameter definition:
Thanks, you're right. Maybe this is due to my ignorance :)