From: ·······@gmail.com
Subject: macro functionality in embedded languages.
Date: 
Message-ID: <1175826732.083190.189590@p77g2000hsh.googlegroups.com>
It's not uncommon to see an embedded language in CL in macro style, as
in:

(html:output (html:body ...))

when one could instead use a function, as in:

(html:output '(html:body ...))

One advantage of the macro style is efficiency - the call expands at
compile time and can be compiled with full optimization.  The macro
can be written in such a way that it expands to reasonably efficient
code.  In the function version, the data is parsed at run time.  While
in this particular example the efficiency is probably not an issue,
one can imagine numerical examples where it's a serious issue.

While I've read discussions of this point as well as other points
regarding the two styles, one thing that's escaped me is how to handle
dynamic data using the macro style.  For example, if one were to read
an html specification from a file, or generate one on the fly from
data, as in:

(setq doc (generate-dynamic-html ...))

or

(setq doc (read))

how can one process it using the macro?  It seems that the function
form is far superior in its flexibility.  The only way I can think of
to get this functionality from the macro form is hacking the macro
into the reader, which will handle the 2nd case, but not the first,
and seems like an awful, convoluted hack.  Another bad way would be
using EVAL.  So, is there a clean way of getting this functionality
from the macro form?

Going the other way, how would one get the efficiency of the macro
version in the function version?  One could imagine doing:

(defun compile-html (form)
   (lambda () ...))

but it seems like it would be harder to make the final returned
(lambda () ...) have the same efficiency as the macro.  One extremely
ugly way to get the same efficiency would be to generate the same
code, write it to a file & load & compile it (or otherwise using
EVAL).  Is there a cleaner way?
Thanks,

Harvey Stein

From: Rob Warnock
Subject: Re: macro functionality in embedded languages.
Date: 
Message-ID: <w4mdnY9a6YqkIYjbnZ2dnUVZ_s-rnZ2d@speakeasy.net>
<·······@gmail.com> wrote:
+---------------
| It's not uncommon to see an embedded language in CL in macro style,
| as in:
|   (html:output (html:body ...))
| when one could instead use a function, as in:
|   (html:output '(html:body ...))
+---------------

But if the body is a constant, as you've shown here, how does
one insert dynamically-calculated HTML or even dynamically-
calculated plain text into the body?!? That is, how do you do
the equivalent of this in your "function" form [taken from
<http://rpw3.org/hacks/lisp/appsrv-demo.lhp>] ?

    (lhp-basic-page ()		; defaults to HTOUT-2.1 syntax
      (:html ()
       (:head () (:title () title))
       (lfd)
       (:body (:bgcolor "#ffffff") (lfd)
       ...[details elided]...

	 ;; Show the results, first a number and its powers in a row.
	 (:table () (lfd)
	   (:tr () (loop for e from 1 to pows and h in headings do 
		  (let ((p (cond ((= e 1) ":") ((= e pows) "") (t ","))))
		    (htm (:th (:nowrap) (fmt h e) p)))))
	   (loop for i from 1 to nums do
	     (htm (lfd) (:tr (:align "right")
	       (loop for e from 1 to pows do
		 (htm (:td () (princ (expt i e) lhp-stream))))))))

       ...[details elided]...)))

+---------------
| One advantage of the macro style is efficiency - the call expands at
| compile time and can be compiled with full optimization.  The macro
| can be written in such a way that it expands to reasonably efficient code.
+---------------

Another *major* advantage of what you're calling "the macro style" is
that one has full access to the full Common Lisp environment at both
macro-expansion time *and* run-time, and can easily interweave the
HTML-generating macros with other more-complex or more-general Common
Lisp code. [See my example above.]

In order to obtain the same flexibility from "the function style",
you'd need to make the arguments of the function be closures,
not constants, so you could do stuff like this [the above <TABLE>
example transliterated into a non-macro "function style"]:

    (html:table
      (lambda ()
	(html:tr
          (lambda ()
	    (loop for e from 1 to pows and h in headings do 
	      (let ((p (cond ((= e 1) ":") ((= e pows) "") (t ","))))
		(html:th
		  (lambda () (fmt h e) p)
		  :nowrap t)))))

	(loop for i from 1 to nums do
	  (html:lfd)
	  (html:tr
            (lambda ()
	      (loop for e from 1 to pows do
	        (html:td
                  (lambda () (princ (expt i e) lhp-stream)))))
	    :align "right"))))

Notice how this also forces the optional "attributes" of any
HTML element into being *after* the closure which generates
the content of the element. Ugly. Error-prone.

No, I'll stick with "the macro style", thanks...


-Rob

-----
Rob Warnock			<····@rpw3.org>
627 26th Avenue			<URL:http://rpw3.org/>
San Mateo, CA 94403		(650)572-2607
From: levy
Subject: Re: macro functionality in embedded languages.
Date: 
Message-ID: <1175846395.359178.234940@l77g2000hsb.googlegroups.com>
There's no reason not to have both macro and functional style
implemented in the same library. Factoring out the common parts is up
to the implementor and this allows the user to choose the best
matching interface for the problem at hand (even mixing them is
possible).

So I think there's no reason to choose.

my 2 cents.

levy
From: Peter Seibel
Subject: Re: macro functionality in embedded languages.
Date: 
Message-ID: <876489bjtr.fsf@gigamonkeys.com>
"levy" <················@gmail.com> writes:

> There's no reason not to have both macro and functional style
> implemented in the same library. Factoring out the common parts is up
> to the implementor and this allows the user to choose the best
> matching interface for the problem at hand (even mixing them is
> possible).
>
> So I think there's no reason to choose.

Indeed. If you'd like to see an elaboration on these ideas, check out:

 <http://www.gigamonkeys.com/book/practical-an-html-generation-library-the-interpreter.html>
 <http://www.gigamonkeys.com/book/practical-an-html-generation-library-the-compiler.html>

-Peter

-- 
Peter Seibel            :  ·····@gigamonkeys.com
Gigamonkeys Consulting  :  http://www.gigamonkeys.com/
From: ·······@gmail.com
Subject: Re: macro functionality in embedded languages.
Date: 
Message-ID: <1176066208.509807.185890@l77g2000hsb.googlegroups.com>
On Apr 5, 11:19 pm, ····@rpw3.org (Rob Warnock) wrote:
> <·······@gmail.com> wrote:
>
> +---------------
> | It's not uncommon to see an embedded language in CL in macro style,
> | as in:
> |   (html:output (html:body ...))
> | when one could instead use a function, as in:
> |   (html:output '(html:body ...))
> +---------------
>
> But if the body is a constant, as you've shown here, how does
> one insert dynamically-calculated HTML or even dynamically-
> calculated plain text into the body?!? That is, how do you do
> the equivalent of this in your "function" form [taken from
> <http://rpw3.org/hacks/lisp/appsrv-demo.lhp>] ?
>
>     (lhp-basic-page ()          ; defaults to HTOUT-2.1 syntax
>       (:html ()
>        (:head () (:title () title))
>        (lfd)
>        (:body (:bgcolor "#ffffff") (lfd)
>        ...[details elided]...
>
>          ;; Show the results, first a number and its powers in a row.
>          (:table () (lfd)
>            (:tr () (loop for e from 1 to pows and h in headings do
>                   (let ((p (cond ((= e 1) ":") ((= e pows) "") (t ","))))
>                     (htm (:th (:nowrap) (fmt h e) p)))))
>            (loop for i from 1 to nums do
>              (htm (lfd) (:tr (:align "right")
>                (loop for e from 1 to pows do
>                  (htm (:td () (princ (expt i e) lhp-stream))))))))
>
>        ...[details elided]...)))

This is easy in the functional version as well.  The lhp-basic-page
macro has to recognize non-html keyword sexps and splice them into the
output sexp.  In the functional version, you'd just escape out when
constructing the data sexp that specifies the html doc, backquoting
the overall input form and using commas as appropriate to run the
pieces of the needed lisp inline.  There's no need to go to the
continuation passing style you pointed out in the rest of your post.

But, this doesn't answer the question of what to do when your html doc
specification is stored in a variable instead of static inline code,
as in:

   (setq doc (do-some-processing (read-html-form)))

After the above, how do you use the macro version on the doc variable
to output the document?  It seems that if all you have is the macro
form, you'd have to do some uglyness like wrap it with lhp-basic-page
and EVAL it or dump it to a file & load it.  As Levy wrote, one could
potentially refactor the code to give both forms of functionality.
Does this mean that without refactoring to give this functionality,
that the only recourse is EVAL or LOAD?

As Tim Bradshaw followed up, DHTML makes some use of HTOUT, but uses
its own parser and walker.  Is this because it couldn't use the macro
version directly from HTOUT for this sort of processing?

So, my point is, if I have to do some sort of dynamic serving of
calculations (like a server that reads calculation specifications and
returns the results of the calculations), it'd look like one's forced
to use a more functional form, rather than the pure macro approach.

--
Harvey Stein
From: Tim Bradshaw
Subject: Re: macro functionality in embedded languages.
Date: 
Message-ID: <1176124851.459002.63340@y80g2000hsf.googlegroups.com>
On Apr 8, 10:03 pm, ·······@gmail.com wrote:

> As Tim Bradshaw followed up, DHTML makes some use of HTOUT, but uses
> its own parser and walker.  Is this because it couldn't use the macro
> version directly from HTOUT for this sort of processing?


No.  It (DTML, not DHTML, I dunno what DHTML is) used them because
it's a completely different markup language.  It happened to use some
of HTOUT because there were useful tools in HTOUT to do with XML/HTML
tag generation and so on.

--tim
From: ·······@gmail.com
Subject: Re: macro functionality in embedded languages.
Date: 
Message-ID: <1176177502.773032.225080@l77g2000hsb.googlegroups.com>
Can you supply a pointer to DTML?

On Apr 9, 9:20 am, "Tim Bradshaw" <··········@tfeb.org> wrote:
> On Apr 8, 10:03 pm, ·······@gmail.com wrote:
>
> > As Tim Bradshaw followed up, DHTML makes some use of HTOUT, but uses
> > its own parser and walker.  Is this because it couldn't use the macro
> > version directly from HTOUT for this sort of processing?
>
> No.  It (DTML, not DHTML, I dunno what DHTML is) used them because
> it's a completely different markup language.  It happened to use some
> of HTOUT because there were useful tools in HTOUT to do with XML/HTML
> tag generation and so on.
>
> --tim
From: Tim Bradshaw
Subject: Re: macro functionality in embedded languages.
Date: 
Message-ID: <1176194860.589974.17340@y5g2000hsa.googlegroups.com>
On Apr 10, 4:58 am, ·······@gmail.com wrote:
> Can you supply a pointer to DTML?

I can't.  It's not publicly available although it might be one day
(long after any possible interest is gone I suspect, given the way
things go).  Sorry!

--tim
From: Tim Bradshaw
Subject: Re: macro functionality in embedded languages.
Date: 
Message-ID: <1175852472.309173.70410@d57g2000hsg.googlegroups.com>
On Apr 6, 3:32 am, ·······@gmail.com wrote:

>
> One advantage of the macro style is efficiency - the call expands at
> compile time and can be compiled with full optimization.  The macro
> can be written in such a way that it expands to reasonably efficient
> code.  In the function version, the data is parsed at run time.

Yes.  HTOUT does that, and in the most recent (I hesitate to call it
current, as it's something like 4 years since it changed) it makes
quite serious attempts to optimize things, to the extent that for
compile-time-constant HTML it will generate a single constant string
at compile time which then just gets printed at runtime.  It can also
do this pretty efficiently with semi-constant HTML, getting in most
cases quite close to the best case.  Other tools can do this too, I'm
sure.

One huge advantage of the macro style, as Rob points out is that:

(let ((x ...))
  (with-html-output (s)
    (:html ...
             (dotimes (i x)
               (htm (:li (fmt "~R" i)))) ... )))

works.  That's a huge win for programmatic HTML generation.

> For example, if one were to read
> an html specification from a file, or generate one on the fly from
> data, as in:
>
> (setq doc (generate-dynamic-html ...))
>

DTML, which is a system based on HTOUT, essentially does this, reading
from a file (typically).  It uses HTOUT's printing support (note HTOUT
also uses this to generate its constant strings as well as at
runtime), but its own parser and walker (DTML's syntax is a sort of
pointy-bracket version of sexps - <header :level 3|text of header
<bold|with some boldface>>).  All it does is to build a big data
structure from the file and walk it.  DTML has its own macro
definition facility, these macros being expanded by the walker, and
those macros end up as little Lisp functions, which it compiles on the
fly.  This takes basically no time (compared to the I/O cost of
reading the file) although it does do some probably-unneeded
optimisation such that macros only get recompiled when it knows they
might have changed.  Typically the macros call Lisp functions which
then do a lot of backquote-based rewriting of the parsed DTML they get
passed.

Previous to DTML I briefly used something which was file-based but
used sexps.  It used READ and then HTOUT's printing support, but I
forget the details.  I probably still have the code.

--tim
From: Rob Warnock
Subject: Re: macro functionality in embedded languages.
Date: 
Message-ID: <KJCdnZZaE-FjtIvbnZ2dnUVZ_rKvnZ2d@speakeasy.net>
Tim Bradshaw <··········@tfeb.org> wrote:
+---------------
| ...  DTML ...  does do some probably-unneeded optimisation such that
| macros only get recompiled when it knows they might have changed.
| Typically the macros call Lisp functions which then do a lot of
| backquote-based rewriting of the parsed DTML they get passed.
| 
| Previous to DTML I briefly used something which was file-based but
| used sexps.  It used READ and then HTOUT's printing support...
+---------------

My "LHP" ("Lisp-Handled Pages", as in the example I mentioned
before, <http://rpw3.org/hacks/lisp/appsrv-demo.lhp>) does a
similar recompilation/caching, but of essentially pure CL code
[albeit using HTOUT for HTML-generation] rather than DTML or
some other tailored syntax. If the application server is handed
a path suffixed with ".lhp", it checks the modification dates
of the corresponding file and the same file with a FASL suffix
(".x86f" for CMUCL), and only recompiles and/or reloads the file
if the source is newer than the internally-cached copy of the
(compiled) page. [Compilation is done only if there's already
a compiled FASL there, so the user can control which LHP pages
get compiled and which ones only get interpreted.]

Development is *very* fast with this model, since to recompile
recently-edited code you need only hit "Refresh" (or "Reload")
button on your browser...


-Rob

-----
Rob Warnock			<····@rpw3.org>
627 26th Avenue			<URL:http://rpw3.org/>
San Mateo, CA 94403		(650)572-2607
From: ·······@gmail.com
Subject: Re: macro functionality in embedded languages.
Date: 
Message-ID: <1176264618.858310.77560@l77g2000hsb.googlegroups.com>
Your LHP sounds interesting.  Where can I find it?
Thanks,

On Apr 6, 7:07 am, ····@rpw3.org (Rob Warnock) wrote:
> Tim Bradshaw <··········@tfeb.org> wrote:
>
> +---------------
> | ...  DTML ...  does do some probably-unneeded optimisation such that
> | macros only get recompiled when it knows they might have changed.
> | Typically the macros call Lisp functions which then do a lot of
> | backquote-based rewriting of the parsed DTML they get passed.
> |
> | Previous to DTML I briefly used something which was file-based but
> | used sexps.  It used READ and then HTOUT's printing support...
> +---------------
>
> My "LHP" ("Lisp-Handled Pages", as in the example I mentioned
> before, <http://rpw3.org/hacks/lisp/appsrv-demo.lhp>) does a
> similar recompilation/caching, but of essentially pure CL code
> [albeit using HTOUT for HTML-generation] rather than DTML or
> some other tailored syntax. If the application server is handed
> a path suffixed with ".lhp", it checks the modification dates
> of the corresponding file and the same file with a FASL suffix
> (".x86f" for CMUCL), and only recompiles and/or reloads the file
> if the source is newer than the internally-cached copy of the
> (compiled) page. [Compilation is done only if there's already
> a compiled FASL there, so the user can control which LHP pages
> get compiled and which ones only get interpreted.]
>
> Development is *very* fast with this model, since to recompile
> recently-edited code you need only hit "Refresh" (or "Reload")
> button on your browser...

--
Harvey Stein
From: Rob Warnock
Subject: Re: macro functionality in embedded languages.
Date: 
Message-ID: <PqGdnVGKndCX-4HbnZ2dnUVZ_silnZ2d@speakeasy.net>
<·······@gmail.com> wrote:
+---------------
| Your LHP sounds interesting.  Where can I find it?
+---------------

You can't, sorry. It depends on my "appsrv" web applications
server infrastructure, which I hope to open-source at some point,
but it would take considerable effort to sanitize it of some
proprietary elements that crept in during its application to a
couple of clients, and my current work situation is chewing up
most/all of my spare time these days.

But given any of the other applications server infrastructures
available out there -- Araneida, cl-modlisp, Hunchentoot -- adding
an LHP-style hack yourself should be pretty easy. The code to add
LHP to my "appsrv" was only ~175 lines of CL. Just register a suffix
handler for ".lhp" with your applications server infrastructure.
Then when you get control, check to see if you've cached that page
yet -- if not, LOAD it -- and then call its driver function.
[Doing a COMPILE/re-COMPILE if needed is an optional extension.]

Note: In order to know the page's driver function, the page
has to have executed a callback into the LHP infrastructure
(an LHP-SET-PAGE-FUNCTION call, in my case) during the LOAD.
A bit clumsy, perhaps, but unfortunately CL's LOAD returns only
a generalized boolean, not a value computable by the LOADed code.


-Rob

-----
Rob Warnock			<····@rpw3.org>
627 26th Avenue			<URL:http://rpw3.org/>
San Mateo, CA 94403		(650)572-2607