Consider the following toy example code:
(asdf:operate 'asdf:load-op :cl-ppcre)
(defconstant +func-def+ "^(\\S+)\\s*\\(.*\\)\\s*\\{")
(defconstant +echo-arg+ "echo\\s+(\\S+)")
(defun all-echo-arg1s (path)
(let ((args (make-hash-table :test #'equal)))
(with-open-file (s path)
(extract-echo-arg1s nil s args))
args))
(defun extract-echo-arg1s (fname stream args)
(loop for line = (read-line stream nil)
while line do
(cl-ppcre:register-groups-bind (arg) (+echo-arg+ line)
(when fname
(push arg (gethash fname args))))
(cl-ppcre:register-groups-bind (fn) (+func-def+ line)
(extract-echo-arg1s fn stream args))))
(defun extract-test ()
(all-echo-arg1s "/usr/lib/clisp/clisp-link"))
If I (load (compile-file "...")) it, the result runs s-l-o-w-l-y.
(Why do I need to repeat the asdf:operate call in the REPL to be
able to compile the code?)
If instead I give the following form to the REPL, extract-test
runs almost instantly:
(progn (load "vars.lisp") (load (compile-file "vars")) (extract-test))
Why is the FASL output different, and does this account for the
difference? (The output in vars.lib is the same in both cases.)
$ diff -ub vars.fas.not vars.fas
--- vars.fas.not 2006-01-31 17:39:57.542633800 -0600
+++ vars.fas 2006-01-31 17:40:35.740225200 -0600
@@ -44,8 +44,9 @@
DB DC DD B1 72 48 2D 06 04 42 04 92 03 FF A3 16 04 AF 63 38 02 80
79 00 1C FF B6 00 19 05)
(|COMMON-LISP|::|T| |COMMON-LISP|::|T| |COMMON-LISP|::|T|)
- #.|COMMON-LISP-USER|::|+ECHO-ARG+| :|START| 0. :|END| |CL-PPCRE|::|SCAN|
- #.#'|COMMON-LISP|::|SUBSEQ| #.|COMMON-LISP-USER|::|+FUNC-DEF+|))
+ #.(|CL-PPCRE|::|CREATE-SCANNER| |COMMON-LISP-USER|::|+ECHO-ARG+|)
+ :|START| 0. :|END| |CL-PPCRE|::|SCAN| #.#'|COMMON-LISP|::|SUBSEQ|
+ #.(|CL-PPCRE|::|CREATE-SCANNER| |COMMON-LISP-USER|::|+FUNC-DEF+|)))
#Y(#:|22 23 (DEFUN EXTRACT-TEST NIL ...)-6|
#20Y(00 00 00 00 00 00 00 00 00 01 DA 2F 01 DA DC 32 83 C5 19 01)
(|COMMON-LISP|::|T| |COMMON-LISP|::|T| |COMMON-LISP|::|T|)
Is the observed difference in performance to be expected? If so, why?
This is with CLISP 2.38 on Cygwin.
Thanks,
Greg
From: Edi Weitz
Subject: Re: clisp: COMPILE-FILE needs two LOADs?
Date:
Message-ID: <uacdcc55i.fsf@agharta.de>
On Tue, 31 Jan 2006 23:46:36 -0000, ······@hiwaay.net (Greg Bacon) wrote:
> Consider the following toy example code:
>
> (asdf:operate 'asdf:load-op :cl-ppcre)
>
> (defconstant +func-def+ "^(\\S+)\\s*\\(.*\\)\\s*\\{")
>
> (defconstant +echo-arg+ "echo\\s+(\\S+)")
>
> (defun all-echo-arg1s (path)
> (let ((args (make-hash-table :test #'equal)))
> (with-open-file (s path)
> (extract-echo-arg1s nil s args))
> args))
>
> (defun extract-echo-arg1s (fname stream args)
> (loop for line = (read-line stream nil)
> while line do
> (cl-ppcre:register-groups-bind (arg) (+echo-arg+ line)
> (when fname
> (push arg (gethash fname args))))
> (cl-ppcre:register-groups-bind (fn) (+func-def+ line)
> (extract-echo-arg1s fn stream args))))
>
> (defun extract-test ()
> (all-echo-arg1s "/usr/lib/clisp/clisp-link"))
>
> If I (load (compile-file "...")) it, the result runs s-l-o-w-l-y.
>
> (Why do I need to repeat the asdf:operate call in the REPL to be
> able to compile the code?)
Wrap the form with an EVAL-WHEN. Forms are evaluated for their side
effect when you LOAD a file, not when you COMPILE-FILE it. However,
when the file is compiled, the compiler already needs to know the
CL-PPCRE package in order to be able to READ the file.
> If instead I give the following form to the REPL, extract-test runs
> almost instantly:
>
> (progn (load "vars.lisp") (load (compile-file "vars")) (extract-test))
>
> Why is the FASL output different, and does this account for the
> difference? (The output in vars.lib is the same in both cases.)
CL-PPCRE uses compiler macros to pre-compile constant regular
expressions. CLISP obviously isn't able to determine (at compile
time) that +FUNC-DEF+ and +ECHO-ARG+ are constant and therefore a new
regular expression has to be compiled for each call. If you first
LOAD the file, then CLISP knows about the constants and now CL-PPCRE
creates pre-compiled regular expressions. Thus the difference in the
FASL files. Again, you could use EVAL-WHEN (around the DEFCONSTANT
forms) to overcome this. A better way would be to have these forms in
different files.
I'm not a language lawyer but I think CLISP's behaviour is
ANSI-compliant. FWIW, with LispWorks CL-PPCRE can generate
pre-compiled regular expressions in one pass.
HTH,
Edi.
--
Lisp is not dead, it just smells funny.
Real email: (replace (subseq ·········@agharta.de" 5) "edi")
In article <·············@agharta.de>,
Edi Weitz <········@agharta.de> wrote:
: [...]
:
: CL-PPCRE uses compiler macros to pre-compile constant regular
: expressions. CLISP obviously isn't able to determine (at compile
: time) that +FUNC-DEF+ and +ECHO-ARG+ are constant and therefore a new
: regular expression has to be compiled for each call. If you first
: LOAD the file, then CLISP knows about the constants and now CL-PPCRE
: creates pre-compiled regular expressions. Thus the difference in the
: FASL files. Again, you could use EVAL-WHEN (around the DEFCONSTANT
: forms) to overcome this. A better way would be to have these forms in
: different files.
Thanks for the explanation. The same problem is the motivation for
Perl's /o workaround: it tells the compiler that it only needs to
compile the pattern once.
I also found that hoisting compilation of the patterns also has the
same effect:
(defvar *func-def*
(cl-ppcre:create-scanner "^(\\S+)\\s*\\(.*\\)\\s*\\{"))
(defvar *echo-arg*
(cl-ppcre:create-scanner "echo\\s+(\\S+)"))
Vielen Dank!
Greg
--
[Freud] raved about "the stimulative effect of coca on the genitalia" and in
a letter to his fiancee, Martha, "forewarned her of the pleasure she could
expect from 'a wild man with cocaine in his body'"--revealing him to be not
just the father of psychoanalysis but a precursor of gangsta rap. --GK
From: Edi Weitz
Subject: Re: clisp: COMPILE-FILE needs two LOADs?
Date:
Message-ID: <uslr2x1gm.fsf@agharta.de>
On Wed, 01 Feb 2006 16:59:52 -0000, ······@hiwaay.net (Greg Bacon) wrote:
> Thanks for the explanation. The same problem is the motivation for
> Perl's /o workaround: it tells the compiler that it only needs to
> compile the pattern once.
>
> I also found that hoisting compilation of the patterns also has the
> same effect:
>
> (defvar *func-def*
> (cl-ppcre:create-scanner "^(\\S+)\\s*\\(.*\\)\\s*\\{"))
>
> (defvar *echo-arg*
> (cl-ppcre:create-scanner "echo\\s+(\\S+)"))
I think what you're doing here (using CREATE-SCANNER explicitely) is
more like Perl's /o because in both cases you're actively telling the
regex engine that you want to compile this regular expression once and
then keep it. (Perl's /o just adds a layer of syntactical sugar which
could be added to CL-INTERPOL if someone feels like it.)
For regular expressions not containing variable interpolations Perl is
usually clever enough to only compile them once even if you don't
provide the /o option. This is similar to what CL-PPCRE is trying to
do but it needs support from the underlying Lisp implementation.
--
Lisp is not dead, it just smells funny.
Real email: (replace (subseq ·········@agharta.de" 5) "edi")
Edi Weitz <········@agharta.de> writes:
> > (defconstant +func-def+ "^(\\S+)\\s*\\(.*\\)\\s*\\{")
> > (defconstant +echo-arg+ "echo\\s+(\\S+)")
> CL-PPCRE uses compiler macros to pre-compile constant regular
> expressions. CLISP obviously isn't able to determine (at compile
> time) that +FUNC-DEF+ and +ECHO-ARG+ are constant and therefore a new
> regular expression has to be compiled for each call.
Are you sure? I'd have expected defconstant inside a file to be
visible to subsequent forms compiled within the same file.
(EVAL-WHEN ... (DEFCONSTANT ...)) would sound pretty stupid to me.
> I'm not a language lawyer but I think CLISP's behaviour is
> ANSI-compliant.
Are you sure this is the reason?
Does CONSTANTP really return NIL?
I'm just guessing, I haven't tried out cl-ppcre yet.
Regards,
Jorg Hohle
Telekom/T-Systems Technology Center
From: Edi Weitz
Subject: Re: clisp: COMPILE-FILE needs two LOADs?
Date:
Message-ID: <u7j8753pa.fsf@agharta.de>
On 07 Feb 2006 19:35:10 +0100, Joerg Hoehle <······@users.sourceforge.net> wrote:
> Edi Weitz <········@agharta.de> writes:
>
>> CLISP obviously isn't able to determine (at compile time) that
>> +FUNC-DEF+ and +ECHO-ARG+ are constant
>
> Are you sure?
Yes.
> I'd have expected defconstant inside a file to be visible to
> subsequent forms compiled within the same file. (EVAL-WHEN ...
> (DEFCONSTANT ...)) would sound pretty stupid to me.
Yes, sounds stupid to me as well. But I usually don't use CLISP.
> Are you sure this is the reason?
Yes.
> I'm just guessing, I haven't tried out cl-ppcre yet.
Here's a simple example:
----------------------> foo.lisp <----------------------
(defconstant +foo+ 42)
(defun frob (arg)
(declare (ignore arg))
nil)
(define-compiler-macro frob (&whole form &environment env arg)
(cond ((constantp arg env) t)
(t form)))
(defun quux ()
(frob +foo+))
----------------------> foo.lisp <----------------------
Start CLISP[1] and then try
(load (compile-file "foo.lisp"))
(quux)
This returns NIL. In LispWorks, CMUCL, and AllegroCL it returns T.
Cheers,
Edi.
[1] "2.36 (2005-12-04) (built 3343370120) (memory 3344787442)" on my
machine
--
Lisp is not dead, it just smells funny.
Real email: (replace (subseq ·········@agharta.de" 5) "edi")
Edi Weitz <········@agharta.de> wrote:
+---------------
| (defconstant +foo+ 42)
| (defun frob (arg)
| (declare (ignore arg))
| nil)
| (define-compiler-macro frob (&whole form &environment env arg)
| (cond ((constantp arg env) t)
| (t form)))
| (defun quux ()
| (frob +foo+))
| ----------------------> foo.lisp <----------------------
|
| Start CLISP[1] and then try
|
| (load (compile-file "foo.lisp"))
| (quux)
|
| This returns NIL. In LispWorks, CMUCL, and AllegroCL it returns T.
+---------------
Note that even though CMUCL passes your above test, it still has an
issue users need to watch out for involving the *value* of a DEFCONSTANT
not being available at compile time, even though your (QUUX) worked
fine. [Tim Bradshaw and I ran into this with Tim's HTOUT macro, which
was failing to compile on CMUCL when fed a constant variable name.]
If you add the following lines to your "foo.lisp":
(defmacro frob2 (arg &environment env)
(if (and (constantp arg env)
(equal 42 (symbol-value arg)))
''win
''lose))
(defun quux2 ()
(frob2 +foo+))
and then try to compile it, you get an UNBOUND-SYMBOL-ERROR:
> (compile-file "foo.lisp")
...
; Converted FROB2.
; Compiling DEFMACRO FROB2:
; File: /usr/u/rpw3/foo.lisp
; In: DEFUN QUUX2
; (FROB2 +FOO+)
; Error: (during macroexpansion)
; Error in KERNEL::UNBOUND-SYMBOL-ERROR-HANDLER: the variable +FOO+ is unbound.
...
; Compilation unit finished.
; 1 error
; foo.x86f written.
> (load *)
; Loading #p"/usr/u/rpw3/foo.x86f".
T
> (quux)
T
> (quux2)
Execution of a form compiled with errors:
(FROB2 +FOO+)
[Condition of type KERNEL:SIMPLE-PROGRAM-ERROR]
Restarts:
0: [ABORT] Return to Top-Level.
Of course, now that it's been loaded once [and thus the DEFCONSTANT
has actually *executed* once], everything will "just work" now:
> (load (compile-file "foo.lisp"))
...[happy chatter]...
T
> (quux)
T
> (quux2)
WIN
>
This gave Tim & me [well, mostly me, since he wasn't using CMUCL]
a ton of grief for a while, and we were just about to file a bug
until we noticed that according to the CLHS CMUCL's behavior is
*PERFECTLY LEGAL* [however counterintuitive it might seem to be so!]:
Macro DEFCONSTANT
...
If a defconstant form appears as a top level form, the compiler must
recognize that name names a constant variable. An implementation
may choose to evaluate the value-form at compile time, load time,
or both. Therefore, users must ensure that the initial-value can be
evaluated at compile time (regardless of whether or not references
to name appear in the file) and that it always evaluates to the
same value.
Notice that this explicitly permits an implementation to evaluate
the "value-form" *ONLY* at load time (as CMUCL does), and in such
implementations the value will *not* be available (e.g., to macros
or compiler-macros) at compile time!! [Absent an EVAL-WHEN around
the DEFCONSTANT, of course...] What *will* be available is the state
of *being* a constant [using CONSTANTP], but not the value itself.
Said another way... Replace my FROB2/QUUX2 with these:
(defmacro frob3 (arg &environment env)
(if (and (constantp arg env)
(boundp arg))
''win
''lose))
(defun quux3 ()
(frob3 +foo+))
Then what you get is this:
> (load (compile-file "foo.lisp"))
...[happy chatter]...
T
> (quux)
T
> (quux3)
LOSE
>
-Rob
p.s. The above is the case in CMUCL-18e through -19c.
-----
Rob Warnock <····@rpw3.org>
627 26th Avenue <URL:http://rpw3.org/>
San Mateo, CA 94403 (650)572-2607
····@rpw3.org (Rob Warnock) writes:
> Notice that this explicitly permits an implementation to evaluate
> the "value-form" *ONLY* at load time (as CMUCL does), and in such
> implementations the value will *not* be available (e.g., to macros
> or compiler-macros) at compile time!! [Absent an EVAL-WHEN around
> the DEFCONSTANT, of course...]
So cmucl and clisp behave the same here.
> What *will* be available is the state
> of *being* a constant [using CONSTANTP], but not the value itself.
So here they differ.
Thanks for this precision about cmucl's behaviour. I looked at clisp,
the CLHS and found clisp consistent as well with CLHS: clisp chooses
the legal option "defconstant not evaluated at compile time".
I'm adding a note to clisp's documentation about that.
>This gave Tim & me [well, mostly me, since he wasn't using CMUCL]
>a ton of grief for a while, and we were just about to file a bug
About CONSTANTP + EVAL, I also found CLISP to be consistent: It won't
yield CONSTANTP -> true during compilation, so user code will not call
EVAL on it -- there's a lot of code out there, which does:
(if (constantp form) (eval form))
So clisp would not have caused you a ton of grief. ;)
From a user POV, the compiler evaluating DEFCONSTANT during
compilation seems best (in terms of least surprise). It's explicitly
allowed by CLHS. But this choice has its quirks w.r.t. EQL-ness and
redefinition at load-time (as cmucl/sbcl users know).
So, OTOH, not evaluating defconstant during compilation seems very
reasonable behaviour as well. Because using EVAL-WHEN, you can
enforce evaluation: the user can choose. Edi Weitz is right to
recommend using EVAL-WHEN to the OP.
Note that (if (constantp form) (eval form)) is questionable code
probably, but so far, there has been no standardization on access to the
compilation environment. And I have not yet found time to look at what
Duane Rettig proposed in that area.
IMHO it's debatable whether (if (constantp form) (eval form)) should
be flagged as unportable (even broken) code?? CONSTANTP could then
return T in clisp. But what would you do knowing that, if you can't
get to the value?
How do cmucl or sbcl in CL-PPCRE manage to make use of the values of
the constants +func-def+ and +echo-arg+ in the OP's example (do
they)?? AFAIK, SBCL is known not to provide the value via EVAL.
Regards,
Jorg Hohle
Telekom/T-Systems Technology Center
From: Edi Weitz
Subject: Re: clisp: COMPILE-FILE needs two LOADs?
Date:
Message-ID: <u8xsfnr7f.fsf@agharta.de>
On 13 Feb 2006 17:33:00 +0100, Joerg Hoehle <······@users.sourceforge.net> wrote:
> How do cmucl or sbcl in CL-PPCRE manage to make use of the values of
> the constants +func-def+ and +echo-arg+ in the OP's example (do
> they)?? AFAIK, SBCL is known not to provide the value via EVAL.
CL-PPCRE uses LOAD-TIME-VALUE, not EVAL, so no problem here. For
example:
(define-compiler-macro scan (&whole form &environment env regex target-string &rest rest)
(cond ((constantp regex env)
`(scan (load-time-value (create-scanner ,regex))
,target-string ,@rest))
(t form)))
Cheers,
Edi.
--
European Common Lisp Meeting 2006: <http://weitz.de/eclm2006/>
Real email: (replace (subseq ·········@agharta.de" 5) "edi")
Joerg Hoehle <······@users.sourceforge.net> wrote:
+---------------
| ····@rpw3.org (Rob Warnock) writes:
| > What *will* be available is the state
| > of *being* a constant [using CONSTANTP], but not the value itself.
|
| So here they [CLISP & CMUCL] differ.
|
| Thanks for this precision about cmucl's behaviour. I looked at clisp,
| the CLHS and found clisp consistent as well with CLHS: clisp chooses
| the legal option "defconstant not evaluated at compile time".
+---------------
Careful with that term "evaluated" -- as I read the CLHS, it's only
the *initial-value* of the DEFCONSTANT for which evaluation at compile
time is "optional". But the DEFCONSTANT expression itself *must* be
"recognized" at compile time (if it's a top-level form):
Macro DEFCONSTANT
...
If a DEFCONSTANT form appears as a top level form, the compiler
must recognize that NAME names a constant variable.
And also:
Function CONSTANTP
...
The following kinds of forms are considered constant forms:
...
* Constant variables, such as keywords, symbols defined by
Common Lisp as constant (such as NIL, T, and PI), and symbols
declared as constant by the user in the indicated environment
using DEFCONSTANT are always considered constant forms and
must be recognized as such by CONSTANTP.
...
So I don't see where the CLHS gives the compiler "the legal option"
of not processing (DEFCONSTANT NAME INITIAL-VALUE) at least to the
extent of making (EVAL-WHEN (:COMPILE-TOPLEVEL :LOAD-TOPLEVEL :EXECUTE)
(CONSTANTP 'NAME)) ==> T.
+---------------
| About CONSTANTP + EVAL, I also found CLISP to be consistent: It won't
| yield CONSTANTP -> true during compilation...
+---------------
And because of the above-quoted CLHS sections, I *think* that's a bug.
+---------------
| ...so user code will not call EVAL on it -- there's a lot of code
| out there, which does: (if (constantp form) (eval form))
+---------------
That code is broken, too, unless it's under a feature test for
the implementation being one which evaluates "initial-value" at
compile time.
+---------------
| From a user POV, the compiler evaluating DEFCONSTANT during
| compilation seems best (in terms of least surprise). It's explicitly
| allowed by CLHS.
+---------------
Again, I think you're confusing the issue somewhat by using the
phrase "evaluating DEFCONSTANT". This needs to be broken up into
the two acts of "processing DEFCONSTANT" and "evaluating the
initial-value of a DEFCONSTANT". As I read the CLHS, the former is
*required*; it is only the latter which is optional at compile time.
+---------------
| But this choice has its quirks w.r.t. EQL-ness and
| redefinition at load-time (as cmucl/sbcl users know).
+---------------
Yes, quite true. Thank goodness there's LOAD-TIME-VALUE... ;-}
+---------------
| IMHO it's debatable whether (if (constantp form) (eval form)) should
| be flagged as unportable (even broken) code?? CONSTANTP could then
| return T in clisp.
+---------------
And is required to, if I'm reading the CLHS correctly.
+---------------
| But what would you do knowing that, if you can't get to the value?
+---------------
Exactly. The real problem here is the standard, which gave
DEFCONSTANT a little too much leniency about evaluating its
"initial-value" *without* tying that leniency in a consistent
way to what CONSTANTP is required/allowed to do.
-Rob
p.s. By the way, my solution for this mess was to use the
following routine instead of CONSTANTP in my macros:
(defun compile-time-constantp (form)
#-cmu (constantp form)
#+cmu (and (constantp form)
(or (keywordp form)
(not (symbolp form)))))
In CMUCL, at least, this still yields T for a useful set of forms:
literal numbers, strings, vectors, quoted lists, and keywords [all of
which have values available at compile time], but not for expressions
such as (+ 1 2) [though the CLHS permits a compiler to recognize the
latter as constant, if it can].
-----
Rob Warnock <····@rpw3.org>
627 26th Avenue <URL:http://rpw3.org/>
San Mateo, CA 94403 (650)572-2607
····@rpw3.org (Rob Warnock) writes:
> Careful with that term "evaluated" -- as I read the CLHS, it's only
> the *initial-value* of the DEFCONSTANT for which evaluation at compile
> time is "optional". But the DEFCONSTANT expression itself *must* be
> "recognized" at compile time (if it's a top-level form):
> Function CONSTANTP
> ...
> The following kinds of forms are considered constant forms:
> ...
> * Constant variables, such as keywords, symbols defined by
> Common Lisp as constant (such as NIL, T, and PI), and symbols
> declared as constant by the user in the indicated environment
> using DEFCONSTANT are always considered constant forms and
> must be recognized as such by CONSTANTP.
> ...
>
> So I don't see where the CLHS gives the compiler "the legal option"
> of not processing (DEFCONSTANT NAME INITIAL-VALUE) at least to the
> extent of making (EVAL-WHEN (:COMPILE-TOPLEVEL :LOAD-TOPLEVEL :EXECUTE)
> (CONSTANTP 'NAME)) ==> T.
You bet I also read that. However, I looked at "indicated environment".
(CONSTANTP 'NAME) refers to the null lexical environment. I take this
environment *NOT* to be the compilation environment, but some "global
(run-time) environment" -- in which EVAL would take place.
To see the difference consider this: an implementation should be able
to compile a '(defconstant foo 2) even if its "global environment"
currently has foo as a constant with value 1.
CLHS says in 3.2.1:
The compilation environment is maintained by the compiler and
is used to hold definitions and declarations to be used
internally by the compiler. Only those parts of a definition
needed for correct compilation are saved. The compilation
environment is used as the environment argument to macro
expanders called by the compiler. It is UNSPECIFIED whether a
definition available in the compilation environment can be
used in an evaluation initiated in the startup environment or
evaluation environment.
The evaluation environment is a run-time environment in which
macro expanders and code specified by EVAL-WHEN to be
evaluated are evaluated. All evaluations initiated by the
compiler take place in the evaluation environment.
So to me, clearly, (CONSTANTP 'NAME) yields NIL during compilation, or
is at best unspecified.
We never discussed
(defmacro foo (&environment env) (constantp 'name env))
which would be completely different.
>Thank goodness there's LOAD-TIME-VALUE... ;-}
I'd appreciate if you could expand on that, as Edi also uses
LOAD-TIME-VALUE in cl-ppcre. E.g., I'm not clear I could use
LOAD-TIME-VALUE in Iterate (which uses EVAL in one place...).
Regards,
Jorg Hohle
Telekom/T-Systems Technology Center
Joerg Hoehle <······@users.sourceforge.net> wrote:
+---------------
| ····@rpw3.org (Rob Warnock) writes:
| >Thank goodness there's LOAD-TIME-VALUE... ;-}
|
| I'd appreciate if you could expand on that, as Edi also uses
| LOAD-TIME-VALUE in cl-ppcre. E.g., I'm not clear I could use
| LOAD-TIME-VALUE in Iterate (which uses EVAL in one place...).
+---------------
I don't recall Iterate, so I'm not sure. But in CL-PPCRE it's
used to get around the very problem we were discussing -- symbols
defined with DEFCONSTANT can show up as CONSTANTP at compile time
(macro-expansion time, really) but the value might not be accessible
then. The use of LOAD-TIME-VALUE pushes the actual reference *and*
any possibly-expensive computation on it to load time, but ensures
that the calcuation is still only done once per LOAD. [Which is worse
than once per compilation, but still a lot better than, say, once
per trip around a tight loop].
Consider one of the uses in CL-PPCRE, and how Edi *might* have been
able to write it if the availability of the values of DEFCONSTANT
forms at compile time were mandated/guaranteed by the standard:
(define-compiler-macro split (&whole form
regex target-string &rest rest)
"Make sure that constant forms are compiled into scanners
at compile time."
(cond ((constantp regex)
`(split ,(create-scanner regex)
,target-string ,@rest))
(t form)))
If the arg REGEX were a constant, the above version would evaluate
(CREATE-SCANNER REGEX) at macro-expansion [compile] time and embed
the resulting complex object as a quasi-literal in the generated
SPLIT call. But as we know from this thread, the values of DEFCONSTANT
forms are *not* guaranteed to be accessible to compile time, which
is why Edi wrote it like this instead:
(define-compiler-macro split (&whole form
regex target-string &rest rest)
"Make sure that constant forms are compiled into scanners
at compile time."
(cond ((constantp regex)
`(split (load-time-value (create-scanner ,regex))
,target-string ,@rest))
(t form)))
Yes, this doesn't allow the CREATE-SCANNER call to occur at compile
time, but it *still* only calls it once, at load time, and not once
per call of SPLIT.
LOAD-TIME-VALUE can also work better than macro-expansion-time
evaluation when the functions being called in the LOAD-TIME-VALUE
expression don't exist at all in the compile time environment,
because of separate compilation, say, but *do* exist at load time
because of the order that the modules of the final application
are loaded.
Does that address your question adequately?
-Rob
-----
Rob Warnock <····@rpw3.org>
627 26th Avenue <URL:http://rpw3.org/>
San Mateo, CA 94403 (650)572-2607
····@rpw3.org (Rob Warnock) writes:
> I don't recall Iterate, so I'm not sure. But in CL-PPCRE it's
> used to get around the very problem we were discussing -- symbols
> defined with DEFCONSTANT can show up as CONSTANTP at compile time
> (macro-expansion time, really) but the value might not be accessible
> then. The use of LOAD-TIME-VALUE pushes the actual reference *and*
> any possibly-expensive computation on it to load time, but ensures
> that the calcuation is still only done once per LOAD. [Which is worse
> than once per compilation, but still a lot better than, say, once
> per trip around a tight loop].
This doesn't help in the situation where other compilation elements
(either parts of the compiler, or compiler-macros, whether system or
user-written) could have done more optimization on the surrounding
code if it had known what the symbol's constant value was. An example
of this is a hairy numeric expression that collapses to one term when
all the sub-terms are constants. Load-time-value forms help to elimnate
an obvious symbol-value runtime action, but many times the real
optimizations are much deeper.
As far as I know, all CL implementations know how to get at their
own constant-values for the compilation environment. However, since
it's not specified standardly, there is no portable way for, say,
user-written macros or compiler-macros to get at the value to perform
further optimizations. This was the reasoning for the addition of
sys:constant-value in the Environments Access module.
--
Duane Rettig ·····@franz.com Franz Inc. http://www.franz.com/
555 12th St., Suite 1450 http://www.555citycenter.com/
Oakland, Ca. 94607 Phone: (510) 452-2000; Fax: (510) 452-0182
····@rpw3.org (Rob Warnock) writes:
> Joerg Hoehle <······@users.sourceforge.net> wrote:
> +---------------
> | ····@rpw3.org (Rob Warnock) writes:
> | >Thank goodness there's LOAD-TIME-VALUE... ;-}
> |
> | I'd appreciate if you could expand on that, as Edi also uses
> | LOAD-TIME-VALUE in cl-ppcre. E.g., I'm not clear I could use
> | LOAD-TIME-VALUE in Iterate (which uses EVAL in one place...).
> +---------------
>
> I don't recall Iterate, so I'm not sure. But in CL-PPCRE it's
> used to get around the very problem we were discussing -- symbols
> defined with DEFCONSTANT can show up as CONSTANTP at compile time
> (macro-expansion time, really) but the value might not be accessible
> then. The use of LOAD-TIME-VALUE pushes the actual reference *and*
> any possibly-expensive computation on it to load time, but ensures
> that the calcuation is still only done once per LOAD. [Which is worse
> than once per compilation, but still a lot better than, say, once
> per trip around a tight loop].
>
> Consider one of the uses in CL-PPCRE, and how Edi *might* have been
> able to write it if the availability of the values of DEFCONSTANT
> forms at compile time were mandated/guaranteed by the standard:
>
> (define-compiler-macro split (&whole form
> regex target-string &rest rest)
> "Make sure that constant forms are compiled into scanners
> at compile time."
> (cond ((constantp regex)
> `(split ,(create-scanner regex)
> ,target-string ,@rest))
> (t form)))
>
> If the arg REGEX were a constant, the above version would evaluate
> (CREATE-SCANNER REGEX) at macro-expansion [compile] time and embed
> the resulting complex object as a quasi-literal in the generated
> SPLIT call.
I hope I'm not bringing something up that was dealt with upthread, but
even if REGEX were available at compile time, wouldn't this still be
busted since CREATE-SCANNER returns a closure which isn't
(necessarily) an externalizable object and thus couldn't (necessarily)
be embedded in the generated code?
-Peter
--
Peter Seibel * ·····@gigamonkeys.com
Gigamonkeys Consulting * http://www.gigamonkeys.com/
Practical Common Lisp * http://www.gigamonkeys.com/book/
From: Edi Weitz
Subject: Re: clisp: COMPILE-FILE needs two LOADs?
Date:
Message-ID: <uu0aqku3f.fsf@agharta.de>
On Thu, 23 Feb 2006 14:20:49 GMT, Peter Seibel <·····@gigamonkeys.com> wrote:
> I hope I'm not bringing something up that was dealt with upthread,
> but even if REGEX were available at compile time, wouldn't this
> still be busted since CREATE-SCANNER returns a closure which isn't
> (necessarily) an externalizable object and thus couldn't
> (necessarily) be embedded in the generated code?
Yes, I think you're right.
--
European Common Lisp Meeting 2006: <http://weitz.de/eclm2006/>
Real email: (replace (subseq ·········@agharta.de" 5) "edi")
Peter Seibel <·····@gigamonkeys.com> wrote:
+---------------
| ····@rpw3.org (Rob Warnock) writes:
| > If the arg REGEX were a constant, the above version would evaluate
| > (CREATE-SCANNER REGEX) at macro-expansion [compile] time and embed
| > the resulting complex object as a quasi-literal in the generated
| > SPLIT call.
|
| I hope I'm not bringing something up that was dealt with upthread, but
| even if REGEX were available at compile time, wouldn't this still be
| busted since CREATE-SCANNER returns a closure which isn't
| (necessarily) an externalizable object and thus couldn't (necessarily)
| be embedded in the generated code?
+---------------
Yes, of course. In writing the above, I was so closely focussed on
the macro-expansion-time versus load-time issue that I didn't think
to go check what kind of object CREATE-SCANNER returned. (Oops.)
-Rob
-----
Rob Warnock <····@rpw3.org>
627 26th Avenue <URL:http://rpw3.org/>
San Mateo, CA 94403 (650)572-2607
····@rpw3.org (Rob Warnock) writes:
[LOAD-TIME-VALUE]
> Does that address your question adequately?
Yes. Sadly you (or anybody else) did not yet respond to the other
half of my post about CONSTANTP and the null lexical environment.
I should have expected this. For years now I can't prevent myself
from putting two topics in sentences or usenet postings, typically
receiving a response only to one of the topics debated. :-)
I haven't read Duane Rettig's environment access proposal yet (and he
already mentioned it months ago on usenet), but I wonder how a
conforming implementation that does *not* evaluate DEFCONSTANT's
initial value can provide SYS:CONSTANT-VALUE?
http://www.lispwire.com/entry-proganal-envaccess-des
I.e., I'd expect this proposal to argue for
if (CONSTANTP foo <env) => (sys:constant-value foo <env>) is defined.
However, I got the impression that some people here expect clisp to yield
(defconstant +foo+ (some-computation))
(constantp '+foo+) -> T ; during e.g. macroexpansion while compiling
=> (sys:constant-value '+foo) -> <value of some computation>
Obviously, implementations legally not evaluating initial value cannot
respond to sys:constant-value. Did I miss something?
BTW, I went to the above URL, which links to
http://www.franz.com/support/documentation/7.0/doc/environments.htm
and found no mention there of constant-value. What's missing?
Regards,
Jorg Hohle
Telekom/T-Systems Technology Center
Joerg Hoehle <······@users.sourceforge.net> writes:
> ····@rpw3.org (Rob Warnock) writes:
> [LOAD-TIME-VALUE]
>> Does that address your question adequately?
>
> Yes. Sadly you (or anybody else) did not yet respond to the other
> half of my post about CONSTANTP and the null lexical environment.
It looks like you're giving an interpretation that there can be
only one "global environment" (after all, the spec does talk about
"_the_ global environment, doesn't it?". But that ignores the rest
of the descriptions of environments, which all work together to
provide a consistent model. First, the dichotomy of lexical environments
and global environments (the null lexical environment is the global
environment) provides a clue that all environments have two components
to them; a lexical component and a global one. And the phrase "null
lexical environment" (why do you suppose they didn't just use the phrase
"the global environment"?) which is notable mainly as its representation
as NIL in arguments where environments are allowed, simply means that the
environment referred to has no lexical component (or, a lexical component
that is null). This doesn't preclude there being multiple global
environments - look at the description of "The global environment" in
3.1.1.1 - it says "The global environment is _that_ part of _an_
environment that..." meaning that there is room for more than one of
these environments (with "the global environment" in each) at various
times. I think of compile-file's environment as an environment which
indeed has a global portion and which shadows the lisp's execution
environment.
If you can look at it that way, I think your concern goes away. Note
that since "the null lexical environment" is represented by nil, only
one of any existing global environments can be represented at a time
in this manner, but this is consistent with the idea of the compile-file
environment shadowing the normal run-time environment. The Environments
Access module represents the compile-file environment as a global variable
that can be bound during compile time (we changed its name in 8.0 and in
the lispwire module from sys:*compile-file-environment* to
sys:*compilation-unit-environment*, for reasons I won't go into here).
> I should have expected this. For years now I can't prevent myself
> from putting two topics in sentences or usenet postings, typically
> receiving a response only to one of the topics debated. :-)
Human nature :-)
> I haven't read Duane Rettig's environment access proposal yet (and he
> already mentioned it months ago on usenet), but I wonder how a
> conforming implementation that does *not* evaluate DEFCONSTANT's
> initial value can provide SYS:CONSTANT-VALUE?
Do you know of any lisps that don't do this evaluation? I don't -
competition forces the usage of constants by the compiler as often
and aggressively as possible. As I look back over this thread, it
seems to me that CMUCL was being discussed; that implementation most
certainly evaluates its defconstant form at compile-time (by a
function, surprisingly enough, named do-defconstant-compile-time).
CMUCL also has its own pseudo-environment access implementation in
the form of an operator called "info", which d-d-c-t uses and which
I use in the cursory implementation of Environments Access for cmucl.
> http://www.lispwire.com/entry-proganal-envaccess-des
>
> I.e., I'd expect this proposal to argue for
> if (CONSTANTP foo <env) => (sys:constant-value foo <env>) is defined.
Yes.
> However, I got the impression that some people here expect clisp to yield
> (defconstant +foo+ (some-computation))
> (constantp '+foo+) -> T ; during e.g. macroexpansion while compiling
> => (sys:constant-value '+foo) -> <value of some computation>
I don't know if clisp has added a sys:constant-value. But I do know that
its defconstant form has a :compile-toplevel eval-when component which
calls c-proclaim-constant, which in turn sets things up to use a function
(I assume it's internal) called c-constant-value. Note that this function
does not accept an environment argument, and assumes the compile-file
environment when operating. Also, it is designed to pair with the
c-constantp function, which has similar semantics.
As you can see, most lisps have all of the stuff environments are made
of; they just don't have the same interface for accessing them, and thus
none are portable.
> Obviously, implementations legally not evaluating initial value cannot
> respond to sys:constant-value. Did I miss something?
No. The Environments Access module makes some assumptions about what
CL compilers would be willing to do _as_ an _extension_. It doesn't
force any implementation to adhere to it, nor do I know of any CL
other than Allegro CL which has fully implemented it (although I have
feature conditionals in the source for all of cmu, clisp, lispworks,
and sbcl, as well as allegro, and the module works for most of these
lisps at what I call a Level 1 portability). I'm hoping that there
would be not much controversy with such assumptions, and I have
attempted to make it worth an implemetor's while to incorporate
the module, since it actually provides an exemplary definition
(with #+allegro currently) of constantp, as well as a suggested
implementation of sys:constant-value.
> BTW, I went to the above URL, which links to
> http://www.franz.com/support/documentation/7.0/doc/environments.htm
> and found no mention there of constant-value. What's missing?
The version. This stuff was not complete in its current form in 7.0.
Try http://www.franz.com/support/documentation/8.0/doc/environments.htm
instead, or even more directly through the 8.0 index to
http://www.franz.com/support/documentation/8.0/doc/operators/system/constant-value.htm
--
Duane Rettig ·····@franz.com Franz Inc. http://www.franz.com/
555 12th St., Suite 1450 http://www.555citycenter.com/
Oakland, Ca. 94607 Phone: (510) 452-2000; Fax: (510) 452-0182
[jumping into this thread midstream, so perhaps I'm missing some
context - but often when I see my name mentioned ...]
Joerg Hoehle <······@users.sourceforge.net> writes:
> About CONSTANTP + EVAL, I also found CLISP to be consistent: It won't
> yield CONSTANTP -> true during compilation, so user code will not call
> EVAL on it -- there's a lot of code out there, which does:
> (if (constantp form) (eval form))
> So clisp would not have caused you a ton of grief. ;)
Constantp and eval aren't really well-paired. constantp takes an
optional environment argument, so it can operate on any explicit
environment available, but eval doesn't take such an argument, and
is defined to operate in the null lexical environment (the dynamic
environment it will operate in is, well... dynamic).
Instead, there should be a constant-value function which can evaluate
a constant form under similar circumstances as constantp does. The
Environments Access module (http://www.lispwire.com/entry-proganal-envaccess-des)
specifies such a function: sys:constant-value, which also takes an
optional environment argument and which can be used to perform the
calculation portably for any implementation which supports Environments
Access. The module even has a definition (it is the definition which
Allegro CL uses) within its source.
> From a user POV, the compiler evaluating DEFCONSTANT during
> compilation seems best (in terms of least surprise). It's explicitly
> allowed by CLHS. But this choice has its quirks w.r.t. EQL-ness and
> redefinition at load-time (as cmucl/sbcl users know).
The split between compile-time and load-time would be less mysterious
if it were more controllable. That is what we sought to do within
the Environments Access module.
> So, OTOH, not evaluating defconstant during compilation seems very
> reasonable behaviour as well. Because using EVAL-WHEN, you can
> enforce evaluation: the user can choose. Edi Weitz is right to
> recommend using EVAL-WHEN to the OP.
But eval-when doesn't guarantee that you'll get the EQ constant at
load time, either. The times when it is a Good Thing to allow the
compiler to evaluate a constant at compile-time is if it can use
that information to perform further optimizations to the code -
especially in numerical situations. But the constant, though it must
be similar at both compile-time and load-time, might be regenerated at
load time, so as not to be EQ for the two phases. This is essential;
how do we even know that the compiling lisp is the same instance as
the loading lisp? They may not only be not EQ, but not even in
the same lisp universe!
> Note that (if (constantp form) (eval form)) is questionable code
> probably, but so far, there has been no standardization on access to the
> compilation environment. And I have not yet found time to look at what
> Duane Rettig proposed in that area.
See above. No standardization, which is why I've proposed some
possible de-facto standardization by way of Environments Access.
> IMHO it's debatable whether (if (constantp form) (eval form)) should
> be flagged as unportable (even broken) code?? CONSTANTP could then
> return T in clisp. But what would you do knowing that, if you can't
> get to the value?
There are many non-portable code segments that don't get flagged. I agree
that (if (constantp form) (eval form)) is non-portable, but the alternative:
(if (constantp form env) (sys:constant-value form env)) would be portable
amongst implementations that supported Environments Access.
> How do cmucl or sbcl in CL-PPCRE manage to make use of the values of
> the constants +func-def+ and +echo-arg+ in the OP's example (do
> they)?? AFAIK, SBCL is known not to provide the value via EVAL.
Left to SBCL experts to answer.
--
Duane Rettig ·····@franz.com Franz Inc. http://www.franz.com/
555 12th St., Suite 1450 http://www.555citycenter.com/
Oakland, Ca. 94607 Phone: (510) 452-2000; Fax: (510) 452-0182