Hello, All!
what's best way to write macros that require complex state for each
macroexpanded instance?
i've expain what do i mean :). imagine DSL used to do SQL-like queries.
like
(sql:select [a] (from [t]) (where [j] < my-j))
simple solution would be to create text template any then just paste local
parameters into it. but it's dull :).
first, i'd like SQL DSL to be able to connect to SQL server and be able to
verify that SQL expression is correct according to current database schema
(all table names and fields are correct, etc) and even macroexpand according
to database schema (for example, get type info from it). this should be done
at macroexpansion time, i think, so SQL errors would be reported during
compilation of sources.
second, SQL query should be able to cache query planning information, so
planning (that might be time-consuming operation) is performed less
frequently that quering itself. for example, it can be functionality of
foreign SQL library, that will produce some compiled-sql when it's fed
text-sql. certainly we won't be able to serialize compiled-sql during
compilation of sources, and it might be not efficient (query planning
depends on data currently in database, that changes over time), so i think
SQL should be compiled during load time or at first query.
what do people think on how to do it's best?
i think it should to macroexpand to something like this:
(let ((ctrl-fun (let ((persistent-hidden-state))
(lambda () (do-something-with persistent-hidden-state)))))
(funcall ctrl-fun))
where variable names, of course, should be gensym'ed. any better
suggestions?
With best regards, Alex Mizrahi.
Alex Mizrahi wrote:
> Hello, All!
>
> what's best way to write macros that require complex state for each
> macroexpanded instance?
>
> i've expain what do i mean :). imagine DSL used to do SQL-like queries.
> like
>
> (sql:select [a] (from [t]) (where [j] < my-j))
>
> simple solution would be to create text template any then just paste local
> parameters into it. but it's dull :).
>
> first, i'd like SQL DSL to be able to connect to SQL server and be able to
> verify that SQL expression is correct according to current database schema
> (all table names and fields are correct, etc) and even macroexpand according
> to database schema (for example, get type info from it). this should be done
> at macroexpansion time, i think, so SQL errors would be reported during
> compilation of sources.
This sounds like an expensive computation which you probably do not want
to do at macroexpansion time. The reason is that you don't know how
often your macros will be expanded. First, a compiler is allowed to
expand a macro invocation multiple times, for example in order to
perform analysis in several steps. Second, a user might expand such
invocations for debugging purposes.
The "right" way to do this is to expand into code that then performs the
necessary computations. It is possible to specify that such code is
executed at compile time, and this is probably what you want here.
Look up eval-when and the compilation model in the HyperSpec or in the
respective sections of your preferred tutorials.
> second, SQL query should be able to cache query planning information, so
> planning (that might be time-consuming operation) is performed less
> frequently that quering itself. for example, it can be functionality of
> foreign SQL library, that will produce some compiled-sql when it's fed
> text-sql. certainly we won't be able to serialize compiled-sql during
> compilation of sources, and it might be not efficient (query planning
> depends on data currently in database, that changes over time), so i think
> SQL should be compiled during load time or at first query.
>
> what do people think on how to do it's best?
>
> i think it should to macroexpand to something like this:
>
> (let ((ctrl-fun (let ((persistent-hidden-state))
> (lambda () (do-something-with persistent-hidden-state)))))
> (funcall ctrl-fun))
>
> where variable names, of course, should be gensym'ed. any better
> suggestions?
No, caching at runtime or via load-time-value is typically a good approach.
Pascal
--
My website: http://p-cos.net
Common Lisp Document Repository: http://cdr.eurolisp.org
Closer to MOP & ContextL: http://common-lisp.net/project/closer/
On Sun, 24 Sep 2006 17:10:55 +0300, "Alex Mizrahi"
<········@users.sourceforge.net> wrote:
>first, i'd like SQL DSL to be able to connect to SQL server and be able to
>verify that SQL expression is correct according to current database schema
>(all table names and fields are correct, etc) and even macroexpand according
>to database schema (for example, get type info from it). this should be done
>at macroexpansion time, i think, so SQL errors would be reported during
>compilation of sources.
I guess the first question is: do you intend your DSL for end users or
for application programmers? And then how much flexibility do you
intend to offer?
I'm not aware of any SQL library that can check query syntax without
being online to the database - query syntax is ad hoc with respect to
the database schema so there is little that can be checked offline.
Opinion: it's a real pain to require a remote database to be available
just to compile client code. Makes it hard to hack at the beach on
the laptop.
>second, SQL query should be able to cache query planning information, so
>planning (that might be time-consuming operation) is performed less
>frequently that quering itself. for example, it can be functionality of
>foreign SQL library, that will produce some compiled-sql when it's fed
>text-sql. certainly we won't be able to serialize compiled-sql during
>compilation of sources, and it might be not efficient (query planning
>depends on data currently in database, that changes over time), so i think
>SQL should be compiled during load time or at first query.
There are (at least) 4 distinct notions of "compiling" here ...
compiling your DSL query to a SQL text query, compiling the Lisp
program, preparing the query, and finally executing the query.
How a SQL query is compiled/optimized depends on connection specific
details, therefore it is compiled on demand when first executed. This
is done by the SQL library. "Preparing" a query checks syntax and
creates internal structures for binding result fields to program
variables but usually does not actually compile/optimize the query -
that is done on the execute. Whether the compiled form of the query
is retained across non-consecutive executions depends on the library.
>what do people think on how to do it's best?
The only part you can really affect at Lisp compile time is generation
of the SQL query text. Binding and query compilation have to be done
at runtime.
All that is really available to cache is some kind of statement handle
to the "prepared" query and the program variables bound to the result
fields. I usually prefer to create an object or structure to bundle
the field variables, but a closure works just as well. Keep in mind
that you may have to pass the statement handle to several different
SQL functions (create,prepare,bind,execute,destroy, etc.).
Of course, what makes sense to do will depend on what functionality
the SQL library gives you - I usually work at low level with ODBC.
YMMV.
George
--
for email reply remove "/" from address
George Neuner <·········@comcast.net> writes:
> There are (at least) 4 distinct notions of "compiling" here ...
> compiling your DSL query to a SQL text query, compiling the Lisp
> program, preparing the query, and finally executing the query.
Indeed.
> The only part you can really affect at Lisp compile time is generation
> of the SQL query text. Binding and query compilation have to be done
> at runtime.
Not necessarily, see below.
> All that is really available to cache is some kind of statement handle
> to the "prepared" query and the program variables bound to the result
> fields.
You may wish to look at CLISP's regexp module, even if you're not
using CLISP. It does memoization of regular expression queries (which
the C regexp library needs time to compile to a regexp virtual
machine), so the regexp is not compiled twice.
Of course, you need to set the details:
- at what time to call the statement compiler,
- which queries are run multiple times so as to benefit from
prepared statements.
- how to keep a pointer to the Lisp object representing that
in the source code.
Once you've set up your mind, it should become straightforward.
Long ago, I did a sort of two-phase processing: compilation of a Lisp
file did an anylsis of the query macros to know which columns where
used. Compilation of a second Lisp file afterwards produced code to
optimize these precise queries. The source code of that second file
was a one-liner, a macro invocation:
(generate-query-aids)
Note that instead of using prepared statements to make multiple
queries faster, but query once slower, you could devise a scheme where
you create instructions for stored procedures into your database as a
side effect of compilation of Lisp code. Your Lisp code expands to
calls to these procedures.
Voil�, all your queries are fast, including all once queries!
Regards,
Jorg Hohle
Telekom/T-Systems Technology Center
On 04 Oct 2006 17:40:08 +0200, Joerg Hoehle
<······@users.sourceforge.net> wrote:
>George Neuner <·········@comcast.net> writes:
>
>> The only part you can really affect at Lisp compile time is generation
>> of the SQL query text. Binding and query compilation have to be done
>> at runtime.
>
>Not necessarily, see below.
I believe you are wrong about both.
Binding query fields to program variables in SQL is an environment and
connection dependent operation - even when the SQL is embedded binding
cannot be done at compile time.
As for query compilation ...
>> All that is really available to cache is some kind of statement handle
>> to the "prepared" query and the program variables bound to the result
>> fields.
>
>You may wish to look at CLISP's regexp module, even if you're not
>using CLISP. It does memoization of regular expression queries (which
>the C regexp library needs time to compile to a regexp virtual
>machine), so the regexp is not compiled twice.
>
>Of course, you need to set the details:
>- at what time to call the statement compiler,
>- which queries are run multiple times so as to benefit from
> prepared statements.
>- how to keep a pointer to the Lisp object representing that
> in the source code.
>Once you've set up your mind, it should become straightforward.
>
>Long ago, I did a sort of two-phase processing: compilation of a Lisp
>file did an anylsis of the query macros to know which columns where
>used. Compilation of a second Lisp file afterwards produced code to
>optimize these precise queries. The source code of that second file
>was a one-liner, a macro invocation:
>(generate-query-aids)
It's unclear exactly what you were doing but it does not seem that you
were compiling SQL into a form directly meaningful to a query engine.
Rather it seems that you were doing some automated textual query
tweaking based on compile time structural knowledge of the database.
Which is perfectly fine ... but it should not be confused with SQL
compilation.
The query engine performs a JIT compilation/optimization of the query
at first issue. The compiled query exists only in the engine and is
associated with a runtime data structure on the client.
Now, there are file based, library implementations of SQL in which the
line is blurred between client and server/engine, but in general the
above model holds - the client typically has no access the compiled
form of the query.
>Note that instead of using prepared statements to make multiple
>queries faster, but query once slower, you could devise a scheme where
>you create instructions for stored procedures into your database as a
>side effect of compilation of Lisp code. Your Lisp code expands to
>calls to these procedures.
Yes. Stored procedures are a method of pre-compiling SQL in a
meaningful way - at least for the stored procedure. Note, however,
that binding and compilation of the query which invokes the stored
procedure are both still dynamic events.
There are some downsides to using stored procedures:
1) Many systems do not allow stored procedures to return tables - you
need to issue a separate view query to see the results of the
processing.
2) The language of stored procedures is not standardized - it is
engine dependent so if you rehost your DB on a new engine you have to
recreate all your procedures - which might not even be possible if the
procedures made use of proprietary features.
3) Deploying an application that requires stored procedures can be a
problem because many security conscious DB engines require each user
to have her own private copy of the procedure. This means that you
can't just create them at compile time in your programmer account and
expect someone else to be able to use them. This puts a burden on the
administrator to create them for everyone and requires the developer
to provide the procedures in source form and installation scripts or
instructions along with the application.
4) Ditto updating procedures for a new application release.
5) Many systems have limits on the number and size of user procedures.
Depending on how many different applications the user must use, it may
be impossible to use stored procedures for everything.
With the exception of trigger procedures which I deem a necessary evil
in a multi-user environment, I am not a fan of generally using stored
procedures - they tend to cause as many problems as they solve.
George
--
for email reply remove "/" from address