Could anyone share experiences or advice on transforming large volumes
of structured data?
I am writing one of those common glue programs that parses, validates, and
transforms structured data from one form to another. You know-- a
compiler.
However, the transformations end up filtering and sorting records in ways
reminiscent of database queries. I am thinking of moving from querying
based on simple path language to using a more general framework based on
some kind of logic or constraint language (paiplog, screamer, or even qi).
I have just started the project, so I not really committed either way at
this point.
Matt
--
"You do not really understand something unless you
can explain it to your grandmother." - Albert Einstein.
Matthew Swank wrote:
> Could anyone share experiences or advice on transforming large volumes
> of structured data?
>
> I am writing one of those common glue programs that parses, validates, and
> transforms structured data from one form to another. You know-- a
> compiler.
>
> However, the transformations end up filtering and sorting records in ways
> reminiscent of database queries. I am thinking of moving from querying
> based on simple path language to using a more general framework based on
> some kind of logic or constraint language (paiplog, screamer, or even qi).
>
> I have just started the project, so I not really committed either way at
> this point.
Yeah, you are committed. You are committed to running screaming from
your problem to some packaged "solution", a word Apple Computer turned
into a joke. STFU and get back to work. Those solutions should only
guide your understanding and your solution, which will map infinitely
better to the problem, will be fully known to you because you will have
created it, and for the same reason will not have a learning curve nor
will it force you into compromises.
Jeez, does no one understand Lisp?
kenny
--
http://www.theoryyalgebra.com/
"Algebra is the metaphysics of arithmetic." - John Ray
"As long as algebra is taught in school,
there will be prayer in school." - Cokie Roberts
"Stand firm in your refusal to remain conscious during algebra."
- Fran Lebowitz
"I'm an algebra liar. I figure two good lies make a positive."
- Tim Allen
On Wed, 01 Aug 2007 01:15:47 -0400, Ken Tilton wrote:
> Yeah, you are committed. You are committed to running screaming from
> your problem to some packaged "solution", a word Apple Computer turned
> into a joke. STFU and get back to work. Those solutions should only
> guide your understanding and your solution, which will map infinitely
> better to the problem, will be fully known to you because you will have
> created it, and for the same reason will not have a learning curve nor
> will it force you into compromises.
>
> Jeez, does no one understand Lisp?
>
> kenny
You seem tense.
--
"You do not really understand something unless you
can explain it to your grandmother." - Albert Einstein.
On 1 Aug., 15:29, Matthew Swank
<································@c.net> wrote:
> On Wed, 01 Aug 2007 01:15:47 -0400, Ken Tilton wrote:
> > Jeez, does no one understand Lisp?
>
> > kenny
>
> You seem tense.
no, that's just his usual frustration ;)
Christopher Koppler <········@chello.at> writes:
> On 1 Aug., 15:29, Matthew Swank
> <································@c.net> wrote:
>> On Wed, 01 Aug 2007 01:15:47 -0400, Ken Tilton wrote:
>> > Jeez, does no one understand Lisp?
>>
>> > kenny
>>
>> You seem tense.
>
> no, that's just his usual frustration ;)
Why do you say that's just his usual frustration?
Zach
Christopher Koppler <········@chello.at> writes:
> On 1 Aug., 15:50, Zach Beane <····@xach.com> wrote:
>> Why do you say that's just his usual frustration?
>
> he's a teacher, c.l.l.ers won't learn
Perhaps this has something to do with to your attitude.
Zach
On Aug 1, 4:10 pm, Zach Beane <····@xach.com> wrote:
> Christopher Koppler <········@chello.at> writes:
> > On 1 Aug., 15:50, Zach Beane <····@xach.com> wrote:
> >> Why do you say that's just his usual frustration?
>
> > he's a teacher, c.l.l.ers won't learn
>
> Perhaps this has something to do with to your attitude.
seems next time I'll have to write the ;) in a bigger font
Matthias Benkard <··········@gmail.com> wrote:
+---------------
| > seems next time I'll have to write the ;) in a bigger font
|
| Do you really think that?
+---------------
We were talking about you, not me...
On Wed, 01 Aug 2007 01:15:47 -0400, Ken Tilton wrote:
> Yeah, you are committed. You are committed to running screaming from
> your problem to some packaged "solution", a word Apple Computer turned
> into a joke. STFU and get back to work.
This seems an odd statement coming from Mr. "Cells Inside".
Embedded languages are tools. I am constantly pulled toward homegrown,
idiosyncratic solutions by the "not invented here" syndrome. However, why
reinvent SQL, or Prolog (or even Cells) when there are well tested,
debugged, and optimized "solutions" out there?
Matt
--
"You do not really understand something unless you
can explain it to your grandmother." - Albert Einstein.
On Aug 1, 3:24 am, Matthew Swank
<································@c.net> wrote:
> Could anyone share experiences or advice on transforming large volumes
> of structured data?
>
> I am writing one of those common glue programs that parses, validates, and
> transforms structured data from one form to another. You know-- a
> compiler.
>
> However, the transformations end up filtering and sorting records in ways
> reminiscent of database queries. I am thinking of moving from querying
> based on simple path language to using a more general framework based on
> some kind of logic or constraint language (paiplog, screamer, or even qi).
>
> I have just started the project, so I not really committed either way at
> this point.
If it's like a compiler, use some sort of declarative framework. KR,
Cells, Allegro Prolog, whatever. Actually, maybe not Prolog -- cut is
a pretty awkward way to get laziness into the system, and laziness can
be important for controlling your big-O behavior.
If you look at the amount of code that's dedicated to trying to keep
things consistent in compilers written in pretty civilized languages
like Lisp or SML, it's shocking -- all that effort, and they still
screw it up! It's like looking from the vantage point of Lisp at a
compiler written in C or Pascal. A whole class of problems, a whole
class of bugs, for no particular reason.
So yeah, pick one. Compared to the alternative, it's really not
important which one you pick.