Suppose you have a game that stores a lot of data in text files that
don't change very often. These text files contain various statistics
and perhaps other stuff that a rules engine would process. Would it
be such a terrible thing to reformat those files into Lisp? The
quantity of data I am talking about is less than 10MB worth of text.
A lot of it is strings, but there are numbers in there as well. I
don't think it would be a memory killer.
The alternative is to read in the files each time the program is
started. That will result in about the same memory usage. If some
bit of data needs to be changed, the appropriate file can be edited
and reloaded either way, right?
What I see when I look at lisp is data and code looking the same.
This seems to be useful beyond making macros.
I'm also thinking that if I format the data correctly, I can dump it
back out into Lisp files. This would be useful if a programmatic
allowed editing of the data or even adding new rules or even complete
rule sets.
Is it so bad to want Lisp to take on the task of parsing these files
(once I get them translated)? They are in some funky format
unfortunately, so more effort is required than parsing XML because the
parser for these data files is burried in a Java program.
I would love to say what the name of the Java program is, but while
the program is available (it's not mine) under a free license, the
data files are not. Don't worry, I'm not planing on redistribution.
--
I wouldn't mind the rat race so much if it wasn't for all the damn cats.
David Steuber <·····@david-steuber.com> writes:
> Suppose you have a game that stores a lot of data in text files that
> don't change very often. These text files contain various statistics
> and perhaps other stuff that a rules engine would process. Would it
> be such a terrible thing to reformat those files into Lisp? The
> quantity of data I am talking about is less than 10MB worth of text.
> A lot of it is strings, but there are numbers in there as well. I
> don't think it would be a memory killer.
Well if the files are not so large and won't change often. Why not
just dump all the information with your lisp-image?
Regards
Friedrich
--
Please remove just-for-news- to reply via e-mail.
Friedrich Dominicus <···················@q-software-solutions.de> writes:
> David Steuber <·····@david-steuber.com> writes:
>
> > Suppose you have a game that stores a lot of data in text files that
> > don't change very often. These text files contain various statistics
> > and perhaps other stuff that a rules engine would process. Would it
> > be such a terrible thing to reformat those files into Lisp? The
> > quantity of data I am talking about is less than 10MB worth of text.
> > A lot of it is strings, but there are numbers in there as well. I
> > don't think it would be a memory killer.
>
> Well if the files are not so large and won't change often. Why not
> just dump all the information with your lisp-image?
I could do that, but I think it is just as easy to let the Lisp files
be compiled into fasl files. If I have to change a lisp file,
reloading it should make a new fasl out of it, right? Or do I have to
explicitly call compile-file on it? Either way, it shouldn't be hard.
I've been looking at the data files and they are in some funky format
that almost makes me think they are some very primative scripting
language. I believe the result of reading them in is to instantiate
Java classes.
I'm also not so confident about reverse engineering the data. I may
have to create it from scratch. Either way, it seems that lisp is a
good file format to use.
--
I wouldn't mind the rat race so much if it wasn't for all the damn cats.
David Steuber wrote:
> Suppose you have a game that stores a lot of data in text files that
> don't change very often. These text files contain various statistics
> and perhaps other stuff that a rules engine would process. Would it
> be such a terrible thing to reformat those files into Lisp? The
> quantity of data I am talking about is less than 10MB worth of text.
> A lot of it is strings, but there are numbers in there as well. I
> don't think it would be a memory killer.
>
> The alternative is to read in the files each time the program is
> started. That will result in about the same memory usage. If some
> bit of data needs to be changed, the appropriate file can be edited
> and reloaded either way, right?
>
> What I see when I look at lisp is data and code looking the same.
> This seems to be useful beyond making macros.
>
> I'm also thinking that if I format the data correctly, I can dump it
> back out into Lisp files. This would be useful if a programmatic
> allowed editing of the data or even adding new rules or even complete
> rule sets.
>
> Is it so bad to want Lisp to take on the task of parsing these files
> (once I get them translated)? They are in some funky format
> unfortunately, so more effort is required than parsing XML because the
> parser for these data files is burried in a Java program.
Let's put it like this:
You will neccessarily need a parser of some sort to read the data into
Lisp. At this point it should be possible to print it back out with
(print), the result of which should be reasonably readable.
Getting easily read datafiles is thus easy. However, your Lisp
implementation is likely to represent the data in some more compact form
than ASCII in memory, so it would save space to save a Lisp core with
your data preloaded, and redo it whenever something changes.
(As an aside, is there a binary version of (print) around? I suppose that
would be implementation-dependent, but I really need a portable
variant... maybe I'll write one.)
Svein Ove Aas <··············@brage.info> writes:
> (As an aside, is there a binary version of (print) around? I suppose that
> would be implementation-dependent, but I really need a portable
> variant... maybe I'll write one.)
What do you mean by that? The question isn't fully formed. Depending
on what you mean, you might be able to use your implementation's FASL
writer.
--
/|_ .-----------------------.
,' .\ / | No to Imperialist war |
,--' _,' | Wage class war! |
/ / `-----------------------'
( -. |
| ) |
(`-. '--.)
`. )----'
Thomas F. Burdick wrote:
> Svein Ove Aas <··············@brage.info> writes:
>
>> (As an aside, is there a binary version of (print) around? I suppose
>> that would be implementation-dependent, but I really need a portable
>> variant... maybe I'll write one.)
>
> What do you mean by that? The question isn't fully formed. Depending
> on what you mean, you might be able to use your implementation's FASL
> writer.
>
I'm looking to sling data around the internet, between possibly several
different Lisp implementations - possibly on several architectures.
I *can* use (print) and (safe-eval) to do it, but sending ASCII, while
being nice for debugging, isn't the most efficient way to go about
things. I'm looking for something like Python's serialization mechanism;
that has a simple flag you can throw to make it use binary data.
Svein Ove Aas wrote:
> Thomas F. Burdick wrote:
>
>> Svein Ove Aas <··············@brage.info> writes:
>>
>>> (As an aside, is there a binary version of (print) around? I suppose
>>> that would be implementation-dependent, but I really need a portable
>>> variant... maybe I'll write one.)
>>
>> What do you mean by that? The question isn't fully formed. Depending
>> on what you mean, you might be able to use your implementation's FASL
>> writer.
>>
>
> I'm looking to sling data around the internet, between possibly several
> different Lisp implementations - possibly on several architectures.
>
> I *can* use (print) and (safe-eval) to do it, but sending ASCII, while
^^^^
That's 'read', of course.
>>>>> "Svein" == Svein Ove Aas <··············@brage.info> writes:
Svein> I'm looking to sling data around the internet, between
Svein> possibly several different Lisp implementations - possibly
Svein> on several architectures.
Svein> I *can* use (print) and (safe-eval) to do it, but sending
Svein> ASCII, while being nice for debugging, isn't the most
Svein> efficient way to go about things. I'm looking for something
Svein> like Python's serialization mechanism; that has a simple
Svein> flag you can throw to make it use binary data.
What about simply compressing the ASCII before sending it? It should
be fairly quick and portable, depending on how you do it. That way,
you still have the more general text version to load back up.
Tim Lavoie wrote:
>>>>>> "Svein" == Svein Ove Aas <··············@brage.info> writes:
>
> Svein> I'm looking to sling data around the internet, between
> Svein> possibly several different Lisp implementations - possibly
> Svein> on several architectures.
>
> Svein> I *can* use (print) and (safe-eval) to do it, but sending
> Svein> ASCII, while being nice for debugging, isn't the most
> Svein> efficient way to go about things. I'm looking for something
> Svein> like Python's serialization mechanism; that has a simple
> Svein> flag you can throw to make it use binary data.
>
> What about simply compressing the ASCII before sending it? It should
> be fairly quick and portable, depending on how you do it. That way,
> you still have the more general text version to load back up.
I do that, and while it *does* work, it still isn't as efficient as using
binary data; that data can be compressed too.
Having text around from non-debugging connections isn't a big deal, as I'd
always have the converter available. I don't see any reason the binary
format should be less general than the text format either; it's still
lists, right?
Writing my own shouldn't be too hard. I'll come up with something.
On 05/15/2004 12:10 AM, in article ··············@david-steuber.com, "David
Steuber" <·····@david-steuber.com> wrote:
> Suppose you have a game that stores a lot of data in text files that
> don't change very often. These text files contain various statistics
> and perhaps other stuff that a rules engine would process. Would it
> be such a terrible thing to reformat those files into Lisp? The
> quantity of data I am talking about is less than 10MB worth of text.
> A lot of it is strings, but there are numbers in there as well. I
> don't think it would be a memory killer.
>
> The alternative is to read in the files each time the program is
> started. That will result in about the same memory usage. If some
> bit of data needs to be changed, the appropriate file can be edited
> and reloaded either way, right?
>
> What I see when I look at lisp is data and code looking the same.
> This seems to be useful beyond making macros.
>
> I'm also thinking that if I format the data correctly, I can dump it
> back out into Lisp files. This would be useful if a programmatic
> allowed editing of the data or even adding new rules or even complete
> rule sets.
>
> Is it so bad to want Lisp to take on the task of parsing these files
> (once I get them translated)? They are in some funky format
> unfortunately, so more effort is required than parsing XML because the
> parser for these data files is burried in a Java program.
>
> I would love to say what the name of the Java program is, but while
> the program is available (it's not mine) under a free license, the
> data files are not. Don't worry, I'm not planing on redistribution.
I read/write objects this way all the time ... and they are much, much
larger than 10MB!
Storing stuff in a structured format is what XML is all about; an XML parser
is just a crude imitation of the Lisp reader.
We store engineering object definitions in files for use by several
ICAD-based applications (ICAD uses Allegro Common Lisp's flavor model for
generating its "parts"). For example, the parameterized surface model of an
aircraft ... fuselage, wings, and all. Simply representing a wing surface as
a data structure or closure that tells you exactly how to build itself is
easier than requiring the "reader" to know what to do with a lot of
disassociated data.
The cool part about this is that the guy responsible for generating the OML
of the aircraft fuselage can change the representation of the fuselage model
whenever he wants ... and it's transparent to his customers.
sl