I'm a Lisp newby, and I'm trying to figure out some design issues for
some code I am going to write.
I am writing some scientific code that should take a little while to
run, so I want to be able to write out the data in some way that is
easily read in. I could save an image, but that doesn't seem like a
clean, especially since I will have many runs that I want to save. In
python, I would probably use the "pickle" package to serialize my data
structures, but that isn't availible here.
What I was thinking it would be neat is to have my scientific code
write a lisp program that can be loaded to load all my data.
In other words my science code would create a file like
data1.lisp:
"""
(in-package :data)
(defparameter *beta* 0.125)
(defparameter *greens-function* #(1.000 0.7342 0.5242 0.223))
;; And so on with more compicated data structures...
"""
then a month later in a different lisp process I could `(load
"data1.lisp")' to get my results back.
Anyone tried anything like this? Any pitfalls I should watch out for?
Anything better?
Thanks,
Robbie
········@yahoo.com (Robbie Sedgewick) writes:
> I'm a Lisp newby, and I'm trying to figure out some design issues
> for some code I am going to write.
>
> I am writing some scientific code that should take a little while to
> run, so I want to be able to write out the data in some way that is
> easily read in. I could save an image, but that doesn't seem like a
> clean, especially since I will have many runs that I want to save.
> In python, I would probably use the "pickle" package to serialize my
> data structures, but that isn't availible here.
Well, you may be overlooking the Lisp equivalent since it's so
naturally built in. Look up READ and PRINT. Most lisp data structures,
including user defined structs and classes, can (or can be made to) be
printed in a form that the Lisp reader can naturally read back in.
(Hashtables, are one of the few data structures that you might want to
save that are not required to be printable readably though may be in a
given Lisp implementation.)
> What I was thinking it would be neat is to have my scientific code
> write a lisp program that can be loaded to load all my data.
>
> In other words my science code would create a file like
>
> data1.lisp:
> """
> (in-package :data)
>
> (defparameter *beta* 0.125)
>
> (defparameter *greens-function* #(1.000 0.7342 0.5242 0.223))
>
> ;; And so on with more compicated data structures...
> """
>
> then a month later in a different lisp process I could `(load
> "data1.lisp")' to get my results back.
That would be one way.
> Anyone tried anything like this? Any pitfalls I should watch out
> for? Anything better?
If you don't need to set specific parameters (such as *beta*, etc.)
you can probably just create a data structure and save it to a file
with PRINT. If all the data you want to save can be put into a single
data structure the following two functions might be all you need.
(defun save-data (data)
(with-open-file (out filename :direction :output)
(print data out)))
(defun load-data ()
(with-open-file (in filename)
(read in)))
If you actually have a lot of independent pieces of data, you might
consider defining a structure with DEFSTRUCT that you can put all the
pieces in and then save that. (That would also give you a chance to
attach various pieces of meta-data to it, such as the date it was
saved, etc.)
I don't know if this approach is "better" than your idea but it seems
a bit cleaner to save data as data rather than as code that recreates
the data. YMMV.
-Peter
--
Peter Seibel ·····@javamonkey.com
Lisp is the red pill. -- John Fraser, comp.lang.lisp
On Wed, 25 Jun 2003 22:50:52 GMT, Peter Seibel wrote:
> If you don't need to set specific parameters (such as *beta*, etc.)
> you can probably just create a data structure and save it to a file
> with PRINT. If all the data you want to save can be put into a single
> data structure the following two functions might be all you need.
> (defun save-data (data)
> (with-open-file (out filename :direction :output)
> (print data out)))
> (defun load-data ()
> (with-open-file (in filename)
> (read in)))
But preferably wrap a WITH-STANDARD-IO-SYNTAX around the PRINT and READ.
--
Just because we Lisp programmers are better than everyone else is no
excuse for us to be arrogant. -- Erann Gat
(setq reply-to
(concatenate 'string "Paul Foley " "<mycroft" '(··@) "actrix.gen.nz>"))
Robbie Sedgewick wrote:
> What I was thinking it would be neat is to have my scientific code
> write a lisp program that can be loaded to load all my data....
> Anyone tried anything like this? Any pitfalls I should watch out for?
> Anything better?
I've done it fairly often. It has the advantage over some of the
approaches mentioned by others that you don't have to write a special
data reader. If you decide you want to save a different data set, you
just change what you write out.
Pitfalls:
* Make sure no one can tamper with your file, because once you start
evaluating everything in it you could wind up with all your files
deleted. (I mention this in the interest of paranoia; it's not really a
serious concern in most environments.)
* Think about what printer-control variables you want to bind before you
print things to the data file. If *print-level* is 2, the data file
might contain expressions such as (setq important-data* '(# # #)).
* If you're writing circular data structures out, or you want to
preserve sharing relationships among data structures, then you have to
be very careful. (How _does_ Java solve this problem just by making up
the word "serialize"?)
* If you're writing really humongous data structures out, then you
should really be doing binary I/O.
-- Drew McDermott
In article <············@news.wss.yale.edu>, Drew McDermott
<··················@at.yale.dot.edu> wrote:
> * If you're writing circular data structures out, or you want to
> preserve sharing relationships among data structures, then you have to
> be very careful. (How _does_ Java solve this problem just by making up
> the word "serialize"?)
No one ever creates circular structures in Java. (Nor Python.)
;-)
E.
On Wed, Jun 25, 2003 at 10:36:42PM -0700, Erann Gat wrote:
> No one ever creates circular structures in Java. (Nor Python.)
Of course not. Not allowing circular structures prevents circular
reasoning, the lack of which will not permit circular structures.
Thus is a whole class of errors avoided at compile-time and even at
thinking-time, because the programmer cannot comprehend what is not
comprehensible to him, so you see.
--
; Matthew Danish <·······@andrew.cmu.edu>
; OpenPGP public key: C24B6010 on keyring.debian.org
; Signed or encrypted mail welcome.
; "There is no dark side of the moon really; matter of fact, it's all dark."
Matthew Danish <·······@andrew.cmu.edu> wrote in message news:<······················@lain.mapcar.org>...
> On Wed, Jun 25, 2003 at 10:36:42PM -0700, Erann Gat wrote:
> > No one ever creates circular structures in Java. (Nor Python.)
>
> Of course not. Not allowing circular structures prevents circular
> reasoning, the lack of which will not permit circular structures.
>
> Thus is a whole class of errors avoided at compile-time and even at
> thinking-time, because the programmer cannot comprehend what is not
> comprehensible to him, so you see.
Ok, I'll bite. I know what a circular structure is and how to go
about creating one, but it's definitely not in my toolkit. I've never
run across one in Lisp code I've read nor have I used one.
When would be a good time to use one? To what types of problems do
they map well? Where do they give good leverage?
Curious,
Chris Perkins
Media Lab, Inc
From: Thomas A. Russ
Subject: Re: Storing data in lisp code - circular structures
Date:
Message-ID: <ymi7k78tzx6.fsf@sevak.isi.edu>
········@medialab.com (Chris Perkins) writes:
>
> When would be a good time to use one? To what types of problems do
> they map well? Where do they give good leverage?
Well, I don't normally use them, so I'm not sure about good leverage,
but I do have at least a cute example of their usefulness. I suppose
the serious answer is when you have naturally circular data structures
such as recurring sequences, cyclic graph structures, doubly linked
lists, etc.
(setq *print-circle* t) ;; Otherwise this gets a bit messy.
(defvar *days* '#1=(monday tuesday wednesday thursday friday
saturday sunday . #1#))
;; Now you can do some fun things with this:
(member 'friday *days*)
;; And use it to build another cute little function:
(defun days-of-month (n-days starting-day)
;; Print out the date and day of the week for a
;; month of N-DAYS starting on STARTING-DAY
(loop for d from 1 to n-days
as day in (member starting-day *days*)
do (format t "The ~:R is a ~:(~A~).~%" d day)))
;; Then try calling, i.e. for June & July 2003:
(days-of-month 30 'sunday)
(days-of-month 31 'tuesday)
--
Thomas A. Russ, USC/Information Sciences Institute
From: Steven M. Haflich
Subject: Re: Storing data in lisp code - circular structures
Date:
Message-ID: <3EFFCDA4.8040703@alum.mit.edu>
Thomas A. Russ wrote:
>>When would be a good time to use one? To what types of problems do
>>they map well? Where do they give good leverage?
>
> Well, I don't normally use them, so I'm not sure about good leverage,
> but I do have at least a cute example of their usefulness.
> (defvar *days* '#1=(monday tuesday wednesday thursday friday
> saturday sunday . #1#))
Any wimp can use circular structure, but _real_ programmers know
how to use circular code. Every once in a while there is discussion
on c.l.l about the tradeoffs between recursion and iteration. Some
say that recursion can be hard to read, and has execution efficiency
issues on real hardware. Others say that iteration is hard to read,
uses implied goto's, and is generally unlispy:
<1> (defun recurse-n (n)
(when (> n 0)
(format t "~a~%" n)
(recurse-n (1- n))))
recurse-n
<2> (defun iterate-n (n)
(loop
(when (= n 0)
(return nil))
(format t "~a~%" n)
(decf n)))
iterate-n
<3> (recurse-n 5)
5
4
3
2
1
nil
<4> (iterate-n 5)
5
4
3
2
1
nil
But _real_ programmers know how to avoid both! No silly repeatuing function calls,
no implied goto's, just do it:
<5> (defun nike-n (n) . #1=((when (> n 0) (format t "~a~%" n) (decf n) . #1#)))
nike-n
<6> (nike-n 5)
5
4
3
2
1
nil
<7>
[Warning: This is a tongue-in-cheek response. The suggested code at end
is _not_ legal CL code, although it will run interpreted in some
implementations and the transcript is an honest copy of a real session.]
Steven M. Haflich wrote:
> But _real_ programmers know how to avoid both! No silly repeatuing
> function calls,
> no implied goto's, just do it:
>
> <5> (defun nike-n (n) . #1=((when (> n 0) (format t "~a~%" n) (decf n) .
> #1#)))
> nike-n
> <6> (nike-n 5)
> 5
> 4
> 3
> 2
> 1
> nil
> <7>
:-)
> [Warning: This is a tongue-in-cheek response. The suggested code at end
> is _not_ legal CL code, although it will run interpreted in some
> implementations and the transcript is an honest copy of a real session.]
Christian Queinnec has shown how to compile such programs in
"Compiling Syntactically Recursive Programs":
<http://youpou.lip6.fr/queinnec/Papers/synrec.ps.gz>
It does so by rewriting into normal Lisp, which incidently show
that you don't gain in expressitivity by allowing recursive
syntax.
--
Jens Axel S�gaard
From: Kalle Olavi Niemitalo
Subject: Re: Storing data in lisp code - circular structures
Date:
Message-ID: <87smpr3oa7.fsf@Astalo.kon.iki.fi>
Jens Axel S�gaard <······@jasoegaard.dk> writes:
> Christian Queinnec has shown how to compile such programs in
> "Compiling Syntactically Recursive Programs":
>
> <http://youpou.lip6.fr/queinnec/Papers/synrec.ps.gz>
Thanks. I've been wondering if there is an algorithm for that.
Now that you write there is, I don't have to read the paper itself. :-)
········@medialab.com (Chris Perkins) writes:
> Ok, I'll bite. I know what a circular structure is and how to go
> about creating one, but it's definitely not in my toolkit. I've never
> run across one in Lisp code I've read nor have I used one.
>
> When would be a good time to use one? To what types of problems do
> they map well? Where do they give good leverage?
>
> Curious,
Funny you should mention mapping, since that's a place that circularity
can be fun AS LONG AS at least one mapped element terminates. Consider:
(setq *print-circle* t) ;You'll be unhappy if you don't do this.
(defun circular-list (&rest elements)
(let ((backbone (copy-list elements)))
(nconc backbone backbone)))
(circular-list 'foo 'bar)
=> #1=(FOO BAR . #1#)
(let ((*print-length* 5) (*print-circle* nil))
(circular-list 'foo 'bar))
(FOO BAR FOO BAR FOO ...)
=> #1=(FOO BAR . #1#)
(mapcar #'list (circular-list 'wrapper) '(a b c))
=> ((WRAPPER A) (WRAPPER B) (WRAPPER C))
====================
But circularity is also useful in other cases that are more subtle.
(defun simple-person
name
parent ;just one, for simplicity
(kids '()))
(let ((cain (make-simple-person :name "Cain"))
(enoch (make-simple-person :name "Enoch")))
(setf (simple-person-parent enoch) cain)
(setf (simple-person-kids cain) (list enoch))
cain)
=> #1=#S(SIMPLE-PERSON :NAME "Cain"
:KIDS (#S(SIMPLE-PERSON :NAME "Enoch"
:PARENT #1#)))
This is classically described as circular in that there are forward
and backward pointers between parents and kids. In the database world,
this would be replaced by using a unique key to each person that was
opaque. In effect, one would put (list "Enoch") not (list enoch)
in Cain's kids slot, hiding the circularity. In early Lisp, this was
done by using symbols (usually called atoms). Lots of early knowledge
systems did:
(putprop 'cain "Cain" 'name)
(putprop 'enoch "Enoch" 'name)
(putprop 'enoch 'cain 'parent)
(putprop 'cain '(enoch) 'kids)
so that
(get 'cain 'kids) => (ENOCH)
and
(get 'enoch 'parent) => CAIN
with no hint of circularity in sight because the name was opaque.
Thanks,
That helps a lot. The mapping example is enlightening. I've been
toying with them this afternoon and circular lists are fun.
I guess I did know about circular structures (like the Cain/Enoch
example) and use things like that all the time, I just never think of
them as being "different" in the way that circular lists are. The
structures aren't circular in and of themselves, they are more
cross-referential.
Chris
········@medialab.com (Chris Perkins) writes:
> Thanks,
>
> That helps a lot. The mapping example is enlightening. I've been
> toying with them this afternoon and circular lists are fun.
>
> I guess I did know about circular structures (like the Cain/Enoch
> example) and use things like that all the time, I just never think of
> them as being "different" in the way that circular lists are. The
> structures aren't circular in and of themselves, they are more
> cross-referential.
Heh. Well, try to print one of them using PRINT with *PRINT-LEVEL*,
*PRINT-LENGTH*, and *PRINT-CIRCLE* set to NIL and see if you still
think the same. ;)
········@medialab.com (Chris Perkins) writes:
> Ok, I'll bite. I know what a circular structure is and how to go
> about creating one, but it's definitely not in my toolkit. I've never
> run across one in Lisp code I've read nor have I used one.
You've never used a doubly-linked list? Or a tree where nodes contain
references to their parents? Hierarchical filesystems often do this ...
-dan
--
http://www.cliki.net/ - Link farm for free CL-on-Unix resources
Drew McDermott <··················@at.yale.dot.edu> writes:
> * If you're writing circular data structures out, or you want to
> preserve sharing relationships among data structures, then you have
> to be very careful. (How _does_ Java solve this problem just by
> making up the word "serialize"?)
The format of the object output stream includes integer abbreviations
for previously seen objects. Pretty much the same as Lisp's '(#1='foo
#1#) syntax except in a binary format.
-Peter
--
Peter Seibel ·····@javamonkey.com
Lisp is the red pill. -- John Fraser, comp.lang.lisp
Robbie Sedgewick wrote:
> I'm a Lisp newby,...
<pounce>
Please consider sharing with us how you became a Lisp newy:
http://www.cliki.net/The%20Road%20to%20Lisp%20Survey
<\pounce>
And without mentioning any names (Thomas, Heow), some of you with my
favorite Road to Lisp stories have yet to contribute.
>... and I'm trying to figure out some design issues for
> some code I am going to write.
>
> I am writing some scientific code that should take a little while to
> run, so I want to be able to write out the data in some way that is
> easily read in. I could save an image, but that doesn't seem like a
> clean, especially since I will have many runs that I want to save. In
> python, I would probably use the "pickle" package to serialize my data
> structures, but that isn't availible here.
>
> What I was thinking it would be neat is to have my scientific code
> write a lisp program that can be loaded to load all my data.
>
> In other words my science code would create a file like
>
> data1.lisp:
> """
> (in-package :data)
>
> (defparameter *beta* 0.125)
>
> (defparameter *greens-function* #(1.000 0.7342 0.5242 0.223))
>
> ;; And so on with more compicated data structures...
> """
>
> then a month later in a different lisp process I could `(load
> "data1.lisp")' to get my results back.
>
> Anyone tried anything like this? Any pitfalls I should watch out for?
> Anything better?
>
I'm a weirdo in that I have never used print/read to store data (I have
to date always had persistent CLOS tools to play with) but I do think
you might at least look at that possibility.
Your approach seems workable, possibly overkill, but possibly superior
if you really do need to bind data to special variables. Without your
approach, you end up with a tricky positional dependency ("lessee, the
first thing Is *beta*, then comes *greens-function*...").
But then I am wondering if another design question here is overuse of
specials. Could you have a dynamic little database in the form of an
assoc or hash-table? '((beta . 0.125)(greens-function . #(...)))?
If so, a single print/read does it all. And then you can write debugging
or trace iterators over the little database.
Jes' thinkin' out loud.
--
kenny tilton
clinisys, inc
http://www.tilton-technology.com/
---------------------------------------------------------------
"Everything is a cell." -- Alan Kay
········@yahoo.com (Robbie Sedgewick) writes:
> I'm a Lisp newby, and I'm trying to figure out some design issues for
> some code I am going to write.
>
> I am writing some scientific code that should take a little while to
> run, so I want to be able to write out the data in some way that is
> easily read in. I could save an image, but that doesn't seem like a
> clean, especially since I will have many runs that I want to save. In
> python, I would probably use the "pickle" package to serialize my data
> structures, but that isn't availible here.
You can put the stuff in a list and use some functions I wrote some
time ago
http://groups.google.com/groups?selm=fz3cinaag0.fsf%40cupid.igpm.rwth-aachen.de
to compile this to a binary "fast load" (fasl) file.
> What I was thinking it would be neat is to have my scientific code
> write a lisp program that can be loaded to load all my data.
That's more or less what happens there.
> Anyone tried anything like this? Any pitfalls I should watch out for?
> Anything better?
The problem with writing out double-floats as text is that you
sometimes loose some (minimal) precission (aside of the bloat). The
compile-data-to-a-file approach should avoid this. The drawback is
that the generated files might not be compatible from one release of
your compiler to the next. I used it because I tend to generate fairly
massive amounts of data, and printing it to a file in text format
resulted in huge files that took hours to read back in, and "fasl"
really is fast (at least on cmucl).
The other approach (writing text files) is fairly easy, as others have
mentioned, so if the precission or size issues are not a concern, you
might as well do this.
Regards,
Mario.
Mario S. Mommer <········@yahoo.com> wrote in message news:<··············@cupid.igpm.rwth-aachen.de>...
>
> You can put the stuff in a list and use some functions I wrote some
> time ago
>
> http://groups.google.com/groups?selm=fz3cinaag0.fsf%40cupid.igpm.rwth-aachen.de
>
> to compile this to a binary "fast load" (fasl) file.
>
This answers the next question I was going to ask!
I think it is a little overkill for the application I am writing now,
since I don't care too much about a little roundoff, and total
filesizes should be small, but this could be useful for some future
projects.
Thanks to everyone for their suggestions. I think I agree with what I
see as the consensus that writing out lisp code would work, but might
be a little overkill. I think that I will follow the suggestion of
just writing out the data structures or, more likely, an assoclist of
datastructures. This way I can have something nice and human readable
_and_ computer readable that is more extendable then printing stuff
out and keeping track of the order that I print it out.
Thanks,
Robbie
"Robbie Sedgewick" <········@yahoo.com> wrote in message
································@posting.google.com...
> I'm a Lisp newby, and I'm trying to figure out some design issues for
> some code I am going to write.
>
> I am writing some scientific code that should take a little while to
> run, so I want to be able to write out the data in some way that is
> easily read in. I could save an image, but that doesn't seem like a
> clean, especially since I will have many runs that I want to save. In
> python, I would probably use the "pickle" package to serialize my data
> structures, but that isn't availible here.
>
> What I was thinking it would be neat is to have my scientific code
> write a lisp program that can be loaded to load all my data.
That will work fine, particularly for simple cases. I mean, how else did
they get in there in the first place save from a Lisp program.
However, if you have any real volume of data AND you don't need to (easily)
access from other programs, perhaps you should look at using the Fast Load
capability of your Lisp, and save that instead. It will load VASTLY faster.
A simple way to do this is to simply wrap all of your data into a DEFUN, and
then compile that, and then load it. That can give you performance on load.
The other advantage of saving data directly as binary is that it keeps all
of the attributes of the data that simply PRINTing the data can lose.
Some have mentioned potential losses of precision with floating point
numbers. Another factor to consider is that as you tweak and tune the system
for higher performance, the data structures get more and more specialized
(particularly arrays), and even though an array will print and read, you
don't necessarily get the same thing.
Simple example:
CL-USER 14 > (setf a (make-array 5 :element-type 'single-float
:initial-contents '(1.0 2.0 3.0 4.0 5.0)))
#(1.0 2.0 3.0 4.0 5.0)
CL-USER 15 > (setf b #(1.0 2.0 3.0 4.0 5.0))
#(1.0 2.0 3.0 4.0 5.0)
CL-USER 16 > (array-element-type a)
SINGLE-FLOAT
CL-USER 17 > (array-element-type b)
T
LispWorks has dump-form and its relatives to help facilitate this. I'm
pretty sure that ACL does as well.
The best part, though, is that simply PRINTing and READing the data is so
trivial to do for many cases, that it can easily get you started and then
you can look to the more detailed techniques as your needs warrant.
Regards,
Will Hartung
(·····@mosft.com)
Will Hartung wrote:
> A simple way to do this is to simply wrap all of your data into a DEFUN, and
> then compile that, and then load it. That can give you performance on load.
>
> The other advantage of saving data directly as binary is that it keeps all
> of the attributes of the data that simply PRINTing the data can lose.
What happens when you upgrade to the next release from my Lisp vendor, and it
refuses to load old incompatible binary files, or perhaps if you want to port
to a different implementation?
Never store your data in a form you cannot write a program to read. If you
don't understand that, then I've got some Lisp implementations on 8-track
paper tape to sell you...