Hi
I thought about serialization and opened PCL chapter about binary file
parsing but found nothing about floats and signed integers. I've
written two functions for saving floats which use functions:
integer-decode-float, ldb, write-byte, read-byte, ash and so on. I
saved and restored 10000 floats. C version appears to be 15 times
faster (Lisp is LispWorks 4.4), 'cause it just saves float value byte
by byte, without decoding ;-(
Here is my float saving code:
(defun write-binary-float (f stream)
(declare (optimize (speed 3)(float 0)(debug 0)(safety 0)(float 0))
(float f)(stream stream))
(multiple-value-bind (signif exp sign) (integer-decode-float f)
(declare (integer signif) (fixnum exp sign))
(write-byte (the fixnum (ldb (byte 8 0) signif)) stream)
(write-byte (the fixnum (ldb (byte 8 8) signif)) stream)
(write-byte (the fixnum (ldb (byte 8 16) signif)) stream)
(write-byte (the fixnum (ldb (byte 8 24) signif)) stream)
(write-byte (the fixnum (ldb (byte 8 32) signif)) stream)
(write-byte (the fixnum (ldb (byte 8 40) signif)) stream)
(write-byte (the fixnum (ldb (byte 8 48) signif)) stream)
(write-byte (the fixnum (ldb (byte 8 0) exp)) stream)
(write-byte (the fixnum (ldb (byte 8 8) exp)) stream)
(write-byte sign stream)))
Thanks in advance!
Lisper
From: Sam Steingold
Subject: Re: What is the fastest way to write floats and signed integers into binary file ?
Date:
Message-ID: <ur7gdcy9m.fsf@gnu.org>
> * <···········@znvy.eh> [2005-05-11 12:55:37 -0700]:
>
> (defun write-binary-float (f stream)
<http://clisp.cons.org/impnotes/stream-dict.html#bin-input>
<http://clisp.cons.org/impnotes/stream-dict.html#bin-output>
--
Sam Steingold (http://www.podval.org/~sds) running w2k
<http://www.camera.org> <http://www.mideasttruth.com/> <http://www.memri.org/>
<http://pmw.org.il/> <http://www.dhimmi.com/> <http://www.iris.org.il>
Life is like a diaper -- short and loaded.
Thank you for answer. But this CLISP function AFAIK is built-in and not
portable. Question was What is the most proper way to save/restore
floats and signed integers (may be another types ?) to binary stream
(file) in Common Lisp.
Another question What are the pros and cons of C-like representation,
why Lisp's are not same ? As far as I know such binary things are not
covered by the language standard. May be LispWorks or CMUCL have
special functions for fast transformation of primitive types to binary
form ???
From: Pascal Bourguignon
Subject: Re: What is the fastest way to write floats and signed integers into binary file ?
Date:
Message-ID: <87r7gcch1b.fsf@thalassa.informatimago.com>
···········@mail.ru writes:
···········@mail.ru writes:
> Thank you for answer. But this CLISP function AFAIK is built-in and not
> portable. Question was What is the most proper way to save/restore
> floats and signed integers (may be another types ?) to binary stream
> (file) in Common Lisp.
>
> Another question What are the pros and cons of C-like representation,
> why Lisp's are not same ? As far as I know such binary things are not
> covered by the language standard. May be LispWorks or CMUCL have
> special functions for fast transformation of primitive types to binary
> form ???
>
> Thank you for answer. But this CLISP function AFAIK is built-in and not
> portable. Question was What is the most proper way to save/restore
> floats and signed integers (may be another types ?) to binary stream
> (file) in Common Lisp.
Use integer-decode-float to get integers, then slice integers to write
them byte by bytes on a :external-format '(unsigned-byte 8) stream.
You might be tempted to use an '(unsigned-byte 32) stream to write the
integers 32-bit by 32-bit, but then you don't know what bytesex is
used by the implementation and the file won't be portable. Even with
(unsigned-byte 8) you might have problems if you're running on a
36-bit architecture...
> Another question What are the pros and cons of C-like representation,
> why Lisp's are not same ?
Binary data is not portable, not even on the same computer (it depends
on the compiler used). Not even with the same program, half of binary
data cannot be written to a file and read back by the same program in
another process (the pointers!).
Therefore it's much better to use text based file formats most often.
For numeric data, text file formats are often more compact, leading to
better I/O throughput.
And since the proper way to handle binary data is to do it byte by
byte anyway, better just use characters.
(let ((*print-readably* t)) (format out "~S" data))
and:
(setf data (let ((*read-eval* nil)) (read in)))
are perfectly portable, and good.
> As far as I know such binary things are not
> covered by the language standard. May be LispWorks or CMUCL have
> special functions for fast transformation of primitive types to binary
> form ???
--
__Pascal Bourguignon__ http://www.informatimago.com/
The mighty hunter
Returns with gifts of plump birds,
Your foot just squashed one.
> Binary data is not portable, not even on the same computer (it
depends
> on the compiler used). Not even with the same program, half of
binary
> data cannot be written to a file and read back by the same program in
> another process (the pointers!).
This feels like a cop out to me. Textual representations are not
inherently more portable than binary representations. You still have
to know how to read in what you wrote out. Sure, if you are using
nothing but Lisp and in Lisp you are using nothing but simple lists and
atoms than this is free with princ and read, but that's not true in
general. Plus, this is what Network Byte Order was created for...
Also, portability can not be the only consideration when designing a
portable language lest you fall victim to
least-common-denominator-itis. The entire notion of files and
pathnames aren't portable, but any language that doesn't have them
built-in is near useless. Frankly, I think the same is true of
networking and threading for many problem domains, though portable
libraries have certainly improved the situation for Lisp..
> For numeric data, text file formats are often more compact, leading
to better I/O throughput.
That's an odd statement to my ear. For what problem domains have you
found this to be true? The only one that I have much familiarity with
is computer graphics, and I assure you that writing all the vertices
and texture-coordinates of my models out in a textual format would
certainly not be more compact or lead to better I/O throughput.
Anyway, just another opinion to throw into the mix. Happy coding,
Justin Dubs
From: Pascal Bourguignon
Subject: Re: What is the fastest way to write floats and signed integers into binary file ?
Date:
Message-ID: <87hdh69lnz.fsf@thalassa.informatimago.com>
······@gmail.com writes:
>> For numeric data, text file formats are often more compact, leading
> to better I/O throughput.
>
> That's an odd statement to my ear. For what problem domains have you
> found this to be true? The only one that I have much familiarity with
> is computer graphics, and I assure you that writing all the vertices
> and texture-coordinates of my models out in a textual format would
> certainly not be more compact or lead to better I/O throughput.
It's because all 32-bit binary numbers take 4 bytes, but a lot of
occuring numbers can be written with only one or two characters.
An example I witnessed: for geographic data (the coordinates of roads
in a small country), the ASCII representation was about 60% the size
of the binary one.
Of course, one may argue that it would be better to gzip the binary
than use the ASCII.
Still ASCII is better for portability, debugging, interactivity,
etc. See why most Internet protocols are ASCII based.
--
__Pascal Bourguignon__ http://www.informatimago.com/
In a World without Walls and Fences,
who needs Windows and Gates?
From: Wade Humeniuk
Subject: Re: What is the fastest way to write floats and signed integers into binary file ?
Date:
Message-ID: <Zi5he.54294$vN2.10675@clgrps13>
Here is a link to a binary float dumper and loader
especially written for LW. It utilizes the FLI
to effectively casts an array of double
floats to an array of (unsigned-byte 8).
http://www3.telus.net/public/whumeniu/lw-dump-float.lisp
I do not know if it is fast but for a 1000000
double floats dumped and then loaded, it takes about
4.6 secs on my Athlon 1.4 GHz.
Tested under LWW 4.3.7.
Wade
CL-USER 1 > (setf array #(1.2s0 1.3d10 234.1716e34 0.0001))
#(1.2 1.3E10 2.3417160000000012E36 1.0E-4)
CL-USER 2 > (lwfu:dump-floats "test.floats" array)
4
CL-USER 3 > (lwfu:load-floats "test.floats")
#(1.2 1.3E10 2.3417160000000012E36 1.0E-4)
CL-USER 4 > (lwfu::test "test.floats" 100000)
Timing the evaluation of (PROGN (LW-FLOAT-UTILITIES:DUMP-FLOATS
LW-FLOAT-UTILITIES::FILENAME ARRAY) (LW-FLOAT-UTILITIES:LOAD-FLOATS
LW-FLOAT-UTILITIES::FILENAME ARRAY) LW-FLOAT-UTILITIES::N)
user time = 0.480
system time = 0.000
Elapsed time = 0:00:01
Allocation = 19616 bytes standard / 1969 bytes conses
0 Page faults
100000
CL-USER 5 > (delete-file "test.floats")
T
CL-USER 6 > (lwfu::test "test.floats" 1000000)
Timing the evaluation of (PROGN (LW-FLOAT-UTILITIES:DUMP-FLOATS
LW-FLOAT-UTILITIES::FILENAME ARRAY) (LW-FLOAT-UTILITIES:LOAD-FLOATS
LW-FLOAT-UTILITIES::FILENAME ARRAY) LW-FLOAT-UTILITIES::N)
user time = 4.536
system time = 0.060
Elapsed time = 0:00:04
Allocation = 40784 bytes standard / 5995 bytes conses
0 Page faults
1000000
CL-USER 7 > (delete-file "test.floats")
T
CL-USER 8 >
From: Duane Rettig
Subject: Re: What is the fastest way to write floats and signed integers into binary file ?
Date:
Message-ID: <4vf5mil4g.fsf@franz.com>
Wade Humeniuk <··················@telus.net> writes:
> Here is a link to a binary float dumper and loader
> especially written for LW. It utilizes the FLI
> to effectively casts an array of double
> floats to an array of (unsigned-byte 8).
>
> http://www3.telus.net/public/whumeniu/lw-dump-float.lisp
>
> I do not know if it is fast but for a 1000000
> double floats dumped and then loaded, it takes about
> 4.6 secs on my Athlon 1.4 GHz.
If you have simple-streams, you can do this directly, as
well as quickly, with no copying other than the i/o transfer
itself. On an Athlon 1.4 Ghz with Allegro CL 7.0:
CL-USER(1): (defun test (filename n)
(let ((array (make-array n :element-type 'double-float)))
(with-open-file (f filename :direction :output :if-exists :supersede)
(write-vector array f))
(with-open-file (f filename)
(read-vector array f))))
TEST
CL-USER(2): (compile *)
TEST
NIL
NIL
CL-USER(3): (time (test "test.floats" 100000))
; cpu time (non-gc) 0 msec user, 20 msec system
; cpu time (gc) 20 msec user, 0 msec system
; cpu time (total) 20 msec user, 20 msec system
; real time 142 msec
; space allocation:
; 142 cons cells, 811,160 other bytes, 0 static bytes
800000
CL-USER(4):
--
Duane Rettig ·····@franz.com Franz Inc. http://www.franz.com/
555 12th St., Suite 1450 http://www.555citycenter.com/
Oakland, Ca. 94607 Phone: (510) 452-2000; Fax: (510) 452-0182
From: Wade Humeniuk
Subject: Re: What is the fastest way to write floats and signed integers into binary file ?
Date:
Message-ID: <328he.63227$tg1.41966@edtnps84>
In case anyone has looked at the file, I have changed it
quite a bit. You can download the new one at the same
url.
Wade
From: Wade Humeniuk
Subject: Re: What is the fastest way to write floats and signed integers into binary file ?
Date:
Message-ID: <XVche.38138$0X6.15684@edtnps90>
Well, poking even further into the LW built-ins,
you can directly write compiled forms into a
FSL file. So you can take a double-float array
and directly dump it to a file. Then you
can load the form back. Doing this with a
(simple-array double-float (1000000)) takes
.36 secs. So I could chuck all my previous
work. Kudos to LispWorks, serves me right,
I have shyed away from fasl dumping for
too long.
(defvar *save-table* (make-hash-table))
(defun save-objects (filename &rest key-values)
"Dumped key-value pairs are restored to the *save-table*
when RESTORE-OBJECT is called."
(sys:dump-forms-to-file
(compile-file-pathname filename)
`(,@(loop for (key value) on key-values by #'cddr
collect `(setf (gethash ',key *save-table*) ,value)))))
(defun restore-objects (filename)
(sys:load-data-file (compile-file-pathname filename)))
LWFU 13 > (let ((array (make-array 1000000 :element-type 'double-float)))
(time
(progn (setf (gethash :float-array *save-table* ) nil)
(save-objects "test-floats" :float-array array)
(restore-objects "test-floats")))
(equalp array (gethash :float-array *save-table*)))
Timing the evaluation of (PROGN (SETF (GETHASH :FLOAT-ARRAY *SAVE-TABLE*) NIL)
(SAVE-OBJECTS "test-floats" :FLOAT-ARRAY ARRAY) (RESTORE-OBJECTS "test-floats"))
; Loading fasl file d:\cygwin\home\wade\test-floats.fsl
user time = 0.360
system time = 0.110
Elapsed time = 0:00:01
Allocation = 8083688 bytes standard / 11506 bytes conses
0 Page faults
Calls to %EVAL 58
T
This save-objects function also allows saving hash-tables.
LWFU 14 > (defvar *test-hash* (make-hash-table))
*TEST-HASH*
LWFU 15 > (setf (gethash 'key1 *test-hash*) 10)
10
LWFU 16 > (setf (gethash 'key2 *test-hash*) 'hello)
HELLO
LWFU 17 > (save-objects "test-hash" :test-hash *test-hash*)
1
LWFU 18 > (restore-objects "test-hash")
; Loading fasl file d:\cygwin\home\wade\test-hash.fsl
#P"d:/cygwin/home/wade/test-hash.fsl"
LWFU 19 > (setf test-hash (gethash :test-hash *save-table*))
#<EQL Hash Table{2} 20669CDC>
.. few typos
LWFU 22 : 1 > (gethash 'key1 test-hash)
10
T
LWFU 23 : 1 > (gethash 'key2 test-hash)
HELLO
T
Wade