From: ···········@yahoo.de
Subject: reading binary files
Date: 
Message-ID: <cf4b598e-57a8-4f0a-9611-e2f875231ac0@z28g2000prd.googlegroups.com>
Hi: How would you read in binary floating point numbers from binary
files? I cannot really find useful lisp code or examples which would
teach me somehting. In Bigloo I do it as follows: Bigloo has some
undocumented functions (ieee-string->double string), where 'string'
here denotes a 4byte.

The Common Lisp version should be fast and it should have the option
to chose between little and big endian. I am not saying I am going to
use Lisp now, but my Fortran code spends most if its time reading in
files (from global climate and chemistriy and satellite simulations
and observations). A day worths of data has typically 10 to 15 arrays
of dimension 7x7x50000 times 365 days + the binary from the climate
models. The code walks more than once through the years until the
solution converges (typically after 5 iterations and a couple of
hours).

For example Bigloo would take nearly 17 seconds to decode a binary
file of dimension 7x7x50000. Nonsensly slow. However, a dramatic boost
can be obtained by using (mmap-get) instead of simple ports. mmap
reads in the binary file as a string and the time to decode 64bit
floats into a 7x7x60000 array takes 0.8 seconds (instead of 17seconds
with ports).

I am not requesting other people to work things out here for me.
However, I would like to learn.

Thanks for any pointers (I am off to the states to SF for a
conference, so likeley cannot quickly answer any further questions),
KingKong
==
Bigloo code:
The binary files consist of 3 integer dimensions (dim1,dim2,dim3), and
the 64bit float numbers
==
;; converts 8 bytes string to ieee-64bit float either little or big
endian
(define-inline
   (ieee-str->f8::double string::bstring endian)
   (if (eq? endian 'swap-bytes)
       (ieee-string->double
	(list->string (reverse (string->list string))))
       (ieee-string->double string)))




;; reading in 3D array: first 3 numbers are dimensions
;; string->uint is nicked from srfi-56: it converts a string to an
integer number
(define
 (rc-3d-bin::class-f64array  file-name::string endian::obj)
 (let* ((buf::mmap  (open-mmap f-avk read: #t))
	(dim1::int (string->uint (mmap-get-string buf 4) endian))
	(dim2::int (string->uint (mmap-get-string buf 4) endian))
	(dim3::int (string->uint (mmap-get-string buf 4) endian)))
    (print "dims " dim1 " " dim2 " " dim3)
    ;(exit)
    (let ((erg::class-f64array (make-f64array dim3 dim1 dim2 fun:
(lambda (x) (make-vector x 0.0)))))
       (do ((i 0 (+fx i 1)))
	   ((=fx i dim3)
	    erg)
	   (do ((j 0 (+fx j 1)))
	       ((=fx j dim1))
	       (do ((z 0 (+fx z 1)))
		   ((=fx z dim2))
		   (let ((val::double (ieee-str->f8 (mmap-get-string buf 8)
endian)))
		      (aset! erg val i j z))))))))

==

From: D Herring
Subject: Re: reading binary files
Date: 
Message-ID: <49440afa$0$17066$6e1ede2f@read.cnntp.org>
···········@yahoo.de wrote:
> Hi: How would you read in binary floating point numbers from binary
> files? I cannot really find useful lisp code or examples which would
> teach me somehting. In Bigloo I do it as follows: Bigloo has some
> undocumented functions (ieee-string->double string), where 'string'
> here denotes a 4byte.

Try the following
http://common-lisp.net/project/ieee-floats/

- Daniel
From: Rob Warnock
Subject: Re: reading binary files
Date: 
Message-ID: <y8mdnT3WFbltVNnUnZ2dnUVZ_vudnZ2d@speakeasy.net>
D Herring  <········@at.tentpost.dot.com> wrote:
+---------------
| ···········@yahoo.de wrote:
| > Hi: How would you read in binary floating point numbers from binary
| > files? I cannot really find useful lisp code or examples which would
| > teach me somehting. In Bigloo I do it as follows: Bigloo has some
| > undocumented functions (ieee-string->double string), where 'string'
| > here denotes a 4byte.
| 
| Try the following
| http://common-lisp.net/project/ieee-floats/
+---------------

Also see:

    http://gigamonkeys.com/book/practical-parsing-binary-files.html


-Rob

-----
Rob Warnock			<····@rpw3.org>
627 26th Avenue			<URL:http://rpw3.org/>
San Mateo, CA 94403		(650)572-2607