With the help of cl-store's serialization support, I'm trying to store
a list of integers into a column (type bytea, which is an blob of data
of type unsigned byte 8) to a postgresql database.
Is there a portable way to hack up a handy macro like with-output-to-
string for (unsigned-byte 8) data stream?
(with-output-to-unsigned-byte-8-array (out)
;; write ascii "abc"
(write-byte 97 out)
(write-byte 98 out)
(write-byte 99 out))
=> return a '(VECTOR (UNSIGNED-BYTE 8)) with content #(97 98 99)
A related question - how do I specify an object literal of type
'(VECTOR (UNSIGNED-BYTE 8)) using the standard reader syntax?
In many languages, they usually provide an escape sequence in string
that you can use like "\x97\x98\x99".
CL-USER> (TYPE-OF #(97 98 99))
(SIMPLE-VECTOR 3)
CL-USER> (TYPEP #(97 98 99) '(VECTOR (UNSIGNED-BYTE 8)))
NIL
Thanks
fungsin
From: Edi Weitz
Subject: Re: with-output-to-unsigned-byte-8-array ?
Date:
Message-ID: <ud4yod9xy.fsf@agharta.de>
On Thu, 19 Jul 2007 06:13:49 -0000, Liu Fung Sin <···········@gmail.com> wrote:
> Is there a portable way to hack up a handy macro like
> with-output-to- string for (unsigned-byte 8) data stream?
http://weitz.de/flexi-streams/#with-output-to-sequence
> A related question - how do I specify an object literal of type
> '(VECTOR (UNSIGNED-BYTE 8)) using the standard reader syntax?>
>
> In many languages, they usually provide an escape sequence in string
> that you can use like "\x97\x98\x99".
Not exactly what you're asking for, but maybe it helps:
http://weitz.de/cl-interpol/
Cheers,
Edi.
--
Lisp is not dead, it just smells funny.
Real email: (replace (subseq ·········@agharta.de" 5) "edi")
Liu Fung Sin <···········@gmail.com> wrote:
+---------------
| With the help of cl-store's serialization support, I'm trying to store
| a list of integers into a column (type bytea, which is an blob of data
| of type unsigned byte 8) to a postgresql database.
|
| Is there a portable way to hack up a handy macro like with-output-to-
| string for (unsigned-byte 8) data stream?
|
| (with-output-to-unsigned-byte-8-array (out)
| ;; write ascii "abc"
| (write-byte 97 out)
| (write-byte 98 out)
| (write-byte 99 out))
|
| => return a '(VECTOR (UNSIGNED-BYTE 8)) with content #(97 98 99)
+---------------
Unfortunately, the CLHS provides no standardized way [AFAIK] to extend
streams beyond the normal variants, but almost every implementation
provides [or has available for it] some version of "Gray streams" (q.v.),
which provides a stream type whose behavior is provided by user code
[somewhat similar to PTYs or TUNs, if you're familiar with those, which
provide a specific driver type on one side for which emulation of the
"hardware" is provided by user-mode software on the other side], see:
http://www.nhplace.com/kent/CL/Issues/stream-definition-by-user.html
http://franz.com/support/documentation/6.0/doc/gray-streams.htm
http://www.cliki.net/trivial-gray-streams
http://www.cliki.net/simple-stream
http://www.franz.com/support/documentation/6.2/doc/streams.htm
http://common-lisp.net/project/cmucl/doc/cmu-user/extensions.html#toc45
So assuming the implementation you use has some version of Gray streams
or simple-streams available, you could define a subsclass of one of
those that, when opened, creates an adjustable (UNSIGNED-BYTE 8)
vector with an initial fill pointer of zero, and on each "write" type
call does a VECTOR-PUSH-EXTEND on the vector with the value written,
plus provide a hook to get at the vector to return it, then define your
WITH-OUTPUT-TO-UNSIGNED-BYTE-8-ARRAY macro to wrap all of that around
the &BODY of a call to the macro.
[If I were adept at Gray streams or simple-streams myself, I'd cheerfully
provide you with a working example at this point. But sadly I am not,
and someone else will have to that bit...]
-Rob
-----
Rob Warnock <····@rpw3.org>
627 26th Avenue <URL:http://rpw3.org/>
San Mateo, CA 94403 (650)572-2607
[Replying to Rob Warnock's post to keep his comments in mind]
* ····@rpw3.org <································@speakeasy.net>
( Wcqdnd-p5e5MEz3bnZ2dnUVZ_gidnZ2d ! speakeasy.net ) :
| Liu Fung Sin <···········@gmail.com> wrote:
| +---------------
| | With the help of cl-store's serialization support, I'm trying to store
| | a list of integers into a column (type bytea, which is an blob of data
| | of type unsigned byte 8) to a postgresql database.
| |
| | Is there a portable way to hack up a handy macro like with-output-to-
| | string for (unsigned-byte 8) data stream?
| |
| | (with-output-to-unsigned-byte-8-array (out)
| | ;; write ascii "abc"
| | (write-byte 97 out)
| | (write-byte 98 out)
| | (write-byte 99 out))
| |
| | => return a '(VECTOR (UNSIGNED-BYTE 8)) with content #(97 98 99)
| +---------------
|
| Unfortunately, the CLHS provides no standardized way [AFAIK] to extend
| streams beyond the normal variants, but almost every implementation
| provides [or has available for it] some version of "Gray streams" (q.v.),
| which provides a stream type whose behavior is provided by user code
Without gray or simple streams, any solution will be inferior and
severely limited. I will however show two hacks (not recommended), the
first of which is code for `WITH-BINARY-OUTPUT-TO-VECTOR' from Frode
Vatvedt Fjeld's excellent binary-types library. This example shadows
WRITE-BYTE, and will "work" ONLY when calls are made to these shadowed
functions (and not to any other such as write-sequence)
The second hack, `WITH-BINARY-OUTPUT-TO-STRING', is even worse (also not
recommended, but is probably most portable) as it writes binary output
to a temporary file #p"/tmp/xyz" and reads this back. (I wrote this a
long time ago before learning lisp, so style caveat emptor)
(defpackage "TEST"
(:use "CL")
(:shadow "WRITE-BYTE"))
(in-package "TEST")
(defvar *binary-write-byte* #'cl:write-byte)
(defun write-byte (integer stream)
(funcall *binary-write-byte* integer stream))
;; from binary-types.lisp, by frodef
(defmacro with-binary-output-to-vector
((stream-var &optional (vector-or-size-form 0)
&key (adjustable (and (integerp vector-or-size-form)
(zerop vector-or-size-form)))
(fill-pointer 0)
(element-type ''(unsigned-byte 8))
(on-full-array :error))
&body body)
"Arrange for STREAM-VAR to collect octets in a vector.
VECTOR-OR-SIZE-FORM is either a form that evaluates to a vector, or an
integer in which case a new vector of that size is created. The vector's
fill-pointer is used as the write-index. If ADJUSTABLE nil (or not provided),
an error will occur if the array is too small. Otherwise, the array will
be adjusted in size, using VECTOR-PUSH-EXTEND. If ADJUSTABLE is an integer,
that value will be passed as the EXTENSION argument to VECTOR-PUSH-EXTEND.
If VECTOR-OR-SIZE-FORM is an integer, the created vector is returned,
otherwise the value of BODY."
(let ((vector-form
(if (integerp vector-or-size-form)
`(make-array ,vector-or-size-form
:element-type ,element-type
:adjustable ,(and adjustable t)
:fill-pointer ,fill-pointer)
vector-or-size-form)))
(let ((save-bwb-var (make-symbol "save-bwb")))
`(let* ((,save-bwb-var *binary-write-byte*)
(,stream-var ,vector-form)
(*binary-write-byte*
#'(lambda (byte stream)
(if (eq stream ,stream-var)
,(cond
(adjustable
`(vector-push-extend byte stream
,@(when (integerp adjustable)
(list adjustable))))
((eq on-full-array :error)
`(assert (vector-push byte stream) (stream)
"Binary output vector is full when writing byte value ~S: ~S"
byte stream))
((eq on-full-array :ignore)
`(vector-push byte stream))
(t (error "Unknown ON-FULL-ARRAY argument ~S, must be one of :ERROR, :IGNORE."
on-full-array)))
(funcall ,save-bwb-var byte stream)))))
,@body
,@(when (integerp vector-or-size-form)
(list stream-var))))))
#+nil
(with-binary-output-to-vector (out)
;; write ascii "abc"
(write-byte 97 out)
(write-byte 98 out)
(write-byte 99 out))
;;; HACK II
(in-package "CL-USER")
(defmacro with-binary-output-to-string
((var &optional string &key (element-type '(unsigned-byte 8))) &body body)
"TESTING. Portably use a temporary file /tmp/xyz. If STRING is
specified, it must be a vector with a fill pointer."
(if string
`(multiple-value-prog1
(with-open-file (,var "/tmp/xyz" :direction :output
:element-type '(unsigned-byte 8)
:if-exists :supersede)
,@body)
(with-open-file (stream "/tmp/xyz" :element-type ,element-type)
(setf (fill-pointer ,string) (file-length stream))
(let ((nchars (read-sequence ,string stream)))
(assert (= nchars (fill-pointer ,string))))))
`(progn (with-open-file (,var "/tmp/xyz" :direction :output
:element-type '(unsigned-byte 8)
:if-exists :supersede)
,@body)
(with-open-file (stream "/tmp/xyz" :element-type ',element-type)
(let* ((file-length (file-length stream))
(string (make-array file-length
:element-type ',element-type))
(nchars (read-sequence string stream)))
(assert (= nchars file-length))
string)))))
#+nil
(with-binary-output-to-string (out)
;; write ascii "abc"
(write-byte 97 out)
(write-byte 98 out)
(write-byte 99 out))