I have a project, where the customer wants some information extracted
from a Macromedia Flash SWF file and later perhaps changing the
information and generating new SWF files. This is a nice little project
for my first commercial Lisp project. The only library I've found was
this:
http://www.xach.com/lisp/cl-flash/
But it writes SWF, only and doesn't read it, so I've used the bitstream
class from this project and started to implement a new version (it uses
a Lispworks extension for subclassing your own streams, but you can
change this to Common Lisp, if you like, and post the changes to this
newsgroup or send me an email):
http://www.frank-buss.de/tmp/swf.lisp.txt
The test function can read for example this file:
http://www.frank-buss.de/links.swf
and extracts all visible texts for every frame as XML:
<file name="c:/tmp/test.swf">
<frame label="" number="1">
<text>Links</text>
<text>G�stebuch</text>
<text>Kontakt</text>
</frame>
<frame label="" number="2">
<text>Links</text>
<text>G�stebuch</text>
<text>Kontakt</text>
</frame>
....
The main class is the movie class, which has all movie meta information
and all SWF tags. The tag-class has the methods load-from-stream and
update-content. All known tags are subclasses of the tag-class. When a
movie is loaded, the method load-from-stream is called for every tag
(which loads the binary data to memory in the tag base class for unknown
tags) and when saving, the binary data is updated first from the
internal slots of the tag-class, with the update-content method and then
saved to a stream. The save method is not tested, but I think this
should work in theory.
The problem is my redundant code. Flash is a very easy format (see
http://download.macromedia.com/pub/flash/flash_file_format_specification.pdf ),
but it uses bit-packed structures, which you have to read like this:
(defmethod read-matrix ((self bitstream))
(synch-bits self)
(let ((scale-x 1)
(scale-y 1)
(skew0 0)
(skew1 0)
(translate-x 0)
(translate-y 0))
(when (read-boolean self)
(let ((bits (read-bits 5 self)))
(setf scale-x (read-bits bits self))
(setf scale-y (read-bits bits self))))
(when (read-boolean self)
(let ((bits (read-bits 5 self)))
(setf skew0 (read-bits bits self))
(setf skew1 (read-bits bits self))))
(let ((bits (read-bits 5 self)))
(setf translate-x (read-bits bits self))
(setf translate-y (read-bits bits self)))))
This is not very Lisp-like, because I don't want to say the computer
every single step, but it should look like more as the description in
the Flash File Format Specification. I think it can be done with some
Lisp macrology, but I don't know how to do it. Would be fine, if the
macros produces both, the read and write functions.
In the end I plan to publish the SWF library under BSD licence, so feel
free to comment my current code to make it a better library for other
people, too.
--
Frank Bu�, ··@frank-buss.de
http://www.frank-buss.de, http://www.it4-systems.de
Frank Buss <··@frank-buss.de> writes:
[snip]
> This is not very Lisp-like, because I don't want to say the computer
> every single step, but it should look like more as the description in
> the Flash File Format Specification. I think it can be done with some
> Lisp macrology, but I don't know how to do it. Would be fine, if the
> macros produces both, the read and write functions.
Peter's book has an excellent chapter on just this topic:
http://www.gigamonkeys.com/book/practical-parsing-binary-files.html
When I resume work on CL-SWF I intend to take advantage of his
techniques.
Here is the SWF dumper I wrote to do some debugging:
http://www.xach.com/lisp/cl-flash/dumper.lisp
Zach
Zach Beane <····@xach.com> wrote:
> Peter's book has an excellent chapter on just this topic:
>
> http://www.gigamonkeys.com/book/practical-parsing-binary-files.html
yes, looks interesting.
> When I resume work on CL-SWF I intend to take advantage of his
> techniques.
>
> Here is the SWF dumper I wrote to do some debugging:
>
> http://www.xach.com/lisp/cl-flash/dumper.lisp
but your dumper.lisp doesn't use it :-) I think it should look like this:
(deftag 123 name
matrix name
boolean has-foo
something foo :if has-foo)
(deftype matrix
boolean has-scale
bits-5 num-bits :if has-scale
bits-num-bits scale-x :if has-scale
bits-num-bits scale-y :if has-scale
...
bits-5 num-bits-trans
bits-num-bits-trans translate-x
bits-num-bits-trans translate-y)
--
Frank Bu�, ··@frank-buss.de
http://www.frank-buss.de, http://www.it4-systems.de
Frank Buss wrote:
> I think it should look like this:
>
> (deftag 123 name
> matrix name
> boolean has-foo
> something foo :if has-foo)
There's a reason why Lisp has so many parentheses. Your DEFTAG
would be difficult to parse. If you change it to something
like
(deftag 123 name
(matrix name)
(boolean has-foo)
(something foo :if has-foo))
parsing would be trivial. This would also make it easier to
extend your DEFTAG at a later stage while remaining backwards
compatible.
Arthur Lemmens
Arthur Lemmens wrote:
> There's a reason why Lisp has so many parentheses. Your DEFTAG
> would be difficult to parse. If you change it to something
> like
>
> (deftag 123 name
> (matrix name)
> (boolean has-foo)
> (something foo :if has-foo))
or consider extending XMLisp (http://agentsheets.com/lisp/XMLisp/)
1) define tags as CLOS classes, e.g.,
(defclass ID3 (xml-serializer)
((name :accessor name :type matrix)
(has-foo :accessor has-foo :type booean)
(foo :accessor :foo :type something)))
2) write CODECs for each type (e.g., matrix, boolean, ..)
3) create binary reader
This should alllow you to read an SWF file and get XML out. You could
modify that and write it back as XML (for ASCII streams) or binary as
SWF file again.
Frank Buss <··@frank-buss.de> writes:
> This is not very Lisp-like, because I don't want to say the computer
> every single step, but it should look like more as the description in
> the Flash File Format Specification. I think it can be done with some
> Lisp macrology, but I don't know how to do it. Would be fine, if the
> macros produces both, the read and write functions.
Could it be that you're stuck thinking about the problem in a
bottom-up manner? When I have to parse some ugly binary format, I
start writing out the specification in an s-expression notation. I
transcribe enough that I could test the resulting parser (assuming it
existed), and try to get some edge cases in there. Then I start
worrying about how to make macros that implement the syntax I came up
with.
Like probably most other Lispers who've tackled problems like this
before, I have my own binary-io library that makes this easier. I
never bothered to make it a Real Library in terms of generality, it
just grew as I needed it. If I were doing this for the first time,
I'd probably take Peter Seibel's library as a starting point, and try
to implement my syntax in terms of that.
Much to my surprise, I really enjoyed doing binary parsing in Lisp the
first time. It came as a sock that it could be so simple and
high-level, and almost fun. So, enjoy!
···@conquest.OCF.Berkeley.EDU (Thomas F. Burdick) writes:
> Could it be that you're stuck thinking about the problem in a
> bottom-up manner? When I have to parse some ugly binary format, I
> start writing out the specification in an s-expression notation. I
> transcribe enough that I could test the resulting parser (assuming it
> existed), and try to get some edge cases in there. Then I start
> worrying about how to make macros that implement the syntax I came up
> with.
Sounds interesting. Can you give a very short example? This, to me, is
one of Lisps greatest strengths and I keep looking for more examples
of it.
> Like probably most other Lispers who've tackled problems like this
> before, I have my own binary-io library that makes this easier. I
> never bothered to make it a Real Library in terms of generality, it
> just grew as I needed it. If I were doing this for the first time,
> I'd probably take Peter Seibel's library as a starting point, and try
> to implement my syntax in terms of that.
>
> Much to my surprise, I really enjoyed doing binary parsing in Lisp the
> first time. It came as a sock that it could be so simple and
> high-level, and almost fun. So, enjoy!
How have you found the performance of your binary parsing routines?
--
Dave Roberts
dave -remove- AT findinglisp DoT com
http://www.findinglisp.com/
Dave Roberts <···········@remove-findinglisp.com> writes:
> ···@conquest.OCF.Berkeley.EDU (Thomas F. Burdick) writes:
>
>> Could it be that you're stuck thinking about the problem in a
>> bottom-up manner? When I have to parse some ugly binary format, I
>> start writing out the specification in an s-expression notation. I
>> transcribe enough that I could test the resulting parser (assuming it
>> existed), and try to get some edge cases in there. Then I start
>> worrying about how to make macros that implement the syntax I came up
>> with.
>
> Sounds interesting. Can you give a very short example? This, to me, is
> one of Lisps greatest strengths and I keep looking for more examples
> of it.
I found Will Hartung's opus on this topic to be pretty inspirational:
http://groups-beta.google.com/group/comp.lang.lisp/msg/86cf454beb8a42f9
Zach
Dave Roberts <···········@remove-findinglisp.com> writes:
> ···@conquest.OCF.Berkeley.EDU (Thomas F. Burdick) writes:
>
>> Could it be that you're stuck thinking about the problem in a
>> bottom-up manner? When I have to parse some ugly binary format, I
>> start writing out the specification in an s-expression notation. I
>> transcribe enough that I could test the resulting parser (assuming it
>> existed), and try to get some edge cases in there. Then I start
>> worrying about how to make macros that implement the syntax I came up
>> with.
>
> Sounds interesting. Can you give a very short example? This, to me, is
> one of Lisps greatest strengths and I keep looking for more examples
> of it.
Here's one from the chapter of my book previously mentioned:
(define-tagged-binary-class id3-tag ()
((identifier (iso-8859-1-string :length 3))
(major-version u1)
(revision u1)
(flags u1)
(size id3-tag-size))
(:dispatch
(ecase major-version
(2 'id3v2.2-tag)
(3 'id3v2.3-tag))))
This defines a base class ID3-TAG as well as methods that knows how to
read the three-character "ID3" identifier, the major version,
revision, flags byte, and the weirdly-encoded size of the id3 tag and
then uses the value of the MAJOR-VERSION just read to instantiate
either an ID3V2.2-TAG or an ID3V2.3-TAG. They are defined like this:
(define-binary-class id3v2.2-tag (id3-tag)
((frames (id3-frames :tag-size size :frame-type 'id3v2.2-frame))))
(define-binary-class id3v2.3-tag (id3-tag)
((extended-header-size (optional :type 'u4 :if (extended-p flags)))
(extra-flags (optional :type 'u2 :if (extended-p flags)))
(padding-size (optional :type 'u4 :if (extended-p flags)))
(crc (optional :type 'u4 :if (crc-p flags extra-flags)))
(frames (id3-frames :tag-size size :frame-type 'id3v2.3-frame))))
From these definitions you get class definitons and methods for
reading and writing these types such that you can now say:
(with-open-file (in file :element-type '(unsigned-byte 8))
(read-value 'id3-tag in)))
and:
(with-open-file (out file :direction :output :element-type '(unsigned-byte 8))
(write-value 'id3-tag out some-id3-tag))
There's a bit more to it, of course, since you have to define the
individual types such as ISO-8859-1, U1, and ID3-TAG-SIZE as well as
things like OPTIONAL. But even that's pretty simple. Check out:
<http://www.gigamonkeys.com/book/practical-parsing-binary-files.html>
<http://www.gigamonkeys.com/book/practical-an-id3-parser.html>
for all the details.
-Peter
--
Peter Seibel ·····@gigamonkeys.com
Lisp is the red pill. -- John Fraser, comp.lang.lisp
Peter Seibel <·····@gigamonkeys.com> writes:
> Dave Roberts <···········@remove-findinglisp.com> writes:
>
> > ···@conquest.OCF.Berkeley.EDU (Thomas F. Burdick) writes:
> >
> >> Could it be that you're stuck thinking about the problem in a
> >> bottom-up manner? When I have to parse some ugly binary format, I
> >> start writing out the specification in an s-expression notation. I
> >> transcribe enough that I could test the resulting parser (assuming it
> >> existed), and try to get some edge cases in there. Then I start
> >> worrying about how to make macros that implement the syntax I came up
> >> with.
> >
> > Sounds interesting. Can you give a very short example? This, to me, is
> > one of Lisps greatest strengths and I keep looking for more examples
> > of it.
>
> Here's one from the chapter of my book previously mentioned:
>
> (define-tagged-binary-class id3-tag ()
> ((identifier (iso-8859-1-string :length 3))
> (major-version u1)
> (revision u1)
> (flags u1)
> (size id3-tag-size))
> (:dispatch
> (ecase major-version
> (2 'id3v2.2-tag)
> (3 'id3v2.3-tag))))
Peter, I have actually read that chapter of your book already. Very
cool stuff. I particularly like the way you automatically dispatch to
the next portion of the format depending on the values of other
fields.
In this case, I was just wanting to see if Thomas had come up with
something a bit different. One of the interesting things about mini
macro languages is that there are so many interesting and different
ideas.
--
Dave Roberts
dave -remove- AT findinglisp DoT com
http://www.findinglisp.com/
Dave Roberts <···········@remove-findinglisp.com> writes:
> ···@conquest.OCF.Berkeley.EDU (Thomas F. Burdick) writes:
>
>> Could it be that you're stuck thinking about the problem in a
>> bottom-up manner? When I have to parse some ugly binary format, I
>> start writing out the specification in an s-expression notation. I
>> transcribe enough that I could test the resulting parser (assuming it
>> existed), and try to get some edge cases in there. Then I start
>> worrying about how to make macros that implement the syntax I came up
>> with.
>
> Sounds interesting. Can you give a very short example? This, to me, is
> one of Lisps greatest strengths and I keep looking for more examples
> of it.
not that this has anything to do with binary formats, but: i just now
noticed that i wrote, about four times, more or less the same code, it
all looked like this:
(defcomponent <NAME> ()
...slots...)
(defmethod label ((x <NAME>))
(format nil "<SOMETHING>" (label (presentation x))))
(defmethod apply-criteria ((x <NAME>) instance)
...logic...)
So i _first_ transformed all those forms into:
(defslot-critera <NAME>
(...slots...)
:label "<SOMETHING>"
:apply-criteria (lambda (x inst) ...logic...))
I then checked that i could, for all the cases that came to mind,
generate the various defcomponent and defmethod forms from my
defslot-critera macro. at this point i wrote the defslot-critera
macro.
--
-Marco
Ring the bells that still can ring.
Forget the perfect offering.
There is a crack in everything.
That's how the light gets in.
-Leonard Cohen