From: Tonguc Yumruk
Subject: AllegroServe with SBCL and Unicode
Date: 
Message-ID: <87mz63rhdn.fsf@terra.galaxy>
--=-=-=
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 8bit

Hi,

I'm working on a web application using AllegroServe and YaclML on
SBCL. I'm using YaclML for templating and use tal files for
presentation. My working environment is:

Debian GNU/Linux testing/Etch SBCL 0.9.16 (Debian Package)
AllegroServe 1.2.42 (Debian Package) YaclML 2006-10-03 Snapshot
(Debian Package) -- Slime 2006-11-12 (CVS Snapshot) Emacs 21.4a

The problem is that, AllegroServe does not handle Unicode characters
at all. When it sees a unicode character in output complains that the
value is not (unsigned-byte 8) and stops. The code is attached. Sorry
that the tal file is in Turkish.

I've also tried using the "vbz" version of AllegroServe [1]. It
rendered all of the page very well but with weird characters instead
of the ones I was using. For example it replaced all occurences of ı
(small letter i without dot) with "1". I really don't know too much
about external formats or where they are used so I can't spot the
problem clearly. But I think it has something o do with gray-streams.

Thanks for all help


--=-=-=
Content-Type: text/plain; charset=utf-8
Content-Disposition: attachment; filename=test.tal
Content-Transfer-Encoding: 8bit
Content-Description: The tal template for the above code

<html xmlns="http://www.w3.org/1999/xhtml"
      xmlns:tal="http://common-lisp.net/project/bese/tal/core">
<head>
<title tal:content="$page.title">Test</title>
</head>
<body>
<p>Merhaba şŞğĞıİ<span tal:replace="$user.name" tal:escape-html="t">Kullanıcı adı</span>. Kullanıcı adin: <span tal:replace="$user.id" tal:escape-html="t">userID</span></p>

<div class="entry">
<div tal:content="$entry.title" class="entrytitle">Baslik</div>
<div tal:content="$entry.body" class="entrybody">Metin</div>
<div tal:content="$entry.date" class="entrydate">2000-01-01</div>
</div>

<div class="comments">
<ul>
<li tal:dolist="$entry.comments"><b tal:content="$comment.title">Yorum basligi</b><br/>
<span tal:replace="$comment.body">Yorum icerigi</span><br/></li>
</ul>
</div>
</body>
</html>
--=-=-=
Content-Type: text/plain; charset=utf-8
Content-Disposition: attachment; filename=test.lisp
Content-Transfer-Encoding: 8bit
Content-Description: The test code

(require :aserve)
(require :yaclml)

(defpackage :net.oneliner
  (:use :common-lisp :net.aserve))

(in-package :net.oneliner)

;;Setup TAL Environment
(defvar *user* (make-hash-table))
(defvar *entry* (make-hash-table))
(defvar *comments* (make-hash-table))
(defvar *page* (make-hash-table))

(setf yaclml:*yaclml-indent* nil)

;;Setup user environment
(setf (gethash 'user.id *user*) "tongucyumruk")
(setf (gethash 'user.name *user*) "Tonguç Yumruk")

;;Setup entry environment
(setf (gethash 'entry.id *entry*) "42")
(setf (gethash 'entry.title *entry*) "Test girdisi")
(setf (gethash 'entry.body *entry*) "cat /dev/hda > /tmp/debian.iso")
(setf (gethash 'entry.desc *entry*) "Bir CD'den iso çıkartmak için pratik bir yol")
(setf (gethash 'entry.date *entry*) "2006-12-04")

;;Define comments
(defparameter *comment1* (make-hash-table))
(setf (gethash 'comment.id *comment1*) "4242")
(setf (gethash 'comment.title *comment1*) "Test yorumu 1")
(setf (gethash 'comment.body *comment1*) "Bu bir test yorumudur.")

(defparameter *comment2* (make-hash-table))
(setf (gethash 'comment.id *comment2*) "2424")
(setf (gethash 'comment.title *comment2*) "Test yorumu 2")
(setf (gethash 'comment.body *comment2*) "Bu ikinci test yorumudur.")

(defparameter *comment3* (make-hash-table))
(setf (gethash 'comment.id *comment3*) "4242")
(setf (gethash 'comment.title *comment3*) "Test yorumu 3")
(setf (gethash 'comment.body *comment3*) "Bu üçüncü test yorumudur.")

;;Setup the comments for entry
(setf (gethash 'entry.comments *entry*) (list (list *comment1*)
					      (list *comment2*)
					      (list *comment3*)))

;;Setup the page environment
(setf (gethash 'page.title *page*) "Örnek sayfa")
(setf (gethash 'page.footer *page*) "Kopirayt mopirayt")

;;Define the TAL generator
(defvar *jenerator* (make-instance 'yaclml:file-system-generator
				   :root-directories (list #P"./tal/")))

;;Create the closure for test page
(defvar *tal-closure* (yaclml:load-tal *jenerator* "test.tal"))

(setf *default-aserve-external-format* :latin-5)
(start :port 8080)

;;Publish the page
(publish
 :path "/test"
 :function
 #'(lambda (req ent)
     (with-http-response (req ent)
       (with-http-body (req ent)
	 (yaclml:with-yaclml-stream (request-reply-stream req)
	   (funcall *tal-closure* (yaclml:make-standard-environment *user* *entry* *page*) *jenerator*))))))
--=-=-=
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 8bit



[1] http://www.cliki.net/Portable%20Allegroserve%20Enhanced%20Version%20for%20SBCL
-- 
Love Respect GNU/Linux
########################################################################
BOFH excuse #421:
  
Domain controller not responding
########################################################################
Tonguç Yumruk

--=-=-=--

From: Pascal Bourguignon
Subject: Re: AllegroServe with SBCL and Unicode
Date: 
Message-ID: <871wnf26ts.fsf@thalassa.informatimago.com>
Tonguc Yumruk <············@member.fsf.org> writes:
> [...]
> I've also tried using the "vbz" version of AllegroServe [1]. It
> rendered all of the page very well but with weird characters instead
> of the ones I was using. For example it replaced all occurences of ı
> (small letter i without dot) with "1". I really don't know too much
> about external formats or where they are used so I can't spot the
> problem clearly. But I think it has something o do with gray-streams.

> [...]
> (setf *default-aserve-external-format* :latin-5)
> [...]

Use :utf-8 with vbz.

-- 
__Pascal Bourguignon__                     http://www.informatimago.com/

THIS IS A 100% MATTER PRODUCT: In the unlikely event that this
merchandise should contact antimatter in any form, a catastrophic
explosion will result.
From: Pascal Bourguignon
Subject: Re: AllegroServe with SBCL and Unicode
Date: 
Message-ID: <87wt57zwd2.fsf@thalassa.informatimago.com>
Pascal Bourguignon <···@informatimago.com> writes:

> Tonguc Yumruk <············@member.fsf.org> writes:
>> [...]
>> I've also tried using the "vbz" version of AllegroServe [1]. It
>> rendered all of the page very well but with weird characters instead
>> of the ones I was using. For example it replaced all occurences of ı
>> (small letter i without dot) with "1". I really don't know too much
>> about external formats or where they are used so I can't spot the
>> problem clearly. But I think it has something o do with gray-streams.
>
>> [...]
>> (setf *default-aserve-external-format* :latin-5)
>> [...]
>
> Use :utf-8 with vbz.

And of course, save your sources in utf-8, 

    ;; -*- coding:utf-8 -*- ;; on the first line.

and load them in sbcl as utf-8!

    (load "source.lisp" :external-format :utf-8)
or:
    (SETF SB-IMPL::*DEFAULT-EXTERNAL-FORMAT* :UTF-8)
    (load "source.lisp")


-- 
__Pascal Bourguignon__                     http://www.informatimago.com/

THIS IS A 100% MATTER PRODUCT: In the unlikely event that this
merchandise should contact antimatter in any form, a catastrophic
explosion will result.
From: Tonguc Yumruk
Subject: Re: AllegroServe with SBCL and Unicode
Date: 
Message-ID: <87irgrrg34.fsf@terra.galaxy>
Pascal Bourguignon <···@informatimago.com> writes:

> Pascal Bourguignon <···@informatimago.com> writes:
>
>> Use :utf-8 with vbz.
>
> And of course, save your sources in utf-8, 
>
>     ;; -*- coding:utf-8 -*- ;; on the first line.
>
> and load them in sbcl as utf-8!
>
>     (load "source.lisp" :external-format :utf-8)
> or:
>     (SETF SB-IMPL::*DEFAULT-EXTERNAL-FORMAT* :UTF-8)
>     (load "source.lisp")
>
>

Oh, sorry. I've actually used utf-8 but I've sent the test file with
latin-5. My SBCL already uses utf-8 as *default-external-format* I'm
uploading a sample screenshot, about what it looks like, into
http://tonguc.ath.cx/images/allegro.png

Thanks.

-- 
Love Respect GNU/Linux
########################################################################
Arbitrary systems, pl.n.:
	Systems about which nothing general can be said, save "nothing
	general can be said."
########################################################################
Tonguç Yumruk