Hello,
I'm playing with pathnames under Xanalys Lispworks (4.2.0, MS WinNT
4.0).
I wanted to get directory structure under some dir for further processing.
I was surprised what is going on, why is it so slow under Lispworks?
It's a listing of contents of CD-ROM with over 17000 files in it.
In winnt cmd simple listing lasts almost 4 seconds:
C:\>timethis "dir /s s: > null"
TimeThis : Command Line : dir /s s: > null
TimeThis : Start Time : Sun Feb 09 13:40:33 2003
TimeThis : Command Line : dir /s s: > null
TimeThis : Start Time : Sun Feb 09 13:40:33 2003
TimeThis : End Time : Sun Feb 09 13:40:37 2003
TimeThis : Elapsed Time : 00:00:03.765
Then this form typed in Lispworks listener (hence not compiled) yields
almost 11 minutes?:
CL-USER 1 > (time (length (setf tmp (directory (make-pathname :host "s"
:directory '(:absolute :wild-inferiors))))))
Timing the evaluation of (LENGTH (SETF TMP (DIRECTORY (MAKE-PATHNAME :HOST
"s" :DIRECTORY (QUOTE (:ABSOLUTE :WILD-INFERIORS))))))
; Loading fasl file C:\Program Files\Xanalys\LispWorks
Personal\lib\4-2-0-0\modules\util\callcoun.fsl
user time = 448.324
system time = 81.847
Elapsed time = 0:11:23
Allocation = 37815184 bytes standard / 10360086 bytes fixlen
0 Page faults
17385
while in Clisp 2.29 its equivalent (as I hope):
[33]> (time (length (setf tmp (directory (make-pathname :device "S"
:directory '
(:absolute :wild-inferiors) :name :wild)))))
Real time: 1.532 sec.
Run time: 1.151656 sec.
Space: 1920852 Bytes
GC: 3, GC time: 0.6208928 sec.
17091
I tried following in Lispworks, compiled:
(defvar *default-root-pathname* (make-pathname :host "s" :directory
'(:absolute)))
(defun create-dirtree2 (&optional (pathname *default-root-pathname*))
"just some stub, that right now only descends down through subdirs and
files under given pathname"
(when (lw:file-directory-p pathname)
(dolist (item (directory (merge-pathnames (make-pathname :name :wild
:type :wild :version :wild)
pathname)))
(create-dirtree2 item))))
CL-USER 11 > (time (create-dirtree2))
Timing the evaluation of (CREATE-DIRTREE2)
user time = 47.147
system time = 77.421
Elapsed time = 0:04:24
Allocation = 37694064 bytes standard / 9705740 bytes fixlen
0 Page faults
NIL
... it is better but... again 4 minutes for 4 seconds directory "listing"?
Or maybe I'm doing something wrong here?
Regards
Frantisek Stafek
"frantisek stafek" <·······@noise.cz> ha scritto nel messaggio ··················@news.vol.cz...
> Hello,
> I'm playing with pathnames under Xanalys Lispworks (4.2.0, MS WinNT
> 4.0).
> I wanted to get directory structure under some dir for further processing.
> I was surprised what is going on, why is it so slow under Lispworks?
> It's a listing of contents of CD-ROM with over 17000 files in it.
> In winnt cmd simple listing lasts almost 4 seconds:
This is a known performance bug of the current version of Lispworks.
As a workaround, on Windows, put the following in your .lispwork
(suggestion from Xanalys):
;; Why is DELETE-DUPLICATES needed at all?
;;
;; Because the filesystem may contain symbolic links. You can actually
;; prevent the call to DELETE-DUPLICATES by
#+win32
(SETQ SYS::*DIRECTORY-LINK-TRANSPARENCY* NIL)
;; Since symbolic links are not usually possible on Windows we will alter
;; the configuration of LWW with this setting.
================================
Or, for all systems, you can do the following (my code. I don't use lw anymore,
and I cannot check it, use at your own risk. *THIS IS A KLUDGE*):
;;; PVI was a package of my extensions, use whatever package you want.
(export 'pvi::delete-duplicates-hash :pvi)
(defun pvi:delete-duplicates-hash
(list &key (test 'eql) (from-end nil) (size-hash (length list)))
(let ((rlist (if from-end list (nreverse list))))
(loop with table = (make-hash-table :test test :size size-hash)
for item in rlist
unless (gethash item table)
collect item
do (setf (gethash item table) t))))
(defvar cl::*delete-duplicates-kludge* nil)
(defparameter cl::*tests* (list 'eq 'eql 'equal 'equalp
#'eq #'eql #'equal #'equalp))
(defadvice (delete-duplicates hash-kludge :around)
(sequence &rest rest
&key
(from-end nil from-end-p)
(test nil testp)
(test-not nil test-not-p)
(start nil startp)
(end nil endp)
(key nil keyp))
(declare (ignore from-end test-not start end key))
(if (and cl::*delete-duplicates-kludge*
(not (or test-not-p startp endp keyp))
(member test cl::*tests*)
(listp sequence))
(ecase cl::*delete-duplicates-kludge*
(:hash (pvi:delete-duplicates-hash sequence :test test))
(:fake sequence))
(apply #'call-next-advice sequence rest)))
==============
And put a (let ((cl::*delete-duplicates-kludge* :hash)) ...) arond your call
to DIRECTORY.
HTH
P.
Pierpaolo BERNARDI wrote:
> "frantisek stafek" <·······@noise.cz> ha scritto nel messaggio ··················@news.vol.cz...
>
>>Hello,
>> I'm playing with pathnames under Xanalys Lispworks (4.2.0, MS WinNT
>>4.0).
>>I wanted to get directory structure under some dir for further processing.
>>I was surprised what is going on, why is it so slow under Lispworks?
>>It's a listing of contents of CD-ROM with over 17000 files in it.
>>In winnt cmd simple listing lasts almost 4 seconds:
>
>
> This is a known performance bug of the current version of Lispworks.
>
> As a workaround, on Windows, put the following in your .lispwork
> (suggestion from Xanalys):
>
> ;; Why is DELETE-DUPLICATES needed at all?
> ;;
> ;; Because the filesystem may contain symbolic links. You can actually
> ;; prevent the call to DELETE-DUPLICATES by
>
> #+win32
> (SETQ SYS::*DIRECTORY-LINK-TRANSPARENCY* NIL)
>
> ;; Since symbolic links are not usually possible on Windows we will alter
> ;; the configuration of LWW with this setting.
But symbolic links are possible on WinNT. Have a look at
http://www.sysinternals.com/ntw2k/source/misc.shtml#junction
>
> ================================
>
> Or, for all systems, you can do the following (my code. I don't use lw anymore,
> and I cannot check it, use at your own risk. *THIS IS A KLUDGE*):
>
> ;;; PVI was a package of my extensions, use whatever package you want.
> (export 'pvi::delete-duplicates-hash :pvi)
>
> (defun pvi:delete-duplicates-hash
> (list &key (test 'eql) (from-end nil) (size-hash (length list)))
> (let ((rlist (if from-end list (nreverse list))))
> (loop with table = (make-hash-table :test test :size size-hash)
> for item in rlist
> unless (gethash item table)
> collect item
> do (setf (gethash item table) t))))
>
> (defvar cl::*delete-duplicates-kludge* nil)
>
> (defparameter cl::*tests* (list 'eq 'eql 'equal 'equalp
> #'eq #'eql #'equal #'equalp))
>
> (defadvice (delete-duplicates hash-kludge :around)
> (sequence &rest rest
> &key
> (from-end nil from-end-p)
> (test nil testp)
> (test-not nil test-not-p)
> (start nil startp)
> (end nil endp)
> (key nil keyp))
> (declare (ignore from-end test-not start end key))
> (if (and cl::*delete-duplicates-kludge*
> (not (or test-not-p startp endp keyp))
> (member test cl::*tests*)
> (listp sequence))
> (ecase cl::*delete-duplicates-kludge*
> (:hash (pvi:delete-duplicates-hash sequence :test test))
> (:fake sequence))
> (apply #'call-next-advice sequence rest)))
>
> ==============
>
> And put a (let ((cl::*delete-duplicates-kludge* :hash)) ...) arond your call
> to DIRECTORY.
>
> HTH
> P.
>
It is faster naw twice:-) that is, I get the listing in 22 seconds :-)
Timing the evaluation of (LENGTH (SETF TMP (DIRECTORY (MAKE-PATHNAME :HOST
"s" :DIRECTORY (QUOTE (:ABSOLUTE :WILD-INFERIORS))))))
user time = 25.777
system time = 52.725
Elapsed time = 0:02:57
Allocation = 26678808 bytes standard / 6484423 bytes fixlen
0 Page faults
17385
or:
Timing the evaluation of (CREATE-DIRTREE2)
user time = 20.559
system time = 51.854
Elapsed time = 0:02:51
Allocation = 26093664 bytes standard / 6247021 bytes fixlen
0 Page faults
NIL
Frantisek Stafek