Is there a way to obtain a listing of all files, symlinks and
subdirectories of a directory using Common Lisp? The directory
function is close but due to generating a listing of truenames
basically drops the symlinks. Under sbcl, I achieved the goal using
opendir, readdir and closedir from the sb-posix package. But, I figure
there must be a better more portable option.
Jeff wrote:
> Is there a way to obtain a listing of all files, symlinks and
> subdirectories of a directory using Common Lisp? The directory
> function is close but due to generating a listing of truenames
> basically drops the symlinks. Under sbcl, I achieved the goal using
> opendir, readdir and closedir from the sb-posix package. But, I figure
> there must be a better more portable option.
Sadly, Common Lisp's pathname functions are so generic that they can not
be used to generate this. You could use wrapper libraries like CL-FAD
that wrap up the differences between each implementation.
If that's not enough, you could use UFFI <http://uffi.b9.com/> to define
an interface to opendir, readdir, and closedir that supports more than
just SBCL. Here's something I cooked up in a five minutes:
> (use-package :uffi)
>
> (def-struct dirent
> (d_ino :int)
> (d_off :int)
> (d_reclen :unsigned-short)
> (d_type :unsigned-char)
> (d_name (:array :char 256)))
>
> (def-function "opendir"
> ((dirname :cstring))
> :returning :pointer-void)
>
> (def-function "closedir"
> ((dirp :pointer-void))
> :returning :int)
>
> (def-function "readdir"
> ((dirp :pointer-void))
> :returning (* dirent))
>
> (defun directory- (dir)
> "Like cl:directory, but using UFFI."
> (let ((dirp (opendir dir)))
> (when (null-pointer-p dirp)
> (error "~A is missing or not a directory" dir))
> (unwind-protect
> (loop for dirent = (readdir dirp) then (readdir dirp)
> until (null-pointer-p dirent)
> collecting (convert-from-foreign-string (get-slot-value dirent 'dirent 'd_name)))
> (closedir dirp))))
-- MJF
Hello!
M Jared Finder <·····@hpalace.com> wrote:
>Jeff wrote:
>> Is there a way to obtain a listing of all files, symlinks and
>> subdirectories of a directory using Common Lisp? The directory
>> function is close but due to generating a listing of truenames
>> basically drops the symlinks. Under sbcl, I achieved the goal using
>> opendir, readdir and closedir from the sb-posix package. But, I figure
>> there must be a better more portable option.
>Sadly, Common Lisp's pathname functions are so generic that they can not
>be used to generate this. You could use wrapper libraries like CL-FAD
>that wrap up the differences between each implementation.
>If that's not enough, you could use UFFI <http://uffi.b9.com/> to define
>an interface to opendir, readdir, and closedir that supports more than
>just SBCL. Here's something I cooked up in a five minutes:
>[...]
Or you could access the fts(3) family of functions, which is available
at least on Linux and *BSD. Additional convenience, especially if you
want to traverse a whole directory subtree.
Kind regards,
Hannah.
Hello!
M Jared Finder <·····@hpalace.com> wrote:
>[...]
>> (use-package :uffi)
>> (def-struct dirent
>> (d_ino :int)
>> (d_off :int)
>> (d_reclen :unsigned-short)
>> (d_type :unsigned-char)
>> (d_name (:array :char 256)))
Will fail on OpenBSD I guess (probably also the other BSDs).
There, the structure is defined like this:
struct dirent {
u_int32_t d_fileno; /* file number of entry */
u_int16_t d_reclen; /* length of this record */
u_int8_t d_type; /* file type, see below */
u_int8_t d_namlen; /* length of string in d_name */
#ifdef _POSIX_SOURCE
char d_name[255 + 1]; /* name must be no longer than this */
#else
#define MAXNAMLEN 255
char d_name[MAXNAMLEN + 1]; /* name must be no longer than this */
#endif
};
>[... the rest looks quite ok ...]
Kind regards,
Hannah.
Thanks to everyone who replied. It confirmed that I would need to
directly interface system api's and could not achieve results via
standard common lisp functions. I came up with something similar to
the above without the uffi and the unwind-protect. I have since added
the unwind-protect which leads to my next question. Shouldn't the
unwind protect also encompass the (let ((dirp ...))) since this opens
the directory? Seems there would be a slight chance for failure
between the let and the unwind-protect that would leave a directory
opened. However, I was unable to come up with something that moved the
(opendir ...) into the unwind-protect that didn't seem like a hack to
reflect the (opendir ...) binding to the closing form of unwind-protect.
"Jeff" <··········@bigfoot.com> writes:
> Thanks to everyone who replied. It confirmed that I would need to
> directly interface system api's and could not achieve results via
> standard common lisp functions. I came up with something similar to
> the above without the uffi and the unwind-protect. I have since added
> the unwind-protect which leads to my next question. Shouldn't the
> unwind protect also encompass the (let ((dirp ...))) since this opens
> the directory? Seems there would be a slight chance for failure
> between the let and the unwind-protect that would leave a directory
> opened. However, I was unable to come up with something that moved the
> (opendir ...) into the unwind-protect that didn't seem like a hack to
> reflect the (opendir ...) binding to the closing form of unwind-protect.
What failure can you get between opendir and unwind-protect?
What could the program do if the processor cannot even go from the end
of one instruction to the beginning of the next?
--
__Pascal Bourguignon__ http://www.informatimago.com/
Until real software engineering is developed, the next best practice
is to develop with a dynamic system that has extreme late binding in
all aspects. The first system to really do this in an important way
is Lisp. -- Alan Kay
From: Kent M Pitman
Subject: Re: Newbie: Directory listing retaining symlinks
Date:
Message-ID: <u64un36lj.fsf@nhplace.com>
Pascal Bourguignon <····@mouse-potato.com> writes:
> "Jeff" <··········@bigfoot.com> writes:
>
> > Thanks to everyone who replied. It confirmed that I would need to
> > directly interface system api's and could not achieve results via
> > standard common lisp functions. I came up with something similar to
> > the above without the uffi and the unwind-protect. I have since added
> > the unwind-protect which leads to my next question. Shouldn't the
> > unwind protect also encompass the (let ((dirp ...))) since this opens
> > the directory? Seems there would be a slight chance for failure
> > between the let and the unwind-protect that would leave a directory
> > opened. However, I was unable to come up with something that moved the
> > (opendir ...) into the unwind-protect that didn't seem like a hack to
> > reflect the (opendir ...) binding to the closing form of unwind-protect.
I didn't read up-thread to see the original code, but the form of your
argument sounds correct. You want to try to close the window as much
as you can.
HOWEVER, IMO, it's impossible to write a reliably correct macro
without OPEN taking a locative to set and making sure the actual file
open is done (internally to the OPEN function) within a critical
section as an atomic operation. (CL has no portable way to ask for
such a critical section.) As a result, there's always the possibility
in principle that the interrupt will happen here
|
+---------------------------------+
|
v
(let ((var (function-call))) ...)
and the result will have been computed but not stored.
So one tries one's best, but it's theoreticaly a bit dicey. This has
bothered me for a goodly long question. I have no good answer to it.
But I've also never seen it be a problem in practice either....
> What failure can you get between opendir and unwind-protect?
>
> What could the program do if the processor cannot even go from the end
> of one instruction to the beginning of the next?
It depends on the implementation.
Some implementations can take interrupts at arbitrary places, some
treat forward code blocks as if they were in no-interrupt mode, only
polling for interrupts on, for example, backward jumps and function
calls, or some such thing. But since a writer of conforming code
cannot make assumptions about such things, one has to assume that a
keyboard, timer, or other system interrupt (power failing, temperature
too high, etc.) is at least a possibility even in non-multithreaded
Lisps. In a multithreaded environment where there's a
PROCESS-INTERRUPT function, there is the additional possibility (again
depending on how the Lisp implementation deals with interruptibility)
that your program is interruptible between any two arbitrary instructions.
On 2 Aug 2005 22:18:27 -0700, "Jeff" <··········@bigfoot.com> wrote:
> Shouldn't the unwind protect also encompass the (let ((dirp ...)))
> since this opens the directory? Seems there would be a slight
> chance for failure between the let and the unwind-protect that would
> leave a directory opened. However, I was unable to come up with
> something that moved the (opendir ...) into the unwind-protect that
> didn't seem like a hack to reflect the (opendir ...) binding to the
> closing form of unwind-protect.
The usual idiom is
(let (dir)
(unwind-protect
(progn
(setq dir (opendir ...))
(do-something))
(when dir
(clean-up-behind dir))))
Cheers,
Edi.
--
Lisp is not dead, it just smells funny.
Real email: (replace (subseq ·········@agharta.de" 5) "edi")
> The usual idiom is
>
> (let (dir)
> (unwind-protect
> (progn
> (setq dir (opendir ...))
> (do-something))
> (when dir
> (clean-up-behind dir))))
>
> Cheers,
> Edi.
This idiom is what my lisp newbieness referred to as a hack (probably
incorrectly). I begun reading "On Lisp" after reading "Practical
Common Lisp". In "On Lisp", Paul Graham lists a few functions as
having a theoretical "tax" and a good indication that another more
lispy approach exists. SETQ being one of these identified functions, I
thought there might be a more lispy idiom. Perhaps, this a situation
where it makes sense to pay the "tax".
Thanks again everyone, it's been quite helpful and reassuring.
Edi Weitz <········@agharta.de> writes:
> The usual idiom is
>
> (let (dir)
> (unwind-protect
> (progn
> (setq dir (opendir ...))
> (do-something))
> (when dir
> (clean-up-behind dir))))
if there is a possibility of error between the return from open and
the binding it this:
(let ((dir (opendir ...)))
(unwind-protect ...))
why can't there be an error between the rutrun from open the setting
in the code you pasted above? (this would cause the directory to be
open'd but DIR would still be NIL since the setting of DIR was the
operation which caused the error)
--
-Marco
Ring the bells that still can ring.
Forget the perfect offering.
There is a crack in everything.
That's how the light gets in.
-Leonard Cohen
From: Kent M Pitman
Subject: Re: Newbie: Directory listing retaining symlinks
Date:
Message-ID: <umzny3ncp.fsf@nhplace.com>
"Marco Baringer" <··@bese.it> writes:
> Edi Weitz <········@agharta.de> writes:
>
> > The usual idiom is
> >
> > (let (dir)
> > (unwind-protect
> > (progn
> > (setq dir (opendir ...))
> > (do-something))
> > (when dir
> > (clean-up-behind dir))))
>
> if there is a possibility of error between the return from open and
> the binding it this:
>
> (let ((dir (opendir ...)))
> (unwind-protect ...))
>
> why can't there be an error between the rutrun from open the setting
> in the code you pasted above? (this would cause the directory to be
> open'd but DIR would still be NIL since the setting of DIR was the
> operation which caused the error)
This is an interesting question, and the answer might be that you're no
worse off in practice, but I think there's a possibility you are worse off.
I believe that some compilers poll for interrupts on backward jumps (to make
sure they don't get infinite loops) and function calls. I don't know that they
do on function returns, which are a bounded activity in a system without
tail calling and can't themselves lead to loops. I'd imagine then that
in such an implementation, if they can exist, the indicated points are
points of concern:
(let ((dir (foo))) (unwind-protect (bar dir) (cleanup)))
^ ^ ^
1 2a 2b
But in here:
(let ((dir nil))
(unwind-protect (progn (setq dir (foo)) (bar dir))
^
(when dir (cleanup)))))
there's only one unsafe point. The reason is that what appears to be
clearly inside the parens may not be once you translate to an instruction
stream, so in fact, in the first example, 2a and 2b are the same point
and the compiler will make a judgment call about whether they are inside
or outside the unwind protect as it collapses things into instructions.
A similar argument is made by Moon in issue UNWIND-PROTECT-NON-LOCAL-EXIT
(superseded by issue EXIT-EXTENT, which appears in CLHS X3J13 issue index)
for talking about how things like
(block foo
(block bar
(unwind-protect (return-from foo 'foo)
(return-from bar 'bar))))
and so on are problematic because FOO and BAR represent the identical
position in the code, so it's odd [at least under some analyses] to
say that one of these RETURN-FROM's is ok while the other is not. The
version of his proposal where this was mentioned did not pass, but it
illustrates a way of thinking that is not prohibited in the analogy-space
of the other issue we're discussing above.
[If you're looking for another analogy, it's the issue of what
<p>Foo <a href="foo"> Bar</a></p>
should mean if you were designing HTML and trying to decide whether the
space before Bar should be part of the underline or not, given that
whitespace is consolidated...]
There are certainly ways in which implementations can define things such
that these issues go away. But my point is that CL itself does not nail
down one of those interpretations as the canonical one, so writing best
code you need to accommodate the full space of possible implementations,
and so were I you, I'd prefer in this case to go for the more conservatively
written expression.
- - - -
I think, by the way, that one reason that with-open-file takes only a single
file is to address the problem of 'let-binding' (rather than let*) and holding
onto the temporaries. It's trickier to avoid intermediate windows when there
are more streams involved. I did a pretty good job in the IOTA macro I wrote
for Maclisp, which took the syntax
(iota ((var1 ...) (var2 ...) ...) ...body...)
but in the end the community went with the simpler WITH-OPEN-FILE design
probably to avoid the hair you have to do in the macro expansion to get this
right.
"Jeff" <··········@bigfoot.com> writes:
> Is there a way to obtain a listing of all files, symlinks and
> subdirectories of a directory using Common Lisp? The directory
> function is close but due to generating a listing of truenames
> basically drops the symlinks. Under sbcl, I achieved the goal using
> opendir, readdir and closedir from the sb-posix package. But, I figure
> there must be a better more portable option.
No, there is not.
--
__Pascal Bourguignon__ http://www.informatimago.com/
This is a signature virus. Add me to your signature and help me to live