I'm trying to protect write access to a file with a lock file, but I'm
using a loop because I can't find a blocking open or something like the
C select() function.
My current implentation is like this:
<code>
(defun write-to-file (data)
(unwind-protect
(progn
(do () ((not (probe-file (merge-pathnames ".lock"))) nil))
(with-open-file (out (merge-pathnames ".lock")
:direction :output :if-does-not-exist :create)
(with-open-file (out (merge-pathnames "foo")
:direction :output :if-exists :append)
(print data out)))
(delete-file (merge-pathnames ".lock")))
(when (probe-file (merge-pathnames ".lock")) (delete-file
(merge-pathnames ".lock")))))
</code>
however, the (do ... ) seems somehow suboptimal. The writes should be
extremely fast, I'm writing tiny amounts of data, so the (do ...) loop
doesn't spin much.
Is there a better approach to this?
> * ···········@gmail.com <···········@tznvy.pbz> [2006-01-25 13:05:25 -0800]:
>
> I'm trying to protect write access to a file with a lock file, but I'm
> using a loop because I can't find a blocking open or something like
> the C select() function.
>
> My current implentation is like this:
>
> <code>
>
> (defun write-to-file (data)
> (unwind-protect
> (progn
> (do () ((not (probe-file (merge-pathnames ".lock"))) nil))
> (with-open-file (out (merge-pathnames ".lock")
> :direction :output :if-does-not-exist :create)
> (with-open-file (out (merge-pathnames "foo")
> :direction :output :if-exists :append)
> (print data out)))
> (delete-file (merge-pathnames ".lock")))
> (when (probe-file (merge-pathnames ".lock")) (delete-file
> (merge-pathnames ".lock")))))
>
> </code>
>
> however, the (do ... ) seems somehow suboptimal. The writes should be
> extremely fast, I'm writing tiny amounts of data, so the (do ...) loop
> doesn't spin much.
>
> Is there a better approach to this?
1. you do not need to call merge-pathnames before passing your pathnames
to pathname functions - they will do that for you anyway.
2. you may suffer from race conditions: two processes discovering that
lock is absent and opening it. I suggest :if-exists nil:
(unwind-protect
(with-open-stream (lock (do (s) ((setq s (open ".lock" :if-exists nil)) s)))
(with-open-stream (out "foo" ...)
...))
(delete-file ".lock"))
3. I suggest that you use OS-level file locking. CLISP offers
OS:STREAM-LOCK, I am sure other implementations are not far behind.
--
Sam Steingold (http://www.podval.org/~sds) running w2k
http://www.honestreporting.com http://www.mideasttruth.com
http://www.jihadwatch.org http://ffii.org http://www.openvotingconsortium.org
Despite the raising cost of living, it remains quite popular.
Sam Steingold wrote:
> > * ···········@gmail.com <···········@tznvy.pbz> [2006-01-25 13:05:25 -0800]:
> >
> > I'm trying to protect write access to a file with a lock file, but I'm
> > using a loop because I can't find a blocking open or something like
> > the C select() function.
> >
> > My current implentation is like this:
> >
> > <code>
> >
> > (defun write-to-file (data)
> > (unwind-protect
> > (progn
> > (do () ((not (probe-file (merge-pathnames ".lock"))) nil))
> > (with-open-file (out (merge-pathnames ".lock")
> > :direction :output :if-does-not-exist :create)
> > (with-open-file (out (merge-pathnames "foo")
> > :direction :output :if-exists :append)
> > (print data out)))
> > (delete-file (merge-pathnames ".lock")))
> > (when (probe-file (merge-pathnames ".lock")) (delete-file
> > (merge-pathnames ".lock")))))
> >
> > </code>
> >
> > however, the (do ... ) seems somehow suboptimal. The writes should be
> > extremely fast, I'm writing tiny amounts of data, so the (do ...) loop
> > doesn't spin much.
> >
> > Is there a better approach to this?
>
> 1. you do not need to call merge-pathnames before passing your pathnames
> to pathname functions - they will do that for you anyway.
>
> 2. you may suffer from race conditions: two processes discovering that
> lock is absent and opening it. I suggest :if-exists nil:
>
> (unwind-protect
> (with-open-stream (lock (do (s) ((setq s (open ".lock" :if-exists nil)) s)))
> (with-open-stream (out "foo" ...)
> ...))
> (delete-file ".lock"))
>
> 3. I suggest that you use OS-level file locking. CLISP offers
> OS:STREAM-LOCK, I am sure other implementations are not far behind.
>
> --
> Sam Steingold (http://www.podval.org/~sds) running w2k
> http://www.honestreporting.com http://www.mideasttruth.com
> http://www.jihadwatch.org http://ffii.org http://www.openvotingconsortium.org
> Despite the raising cost of living, it remains quite popular.
Sam,
Thanks for you advice, your suggested solution works for me.
Regards,
Jeff
>> I'm trying to protect write access to a file with a lock file,
Sam Steingold <···@gnu.org> writes:
> 2. you may suffer from race conditions: two processes discovering that
> lock is absent and opening it. I suggest :if-exists nil:
Even that will probably be unsafe if running over a remote file system
(e.g. NFS). I strongly suggest that you call-out to an established
lock-file library written to use raw syscalls if you care about your
data.
Juliusz
I am grateful for all the advice this topic has generated.
Let me respond with more detail about the project for which this code
is being used. I work for a small company which processes small phone
company records to generate bills, ie performs outsourced billing. The
billing process must be run sequentially (order is unimportant, but
individuality is). However a number of pre-billing "cleanup" and
preparation tasks can be performed in parallel. Generally a process per
customer is started for this prebilling work to be done. At the point
that each customer is ready to be billed, the processing has to stop to
allow only one customer at a time to be processed. At that point I'm
writing the "hook" to the billing process for that customer to a file,
and I'll do it across several processes as each completes the
prebilling phase. Since a human (currently) is performing the
operation, the lock file mechanism is unnecessary due to the fact that
I have to discretely tell each process to queue itself, and I can't do
that to more than one process simultaneously.
Once all processes are in the "queue" (the file written by the code
which started this topic) to be run, I'll start a billing process which
will run them one at a time. Currently, this is being run on a single
windows nt computer by one human. For a future goal, I would like to
write enough lisp to handle the whole thing sans human intervention. To
that end, I'm writing (or trying to write) code to facilitate the
computer doing more of the work than I'm doing. At some point I'll need
to have something like a lock file mechanism in place so either the
computer can safely write to the "queue" file, or more than one person
can perform the prebilling work simultaneously. So while the network
file system and multiprocess issues don't exist for the moment, I'm
trying to consider them when I write this code.
To whit, I have this code for a currently working solution:
(implemented with CLISP)
<code>
(defun write-to-file (data)
(unwind-protect
(with-open-stream
(lock (do (s) ((setq s (open ".lock" :direction :output
:if-exists nil
:if-does-not-exist :create)) s)
(sleep 0.02))))
;; 0-length files in windows allow more than one process
;; to have an exclusive lock - therefore add some bytes
(print "1" lock)
(posix:with-stream-lock
(lock :block t :shared nil)
(with-open-file (out "foo" :direction :output
:if-exists :supersede
:if-does-not-exist :create)
(print data out))))
(when (probe-file ".lock") (delete-file ".lock"))))
</code>
In article <·······················@o13g2000cwo.googlegroups.com>,
············@gmail.com" <···········@gmail.com> wrote:
> I'm trying to protect write access to a file with a lock file, but I'm
> using a loop because I can't find a blocking open or something like the
> C select() function.
Just to clarify, C doesn't have a select() function. That's a POSIX
function.
If you run Lisp on a POSIX system, and your Lisp has a FFI (as most do),
you can call select() from it as well.
--
Barry Margolin, ······@alum.mit.edu
Arlington, MA
*** PLEASE post questions in newsgroups, not directly to me ***
*** PLEASE don't copy me on replies, I'll read them in the group ***
············@gmail.com" <···········@gmail.com> writes:
> I'm trying to protect write access to a file with a lock file, but I'm
> using a loop because I can't find a blocking open or something like the
> C select() function.
Why use the lock file?
Is this file written by more than one program?
Or is it written by a single Lisp application with more than one process
(or thread)?
If the latter, then you should be able to use the multi-processing
system to use a different locking scheme.
If these are independent programs, then there may be an issue.
> My current implentation is like this:
>
> <code>
>
> (defun write-to-file (data)
> (unwind-protect
> (progn
> (do () ((not (probe-file (merge-pathnames ".lock"))) nil))
> (with-open-file (out (merge-pathnames ".lock")
> :direction :output :if-does-not-exist :create)
> (with-open-file (out (merge-pathnames "foo")
> :direction :output :if-exists :append)
> (print data out)))
> (delete-file (merge-pathnames ".lock")))
> (when (probe-file (merge-pathnames ".lock")) (delete-file
> (merge-pathnames ".lock")))))
>
> </code>
>
> however, the (do ... ) seems somehow suboptimal. The writes should be
> extremely fast, I'm writing tiny amounts of data, so the (do ...) loop
> doesn't spin much.
>
> Is there a better approach to this?
Well, I still see problems. I mean, what happens if someone creates a
lock file between the execution of (PROBE-FILE ...) and the subsequent
WITH-OPEN-FILE. It may be safer to add an :IF-EXISTS :ERROR to the
lock file creation. But if you end up doing that, then you might as
well just dispense with the PROBE-FILE test entirely and use the file
opening as the test for the existence of the lock file. You will need
to write an error handler anyway, so it just makes sense to consolidate
the handling of trying to write to a locked file.
OK. Now about the busy waiting. I think you can't really get around
some of that if these really are different OS-level programs that you
are trying to synchronize through the file system. But you can make the
waiting a bit more resource friendly by including a SLEEP call in the
loop.
So, I would try doing something like the following (untested) code:
(block exclusive-write
(loop (handler-case
(with-open-file (out ".lock" :direction :output
:if-exists :error
:if-does-not-exist :create)
(unwind-protect
(with-open-file (data-file "foo" ...)
(print data data-file)
(return-from exclusive-write nil))
(delete-file ".lock")))
(<whatever-the-proper-error-type-is>
(sleep 0.2))))) ; Or whatever seems reasonable
Add in any MERGE-PATHNAMES and other features as needed.
--
Thomas A. Russ, USC/Information Sciences Institute